QMDB Core Integration for Substrate Storage Enhancement

Treasury Proposal: QMDB Core Integration for Substrate Storage

Enhancement Proponent: 12K97oseeBxoWnyYRU3DJkxKTiDGFBh1pK4jLyuUJzYHkqdD

Date: [29.01.2025]

Requested DOT: 5 000

Short description: Initial integration of QMDB (https://github.com/LayerZero-Labs/qmdb) core functionality into Substrate (https://github.com/paritytech/polkadot-sdk). Project Category/Type: Storage and Performance Optimization

Previous Treasury Proposals:

**Context of the Proposal

**This proposal focuses on the first phase of QMDB integration, specifically implementing core storage functionality without iterators and prefixes with limited support or non-optimal support. This foundational work will enable future enhancements while maintaining compatibility with existing Substrate systems.

Project Details

Technology stack:

RUST language for QMDB integration with Substrate
DevOps tools for deployment and monitoring

Documentation:

Core components and architecture documentation to be developed
Integration specifications and API documentation
Performance benchmarking methodology

Research and PoC:

What this project will not provide:

Full prefix search functionality
Production-ready indexing solution
Advanced iterator implementation

Team members

Team Lead: Vas Soshnikov
Team Member: Andrew Orlov

Team's experience

Previous Polkadot grants:

Team Code Repos

Problem Statement
Current Substrate storage implementation faces several fundamental challenges that this initial integration phase aims to address: Current crate ‘sp-trie’ implementation requires optimisation Inefficient key-based operations affecting basic transaction processing Suboptimal transaction verification process

Key Limitations of QMDB

Does not support iterators and prefix searches
Non-limitation, mostly its design of QMDN, requires additional memory (RAM, Disk) for indexing (indexer)

Proposal Objectives & Solutions Objectives

Integrate QMDB as the proof storage engine within Substrate
Enable Merkle Proof-based transaction verification
Benchmark QMDB inside our DevNet (https://dev.qfnetwork.xyz)

Milestones, Timelines, and Budgets Milestone

1. Implement QMDB storage traits Working prototype integrated with Substrate
3 weeks
2 000 DOT

2. Develop a storage indexing mechanism In-memory B-Tree or sorted sparse list
2 weeks
1 500 DOT

3. Conduct full system testing All Substrate tests pass
1 week
750 DOT

4. Deploy, benchmark and optimise Live on DevNet with monitoring
1 week
750 DOT

Risks and Mitigation Risk Mitigation Strategy

QMDB may not be production-ready
Extensive DevNet testing before MainNet deployment
Substrate’s evolving architecture may cause compatibility issues
Freeze the Substrate version until integration is stable
Increased resource consumption due to indexers
Optimise RAM/disk usage and implement caching mechanisms

Resources Required

Senior Rust Developer: Expertise in Substrate and low-level systems
Senior Rust Developer: Specializing in database architecture
DevNet Environment: Dedicated nodes for rigorous testing

Success Criteria

Substrate nodes use QMDB for transaction storage
All Substrate and pallet-specific tests pass successfully
Performance benchmarks demonstrate improved transaction processing times
DevNet deployment runs stably with real-time monitoring and alerting

Payment Conditions

Payments will be made in instalments based on milestone completion. Each instalment will be subject to the successful delivery of agreed milestones. A detailed progress report will be submitted on Polkassembly/Subsquare.

Why Polkadot Network?

Polkadot’s focus on modular and scalable blockchain infrastructure makes it an ideal candidate for advanced storage optimisations like QMDB integration. By enhancing storage efficiency, this proposal aligns with Polkadot’s long-term vision of high-performance, decentralized systems.

Additional References
QMDB Technical Paper
QMDB GitHub Repository

The Treasury Proposal is about integrating QMDB Core into Substrate Storage. QMDB is a storage system that will help make Substrate faster and better. The project will happen in four steps:

Make QMDB work with Substrate (3 weeks, 2,000 DOT)
Create a way to organize the stored information (2 weeks, 1,500 DOT)
Test everything to make sure it works (1 week, 750 DOT)
Put it on a test network and make it better (1 week, 750 DOT)

The team working on this has experience with Polkadot and has worked on similar projects before. They will use the Rust language and some tools to help them deploy and watch the project.

This project will make Substrate better by making it faster and more efficient. But it won't have some advanced features like full prefix search functionality. The team will test the project a lot to make sure it works well before using it on the main network.

Polkadot is a good place for this project because it wants to make fast and scalable blockchain systems. This project will help make Polkadot's storage better and faster.