Implementing a Custom Storage Backend for OpenRaft: A Complete Guide

To implement a custom storage backend for OpenRaft, you must implement the RaftLogStorage and RaftStateMachine traits from the openraft crate, ensuring strict serialization guarantees and atomic persistence of log entries, votes, and state machine snapshots.

OpenRaft, the Raft consensus implementation maintained by databendlabs, separates the consensus algorithm from persistence concerns through well-defined storage traits. This architecture allows you to plug in any durable storage solution—whether RocksDB, PostgreSQL, S3, or a custom in-memory store—by implementing two core interfaces. This guide walks through the concrete steps required to build a production-ready custom storage backend for OpenRaft, referencing the actual source files and trait definitions from the repository.

Understanding the Storage Architecture

OpenRaft defines two primary storage traits that decouple consensus logic from data persistence. According to the source code in openraft/src/storage/v2/, these contracts are:

• RaftLogStorage (raft_log_storage.rs): Manages the replicated log, persistent voting state, and optional committed index pointers.
• RaftStateMachine (raft_state_machine.rs): Holds application-specific state and applies committed log entries.

A third trait, RaftLogReader, is required for streaming log entries to replication tasks. The reference implementation in stores/memstore/src/lib.rs demonstrates a complete in-memory backend that satisfies all three contracts.

Step 1: Define Your Type Configuration

Before implementing storage, you must define a type configuration that describes the request/response payloads, node identifier, and leader-id type. OpenRaft uses the declare_raft_types! macro to generate the necessary boilerplate.

use openraft::declare_raft_types;

declare_raft_types! {
    /// Your concrete configuration.
    pub MyConfig:
        D = MyRequest,          // client request type
        R = MyResponse,         // client response type
        Node = u64,             // node identifier (could be UUID, string, etc.)
        LeaderId = openraft::impls::leader_id_std::LeaderId<MyConfig>; // or custom
}

This pattern appears in stores/memstore/src/lib.rs and establishes the generic parameters used throughout your storage implementation.

Step 2: Implement the RaftLogStorage Trait

The RaftLogStorage trait in openraft/src/storage/v2/raft_log_storage.rs requires you to persist log entries, votes, and purge pointers. Your implementation must guarantee strict serialization—all writes must complete in order, and later writes must not complete before earlier ones.

use openraft::{
    storage::{IOFlushed, RaftLogReader, RaftLogStorage},
    type_config::RaftTypeConfig,
    LogState, VoteOf, LogIdOf,
};
use std::io;

pub struct MyLogStore {
    // e.g., rocksdb::DB, sled::Db, etc.
    db: MyDb,
}

#[async_trait::async_trait]
impl<C> RaftLogStorage<C> for MyLogStore
where
    C: RaftTypeConfig,
{
    type LogReader = MyLogReader; // must implement `RaftLogReader<C>`

    async fn get_log_state(&mut self) -> Result<LogState<C>, io::Error> {
        // Read last_purged_log_id and last_log_id from storage.
        // Return `LogState { last_purged_log_id, last_log_id }`
        todo!()
    }

    async fn get_log_reader(&mut self) -> Self::LogReader {
        MyLogReader::new(self.db.clone())
    }

    async fn save_vote(&mut self, vote: &VoteOf<C>) -> Result<(), io::Error> {
        // Persist `vote` atomically.
        todo!()
    }

    async fn append<I>(&mut self, entries: I, callback: IOFlushed<C>) -> Result<(), io::Error>
    where
        I: IntoIterator<Item = C::Entry> + OptionalSend,
        I::IntoIter: OptionalSend,
    {
        // Serialize each entry and write to `db`.
        // After flushing to disk, invoke `callback.flushed()`.
        todo!()
    }

    async fn truncate_after(&mut self, last_log_id: Option<LogIdOf<C>>) -> Result<(), io::Error> {
        // Delete entries > `last_log_id`.
        todo!()
    }

    async fn purge(&mut self, log_id: LogIdOf<C>) -> Result<(), io::Error> {
        // Delete entries ≤ `log_id` and update `last_purged_log_id`.
        todo!()
    }
}

Critical correctness guarantees from the trait documentation:

• No gaps in the log – truncation must leave a continuous sequence.
• Strict write serialization – a later write must not complete before an earlier one.
• Durable callback invocation – the callback passed to append must only be invoked after entries are flushed to stable storage.

Implementing the RaftLogReader

The RaftLogReader trait requires methods for streaming log entries to replication tasks. Define a separate reader struct that shares access to your underlying database:

pub struct MyLogReader {
    db: MyDb,
}

impl<C> RaftLogReader<C> for MyLogReader
where
    C: RaftTypeConfig,
{
    async fn get_log_entries(&mut self, start: u64, limit: Option<u64>) -> Result<Vec<C::Entry>, io::Error> {
        // Fetch entries from `start` up to `limit`.
        todo!()
    }

    async fn get_log_state(&mut self) -> Result<LogState<C>, io::Error> {
        // Same as `RaftLogStorage::get_log_state`, but read‑only.
        todo!()
    }
}

The MemStore reader in stores/memstore/src/lib.rs demonstrates how to share an underlying data structure between the storage and reader implementations.

Step 3: Implement the RaftStateMachine Trait

The RaftStateMachine trait in openraft/src/storage/v2/raft_state_machine.rs manages your application state. It must apply committed log entries and handle snapshot generation and installation.

use openraft::{
    storage::{Snapshot, RaftSnapshotBuilder, RaftStateMachine, RaftStateMachineApplier},
    type_config::RaftTypeConfig,
    StorageError,
};

pub struct MyStateMachine {
    // e.g., in‑memory hash map, RocksDB column family, etc.
    inner: MyDb,
}

#[async_trait::async_trait]
impl<C> RaftStateMachine<C> for MyStateMachine
where
    C: RaftTypeConfig,
{
    async fn apply(&mut self, applier: &mut impl RaftStateMachineApplier<C>) -> Result<(), StorageError<C>> {
        // Iterate over `applier.entries()` and update `inner` accordingly.
        // Update your own `last_applied_log` and `last_membership`.
        todo!()
    }

    async fn get_snapshot_builder(&mut self) -> Result<Box<dyn RaftSnapshotBuilder<C>>, StorageError<C>> {
        // Return a builder that can serialize the current state into a snapshot.
        todo!()
    }

    async fn install_snapshot(&mut self, snapshot: &mut impl SnapshotData<C>) -> Result<(), StorageError<C>> {
        // Replace current state with `snapshot` data.
        todo!()
    }
}

If your state machine persists the committed index itself, you can omit save_committed and read_committed from your log store implementation.

Step 4: Wire Everything Together

With your storage traits implemented, instantiate the Raft struct using your custom backend. This initialization pattern appears in the repository’s integration tests such as tests/tests/public_api_test.rs.

use openraft::{Raft, Config};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::build("my-node".into())
        .validate()
        .unwrap();

    let log_store = MyLogStore::new(/* db handle */);
    let state_machine = MyStateMachine::new(/* db handle */);

    // Create the Raft node.
    let raft = Raft::new(config, log_store, state_machine).await?;

    // Use `raft` to join a cluster, propose commands, etc.
    Ok(())
}

Step 5: Validate with the Test Suite

OpenRaft provides a store builder in openraft/src/testing/log/store_builder.rs that runs the full test matrix against your implementation. This validates that your backend satisfies the strict correctness contracts required for consensus safety.

use openraft::testing::log::store_builder::StoreBuilder;

#[tokio::test]
async fn custom_backend_passes_all_tests() {
    let builder = StoreBuilder::new(|| async {
        // Return a tuple (log_store, state_machine).
        (MyLogStore::new(...), MyStateMachine::new(...))
    });
    builder.run_all().await.unwrap();
}

Running this suite verifies that your implementation maintains no holes in the log, serializes writes correctly, and invokes IO callbacks only after durable flushing.

Summary

Implementing a custom storage backend for OpenRaft requires careful adherence to the trait contracts defined in the openraft crate. The essential steps include:

• Defining a TypeConfig using the declare_raft_types! macro to establish your application’s request/response types and node identifiers.
• Implementing RaftLogStorage in openraft/src/storage/v2/raft_log_storage.rs to persist log entries, votes, and purge pointers with strict serialization guarantees.
• Providing a RaftLogReader implementation for streaming log entries to replication tasks.
• Implementing RaftStateMachine in openraft/src/storage/v2/raft_state_machine.rs to apply committed entries and manage snapshots.
• Validating with StoreBuilder from openraft/src/testing/log/store_builder.rs to ensure compliance with consensus safety requirements.

By following these patterns demonstrated in stores/memstore/src/lib.rs, you can integrate any durable storage solution into OpenRaft, giving you a flexible, production-ready consensus layer tailored to your infrastructure.

Frequently Asked Questions

What are the consistency requirements for a custom OpenRaft storage backend?

Your implementation must guarantee three critical properties derived from the trait documentation in raft_log_storage.rs: no gaps in the log (truncation must leave a continuous sequence), strict write serialization (later writes must not complete before earlier ones), and durable callback invocation (the IOFlushed callback passed to append must only be invoked after entries are flushed to stable storage). Violating these contracts can lead to consensus failures or split-brain scenarios.

Can I use an existing database like RocksDB or PostgreSQL as the storage backend?

Yes. The RaftLogStorage and RaftStateMachine traits are database-agnostic interfaces. You can wrap a rocksdb::DB instance, a PostgreSQL connection pool, or an S3 client in your struct and implement the required async methods. The reference implementation in stores/memstore/src/lib.rs demonstrates the structural pattern for wrapping external storage handles. Ensure your underlying database transactions satisfy the strict serialization guarantees required by the traits.

How does OpenRaft handle storage errors during log replication?

Storage errors returned from RaftLogStorage methods (such as append or save_vote) are treated as fatal to the current operation. If append fails, you must not invoke the IOFlushed callback, and the error propagates upward to the Raft engine. The consensus layer expects that storage implementations handle transient errors internally (e.g., through retries) and only return errors for unrecoverable failures. Persistent storage failures will cause the node to step down or halt to prevent safety violations.

What is the difference between RaftLogStorage and RaftStateMachine?

RaftLogStorage, defined in openraft/src/storage/v2/raft_log_storage.rs, manages the Write-Ahead Log (WAL)—it stores raw log entries, the node's persistent voting state, and log purge pointers. It is concerned with durability and replication. RaftStateMachine, defined in openraft/src/storage/v2/raft_state_machine.rs, manages application state—it applies committed log entries to your business logic and handles snapshot generation and installation. While the log storage is mandatory for consensus safety, the state machine is where your application logic lives. They are architecturally separate to allow log compaction (snapshots) to occur independently of log storage operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}