Storage Backend Requirements for OpenRaft: Implementing RaftLogStorage and RaftStateMachine Traits

OpenRaft requires storage backends to implement two async traits—RaftLogStorage for log persistence and RaftStateMachine for state machine snapshots—guaranteeing serialized writes, hole-free logs, and immediate durability.

OpenRaft, the high-performance Raft consensus library from databendlabs/openraft, strictly separates the consensus algorithm from concrete persistence mechanisms. To integrate a custom storage layer, developers must satisfy rigorous storage backend requirements for OpenRaft by implementing specific async traits that ensure correctness, durability, and thread safety across the distributed system.

Core Storage Traits in OpenRaft

The storage layer in openraft/src/storage/v2/ defines the contract between the Raft core and persistence. Two traits form the foundation of any backend implementation.

RaftLogStorage Trait Requirements

Defined in openraft/src/storage/v2/raft_log_storage.rs, RaftLogStorage<C> manages the replicated log, vote persistence, and committed log tracking. Implementations must provide several async methods with strict behavioral guarantees:

• get_log_state(): Returns LogState containing last_purged_log_id and last_log_id to establish the current log bounds.
• save_vote(): Persists the current vote to stable storage before returning, ensuring leader election safety.
• append(): Adds entries to the log, makes them immediately readable, and invokes the IOFlushed callback only after entries are durable on disk.
• truncate_after(): Removes entries after a specific log ID while maintaining the hole-free invariant.
• purge(): Removes entries up to a specific log ID, typically invoked after snapshot installation to reclaim space.

RaftStateMachine Trait Requirements

Defined in openraft/src/storage/v2/raft_state_machine.rs, RaftStateMachine<C> handles business logic application and snapshot management. Critical methods include:

• applied_state(): Must return the last applied log ID and last applied membership, which the Raft core uses for correctness checks during restarts.
• apply(): Applies log entries to the state machine, persists the last-applied log ID, executes business logic, and optionally triggers state machine persistence.
• begin_receiving_snapshot(), install_snapshot(), get_current_snapshot(): Manage snapshot reception, installation, and retrieval during replication and node restart.
• build_snapshot(): Creates a consistent point-in-time snapshot of the state machine.

Critical Durability and Consistency Constraints

OpenRaft imposes strict architectural constraints documented directly in the trait definitions. Violating these invariants compromises consensus safety and leader election guarantees.

Requirement	Implementation Detail	Source Location
Serialized Writes	All write operations (append, truncate, purge, vote) must be serialized so later writes never overtake earlier ones.	RaftLogStorage serialization comment in raft_log_storage.rs
Hole-Free Logs	The log must contain no gaps; truncate_after and purge must maintain contiguous indices.	RaftLogStorage "no hole" comment
Vote Durability	save_vote must persist the vote to stable storage before returning.	save_vote documentation
General Durability	Once a write returns, data must be durable (e.g., via fsync or database WAL).	RaftLogStorage durability paragraph
Append Callback	append must invoke the IOFlushed callback only after entries are persisted, while making entries readable immediately.	append documentation
Committed Log Tracking	Optional save_committed/read_committed for storing last committed log ID; required if snapshots are non-atomic.	save_committed docs in raft_log_storage.rs
Thread Safety	Implementations must be Send + Sync and async-compatible (OptionalSend + OptionalSync).	Trait bounds in both storage files

Production Storage Backend Options

The OpenRaft repository provides reference implementations demonstrating different durability levels and architectural patterns.

In-Memory Storage (MemStore)

Located in examples/raft-kv-memstore/src/store/mod.rs, the in-memory backend implements both RaftLogStorage and RaftStateMachine using RAM-backed structures. It provides no persistence and is suitable only for testing and demos, as explicitly noted in the example READMEs.

RocksDB Backend (RocksStore)

The examples/rocksstore/ directory contains a production-ready implementation:

• src/log_store.rs: Persistent RaftLogStorage using RocksDB with WAL and sync options for durability.
• src/state_machine.rs: Persistent RaftStateMachine with snapshot support.

This backend guarantees durability through RocksDB's write-ahead log and is the recommended pattern for production deployments.

OpenDAL Integration

For remote object storage (S3, OSS, etc.), OpenRaft supports OpenDalStore, where logs remain local (e.g., RocksDB) while snapshots are uploaded via OpenDAL. Snapshot durability then depends on the remote storage's consistency model.

Implementing a Custom Storage Backend

To create a production storage layer, implement the traits with strict adherence to the durability and ordering constraints.

Minimal In-Memory Log Store

The following skeleton demonstrates the required structure for RaftLogStorage, satisfying the async, Send + Sync, and callback requirements:

use openraft::storage::RaftLogStorage;
use openraft::{RaftTypeConfig, LogState, Vote, LogId};

pub struct MemLogStore<C: RaftTypeConfig> {
    pub logs: Vec<C::Entry>,
    pub vote: Option<Vote<C::NodeId>>,
}

#[openraft::async_trait]
impl<C> RaftLogStorage<C> for MemLogStore<C>
where
    C: RaftTypeConfig,
{
    type LogReader = Self;

    async fn get_log_state(&mut self) -> std::io::Result<LogState<C>> {
        let last = self.logs.last().map(|e| e.log_id().clone());
        Ok(LogState {
            last_purged_log_id: None,
            last_log_id: last,
        })
    }

    async fn get_log_reader(&mut self) -> Self::LogReader {
        self.clone()
    }

    async fn save_vote(&mut self, vote: &Vote<C::NodeId>) -> std::io::Result<()> {
        self.vote = Some(vote.clone());
        Ok(())
    }

    async fn append<I>(&mut self, entries: I, callback: openraft::storage::IOFlushed<C>) -> std::io::Result<()>
    where
        I: IntoIterator<Item = C::Entry> + openraft::OptionalSend,
        I::IntoIter: openraft::OptionalSend,
    {
        self.logs.extend(entries);
        callback();
        Ok(())
    }

    async fn truncate_after(&mut self, last_log_id: Option<LogId<C::NodeId>>) -> std::io::Result<()> {
        if let Some(id) = last_log_id {
            self.logs.retain(|e| e.log_id() <= &id);
        } else {
            self.logs.clear();
        }
        Ok(())
    }

    async fn purge(&mut self, log_id: LogId<C::NodeId>) -> std::io::Result<()> {
        self.logs.retain(|e| e.log_id() > &log_id);
        Ok(())
    }
}

Key implementation details:

• append calls the callback after entries are logically stored.
• All methods respect the no-hole rule by maintaining contiguous indices.
• The trait bounds (OptionalSend, OptionalSync) are satisfied automatically.

RocksDB-Backed Log Store Skeleton

For production durability, implement RaftLogStorage using RocksDB with explicit fsync:

use openraft::{RaftLogStorage, RaftTypeConfig, LogState, Vote, LogId};
use rocksdb::{DB, WriteBatch};

pub struct RocksLogStore<C: RaftTypeConfig> {
    db: DB,
    _t: std::marker::PhantomData<C>,
}

#[openraft::async_trait]
impl<C> RaftLogStorage<C> for RocksLogStore<C>
where
    C: RaftTypeConfig,
{
    type LogReader = Self;

    async fn get_log_state(&mut self) -> std::io::Result<LogState<C>> {
        unimplemented!()
    }

    async fn get_log_reader(&mut self) -> Self::LogReader {
        self.clone()
    }

    async fn save_vote(&mut self, vote: &Vote<C::NodeId>) -> std::io::Result<()> {
        self.db.put(b"vote", bincode::serialize(vote)?)?;
        self.db.flush()?; // fsync for durability
        Ok(())
    }

    async fn append<I>(&mut self, entries: I, callback: openraft::storage::IOFlushed<C>) -> std::io::Result<()>
    where
        I: IntoIterator<Item = C::Entry> + openraft::OptionalSend,
        I::IntoIter: openraft::OptionalSend,
    {
        let mut batch = WriteBatch::default();
        for e in entries {
            let key = format!("log:{:020}", e.log_id().index);
            batch.put(key.as_bytes(), bincode::serialize(&e)?);
        }
        self.db.write(batch)?;
        self.db.flush()?; // Guarantee durability before callback
        callback();
        Ok(())
    }
}

Key production requirements:

• Calls db.flush() (or equivalent sync) to meet the durability requirement.
• Uses bincode (or any serializer) to store binary entries.
• Implements the same no-hole guarantee by using a contiguous key space (log:<index>).

Snapshot Builder Implementation

The RaftSnapshotBuilder trait creates consistent snapshots:

use openraft::{RaftSnapshotBuilder, RaftTypeConfig, Snapshot, SnapshotMeta};

pub struct MySnapshotBuilder<C: RaftTypeConfig> {
    db: rocksdb::DB,
    _t: std::marker::PhantomData<C>,
}

#[openraft::async_trait]
impl<C> RaftSnapshotBuilder<C> for MySnapshotBuilder<C>
where
    C: RaftTypeConfig,
{
    async fn build_snapshot(&mut self) -> std::io::Result<Snapshot<C>> {
        let meta = SnapshotMeta {
            last_log_id: /* fetch from log store */,
            last_membership: /* fetch from state machine */,
            ..Default::default()
        };
        
        let tmp = tempfile::NamedTempFile::new()?;
        // ... copy DB contents to temp file ...
        
        Ok(Snapshot {
            meta,
            data: std::fs::File::open(tmp.path())?,
        })
    }
}

Critical implementation notes:

• The snapshot builder must capture a consistent view of both the log and the state machine.
• The produced Snapshot is later installed via RaftStateMachine::install_snapshot.

Wiring Storage into a Raft Instance

Finally, instantiate the Raft node with your storage backend:

use openraft::{Raft, Config, declare_raft_types};

declare_raft_types! {
    pub TypeConfig:
        D = MyCommand,
        R = MyResponse;
}

let log_store = RocksLogStore::<TypeConfig>::new("path/to/db")?;
let state_machine = MyStateMachine::new(...);
let snapshot_builder = MySnapshotBuilder::new(log_store.db.clone());

let raft = Raft::new(
    node_id,
    Arc::new(Config::default()),
    MyNetwork, // implements RaftNetworkV2
    log_store,
    Arc::new(state_machine),
).await?;

This wiring pattern mirrors the production rocksstore example in the repository (examples/rocksstore/src/test.rs), demonstrating the required integration of the storage backend with the Raft core.

Summary

Implementing a storage backend for OpenRaft requires strict adherence to architectural constraints defined in the openraft/src/storage/v2/ module:

• Implement two core traits: RaftLogStorage for log and vote persistence, and RaftStateMachine for state machine application and snapshots.
• Guarantee serialization: All write operations must execute sequentially without reordering, ensuring later writes never overtake earlier ones.
• Maintain hole-free logs: Log indices must remain contiguous; truncation and purge operations must not create gaps.
• Ensure durability: Writes must be durable (e.g., via fsync or database WAL) before returning, with save_vote persisting votes immediately and append invoking the IOFlushed callback only after persistence.
• Satisfy thread safety: Implementations must be Send + Sync and compatible with OptionalSend and OptionalSync bounds to function across async replication tasks.

Frequently Asked Questions

What are the two core traits required for an OpenRaft storage backend?

OpenRaft requires implementations of RaftLogStorage<C> and RaftStateMachine<C>, defined in openraft/src/storage/v2/raft_log_storage.rs and openraft/src/storage/v2/raft_state_machine.rs respectively. RaftLogStorage handles log entries, vote persistence, and committed log tracking, while RaftStateMachine manages business logic application, snapshot creation, and snapshot installation.

Why must OpenRaft storage operations be serialized?

All write operations—including append, truncate_after, purge, and save_vote—must be serialized so that later writes never overtake earlier ones, as documented in the RaftLogStorage trait comments. This ordering guarantee ensures that the Raft core can rely on a consistent view of the log during leader election, log replication, and crash recovery, preventing split-brain scenarios and log divergence.

How does OpenRaft ensure durability guarantees are met?

OpenRaft requires that once a write method returns, the data must be durable on stable storage, typically achieved through fsync, database write-ahead logs, or equivalent mechanisms. Specifically, save_vote must persist the vote before returning, and append must invoke the IOFlushed callback only after entries are confirmed on disk, as specified in the trait documentation for RaftLogStorage in openraft/src/storage/v2/raft_log_storage.rs.

Can I use an in-memory storage backend for production OpenRaft deployments?

No, the in-memory backend (MemStore) found in examples/raft-kv-memstore/src/store/mod.rs is explicitly designed for testing and demonstrations only, as noted in the example READMEs. Production deployments must use a persistent backend such as the RocksDB implementation in examples/rocksstore/, which guarantees durability through write-ahead logs and sync operations, ensuring data survives process crashes and restarts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}