# OpenRaft Storage Best Practices: A Complete Implementation Guide > Master OpenRaft storage best practices implementing RaftLogReader, RaftLogStorage, and RaftStateMachine for consensus safety. Ensure durability and atomic appends. - Repository: [Databend Labs/openraft](https://github.com/databendlabs/openraft) - Tags: best-practices - Published: 2026-02-28 --- **OpenRaft storage implementations require three core traits—`RaftLogReader`, `RaftLogStorage`, and `RaftStateMachine`—with strict durability guarantees, atomic appends, and proper snapshot handling to ensure consensus safety.** The `databendlabs/openraft` repository provides a flexible, trait-based storage abstraction that separates the Raft consensus protocol from underlying persistence mechanisms. Implementing OpenRaft storage correctly is critical for maintaining safety properties and achieving high performance in distributed systems. ## Understanding OpenRaft Storage Architecture OpenRaft delegates all persistence concerns to user-provided storage implementations through a well-defined trait hierarchy. Understanding these boundaries is essential before writing production storage code. ### The Three Core Storage Traits OpenRaft storage implementations must provide three distinct capabilities: 1. **`RaftLogReader`** – Provides read access to log entries and the current vote. Implemented by types that can iterate over the replicated log. 2. **`RaftLogStorage`** – Handles durable persistence of log entries, votes, snapshots, and log truncation. This trait requires `append` operations to be atomic and durable before invoking the `IOFlushed` callback. 3. **`RaftStateMachine`** – Applies committed commands to the application state, builds and installs snapshots, and exposes the current applied state. In OpenRaft, the state machine type also implements `RaftSnapshotBuilder`. ### Type Configuration and Generic Implementations All storage implementations must be generic over the application's `TypeConfig`. The `declare_raft_types!` macro in [`openraft/src/type_config.rs`](https://github.com/databendlabs/openraft/blob/main/openraft/src/type_config.rs) defines the request/response types, node ID, and leader ID implementation. Storage code must use these associated types rather than concrete types to remain compatible with the generic Raft core. ## Implementing the Log Store The log store is the most performance-critical component, handling every write to the replicated log. Implementation mistakes here directly impact cluster availability and durability. ### Core Requirements for Log Storage Every OpenRaft log store must satisfy these invariants: - **Atomic Append**: The `append` method in `RaftLogStorage` must write all entries and invoke the `IOFlushed` callback only after the write is truly durable (fsync or WAL flush). - **Consistent Reads**: `try_get_log_entries` and `limited_get_log_entries` must return entries in strict index order without gaps. - **Metadata Persistence**: Vote and `last_purged_log_id` must survive restarts, typically stored as separate key/value pairs. - **Non-blocking Operations**: All methods are `async`; heavy IO should use `C::spawn_blocking` to avoid blocking the async runtime. ### In-Memory Log Store Example The reference in-memory implementation in [`stores/memstore/src/lib.rs`](https://github.com/databendlabs/openraft/blob/main/stores/memstore/src/lib.rs) demonstrates the trait contract: ```rust impl RaftLogStorage for Arc { type LogReader = Self; async fn append(&mut self, entries: I, callback: IOFlushed) -> Result<(), io::Error> where I: IntoIterator> + OptionalSend, { // Serialize each entry as JSON and store in a BTreeMap. let mut log = self.log.write().await; for entry in entries { let s = serde_json::to_string(&entry) .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e.to_string()))?; log.insert(entry.index(), s); } // Notify the Raft core that the write is durable. callback.io_completed(Ok(())); Ok(()) } // … other required methods (get_log_state, purge, truncate_after, etc.) … } ``` Key implementation details from [`stores/memstore/src/lib.rs`](https://github.com/databendlabs/openraft/blob/main/stores/memstore/src/lib.rs) lines 55-74: - The log map uses `RwLock` to allow concurrent readers while serializing writes. - JSON serialization ensures compatibility with the state machine snapshot format. - The `IOFlushed` callback signals durability immediately for in-memory storage. ### Production-Grade RocksDB Implementation For production deployments, the RocksDB example in [`examples/rocksstore/src/log_store.rs`](https://github.com/databendlabs/openraft/blob/main/examples/rocksstore/src/log_store.rs) provides a durable, high-performance implementation: ```rust async fn append(&mut self, entries: I, callback: IOFlushed) -> Result<(), io::Error> where I: IntoIterator> + Send, { for entry in entries { let id = id_to_bin(entry.index()); self.db.put_cf( self.cf_logs(), id, serde_json::to_vec(&entry).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?, )?; } // Flush WAL in a background thread; the callback will be invoked when the OS flush finishes. let db = self.db.clone(); std::thread::spawn(move || { let res = db.flush_wal(true).map_err(io::Error::other); callback.io_completed(res); }); Ok(()) } ``` Critical patterns from [`examples/rocksstore/src/log_store.rs`](https://github.com/databendlabs/openraft/blob/main/examples/rocksstore/src/log_store.rs) lines 81-122: - Separate column families (`meta` and `logs`) isolate metadata from log data. - Entries are serialized to JSON vectors for storage. - The WAL flush occurs in a dedicated thread to prevent blocking the async runtime, with `IOFlushed` invoked only after `flush_wal(true)` completes. ## Building the State Machine and Snapshot System The state machine represents the application-specific logic that processes committed Raft commands. Unlike the log store, which is generic to Raft, the state machine implements your business logic while conforming to OpenRaft's trait boundaries. ### Command Application Logic The `apply` method in `RaftStateMachine` receives a stream of committed entries and must apply them to the application state. From [`stores/memstore/src/lib.rs`](https://github.com/databendlabs/openraft/blob/main/stores/memstore/src/lib.rs) lines 100-130: ```rust async fn apply(&mut self, mut entries: Strm) -> Result<(), io::Error> where Strm: Stream, io::Error>> + Unpin + OptionalSend, { let mut sm = self.sm.write().await; while let Some((entry, responder)) = entries.try_next().await? { sm.last_applied_log = Some(entry.log_id); let response = match entry.payload { EntryPayload::Blank => ClientResponse(None), EntryPayload::Normal(ref data) => { let previous = sm.client_status.insert(data.client.clone(), data.status.clone()); ClientResponse(previous) } EntryPayload::Membership(ref mem) => { sm.last_membership = StoredMembership::new(Some(entry.log_id), mem.clone()); ClientResponse(None) } }; if let Some(responder) = responder { responder.send(response); } } Ok(()) } ``` Key implementation requirements: - The state machine uses `RwLock` to allow concurrent snapshot reads during application. - Each entry updates `last_applied_log` to track progress. - Membership changes are stored in `last_membership` to enable cluster topology recovery from snapshots. - The `responder` channel sends results back to the client once the entry is applied. ### Snapshot Building Best Practices Snapshots enable log compaction and efficient node recovery. The `RaftSnapshotBuilder` trait (implemented by the state machine type) creates point-in-time snapshots. From [`stores/memstore/src/lib.rs`](https://github.com/databendlabs/openraft/blob/main/stores/memstore/src/lib.rs) lines 109-135: ```rust async fn build_snapshot(&mut self) -> Result, io::Error> { // Optional delay for testing. if let Some(d) = self.block.get_blocking(&BlockOperation::DelayBuildingSnapshot) { TypeConfig::sleep(d).await; } // Serialize the whole machine. let sm = self.sm.read().await; let data = serde_json::to_vec(&*sm) .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e.to_string()))?; let last_applied_log = sm.last_applied_log; let last_membership = sm.last_membership.clone(); // Create meta information. let meta = SnapshotMeta { last_log_id: last_applied_log, last_membership, snapshot_id: format!("snap-{}", uuid::Uuid::new_v4()), }; // Store the snapshot for later retrieval. let snapshot = MemStoreSnapshot { meta: meta.clone(), data: data.clone() }; *self.current_snapshot.write().await = Some(snapshot); Ok(Snapshot { meta, snapshot: Cursor::new(data) }) } ``` Critical snapshot patterns: - **Consistency**: Acquire the state machine read lock before serializing to ensure a consistent point-in-time view. - **Metadata**: Include `last_log_id` and `last_membership` in `SnapshotMeta` so Raft can verify snapshot validity and recover cluster membership. - **Unique IDs**: Generate unique snapshot IDs (e.g., using UUID) to prevent collision during concurrent operations. - **Format**: Use binary serialization (JSON, protobuf, or bincode) rather than text formats for production deployments. ### Snapshot Installation and Recovery When a lagging node needs to catch up or a new node joins, OpenRaft installs snapshots via the `install_snapshot` method. From [`stores/memstore/src/lib.rs`](https://github.com/databendlabs/openraft/blob/main/stores/memstore/src/lib.rs): ```rust async fn install_snapshot( &mut self, meta: &SnapshotMeta, snapshot: SnapshotDataOf, ) -> Result<(), io::Error> { let new_snapshot = MemStoreSnapshot { meta: meta.clone(), data: snapshot.into_inner(), }; // Deserialize into the real machine. let new_sm: MemStoreStateMachine = serde_json::from_slice(&new_snapshot.data) .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e.to_string()))?; *self.sm.write().await = new_sm; *self.current_snapshot.write().await = Some(new_snapshot); Ok(()) } ``` Installation requirements: - **Validation**: Verify the snapshot format before replacing the state machine to prevent corruption. - **Atomic Replacement**: Update both the state machine and the cached snapshot atomically (under the write lock) to ensure consistency. - **Error Handling**: Return `io::Error` with `InvalidData` kind for deserialization failures to signal unrecoverable snapshot corruption. ## Testing and Fault Injection Robust OpenRaft storage implementations must handle adverse conditions including disk failures, network partitions, and slow IO. The reference `MemLogStore` in [`stores/memstore/src/lib.rs`](https://github.com/databendlabs/openraft/blob/main/stores/memstore/src/lib.rs) provides a `BlockConfig` mechanism for deterministic fault injection. ```rust // In a test: let block = BlockConfig::default(); block.set_blocking(BlockOperation::DelayBuildingSnapshot, Duration::from_millis(100)); ``` Available fault injection points include: - **`DelayBuildingSnapshot`**: Simulates slow snapshot serialization to test Raft's timeout handling. - **`return_empty_limited_get`**: Forces `limited_get_log_entries` to return empty vectors, exercising the Raft core's retry logic. - **`enable_saving_committed`**: Toggles whether the log store remembers the last committed index, testing state machine recovery scenarios. These mechanisms are extensively exercised in the repository's test suite (`tests/tests/*`) to verify correctness under failure conditions. ## Performance Optimization Strategies Optimizing OpenRaft storage requires balancing durability guarantees with throughput. The following strategies derive from production implementations in the codebase: | Situation | Recommendation | Implementation Details | |-----------|----------------|------------------------| | High write throughput | Batch entries in `append` and flush once per batch | In [`examples/rocksstore/src/log_store.rs`](https://github.com/databendlabs/openraft/blob/main/examples/rocksstore/src/log_store.rs), collect entries into a vector before calling `flush_wal(true)` | | Large state machines | Use binary serialization formats (protobuf, bincode) instead of JSON | The `MemStore` uses JSON for clarity, but production systems should replace `serde_json::to_vec` with binary codecs | | Frequent compaction | Promptly purge old logs and update `last_purged_log_id` | Call `purge` when the state machine advances; store `last_purged_log_id` in the metadata column family | | Multi-threaded workloads | Use `Arc>` for the log map | Allows concurrent `limited_get_log_entries` calls while serializing appends | | Testing latency | Inject delays via `BlockConfig` | Verify Raft's retry logic handles slow IO gracefully | ## Summary Implementing OpenRaft storage correctly requires careful attention to trait contracts and durability guarantees. Key takeaways include: - Implement **both** `RaftLogReader` and `RaftLogStorage` for the log store, and `RaftStateMachine` for the application state. - Persist **vote** and **last_purged_log_id** as separate metadata keys to survive restarts. - Ensure `append` invokes the `IOFlushed` callback **only after** writes are truly durable (post-fsync or WAL flush). - Use `RwLock` or equivalent to allow concurrent log reads while serializing writes. - Provide a **snapshot builder** that captures consistent state machine snapshots with proper `SnapshotMeta` including `last_log_id` and `last_membership`. - Implement `install_snapshot` to atomically replace state machine state and validate snapshot format integrity. - Use `BlockConfig` during testing to simulate slow IO, empty reads, and forced failures. ## Frequently Asked Questions ### What are the three main traits required for OpenRaft storage? OpenRaft storage implementations must provide **`RaftLogReader`** for reading log entries and votes, **`RaftLogStorage`** for persisting log entries and metadata, and **`RaftStateMachine`** for applying commands and managing snapshots. These traits are defined in [`openraft/src/storage/mod.rs`](https://github.com/databendlabs/openraft/blob/main/openraft/src/storage/mod.rs) and separate concerns between the consensus engine and persistence layer. ### How does OpenRaft ensure log entry durability? Durability is enforced through the **`IOFlushed`** callback mechanism. When `RaftLogStorage::append` is called, the implementation must write entries to durable storage (such as RocksDB with WAL enabled or files with fsync) and only invoke `callback.io_completed(Ok(()))` after the operating system confirms the write is persistent. This ensures the Raft core never acknowledges a log entry until it is safely stored. ### What is the purpose of the IOFlushed callback in OpenRaft storage? The **`IOFlushed`** callback bridges asynchronous storage operations and the Raft core's durability requirements. It signals that previously appended entries have reached non-volatile storage and the Raft state machine can safely advance. Implementations must call `io_completed` with `Ok(())` only after fsync or WAL flush completes, or with an error if the write fails, allowing Raft to handle the failure appropriately. ### How should snapshots be handled in OpenRaft storage implementations? Snapshots require implementing **`RaftSnapshotBuilder`** (typically on the state machine type) and the **`install_snapshot`** method. The builder must serialize the entire state machine state while holding a read lock to ensure consistency, producing a `SnapshotMeta` containing `last_log_id` and `last_membership`. The `install_snapshot` method must atomically replace the current state machine with the deserialized snapshot data, validating the format before updating internal state. Both operations should use `Arc>` for safe concurrent access during snapshotting.