How is Data Persisted in OpenRaft: A Complete Guide to Log and State Machine Storage

OpenRaft persists data through two distinct storage traits—RaftLogStorage for the distributed log and RaftStateMachine for application state—requiring implementations to durably flush votes, log entries, and snapshots before acknowledging completion.

OpenRaft, the high-performance Raft consensus implementation maintained by databendlabs/openraft, delegates all durability concerns to user-provided storage backends. Understanding how data is persisted in OpenRaft requires examining the separation between log storage and state-machine storage, the specific durability contracts each trait demands, and the snapshot lifecycle that enables crash recovery.

The Two-Layer Persistence Architecture

OpenRaft splits persistence responsibilities into two orthogonal components to maximize flexibility:

• RaftLogStorage (openraft/src/storage/v2/raft_log_storage.rs): Handles the Raft log—votes, log entries, and the optional committed index. This must survive crashes to prevent double-voting or log loss.
• RaftStateMachine (openraft/src/storage/v2/raft_state_machine.rs): Handles application state and snapshots. This can be either fully persistent (e.g., RocksDB) or transient with snapshot-based recovery.

This separation allows deployments to optimize log storage for write-heavy append workloads while using different backends (memory, RocksDB, custom databases) for the state machine.

Persisting the Raft Log with RaftLogStorage

The RaftLogStorage trait defines three critical durability points that implementations must honor to guarantee safety.

Saving the Vote

Before responding to any RPC, a node must persist its current term and voted-for candidate. The save_vote method must not return until the vote is flushed to durable storage:

async fn save_vote(&mut self, vote: &VoteOf<C>) -> Result<(), io::Error>;

Failure to durably save the vote can lead to split-brain scenarios after a crash, where a node might vote twice in the same term.

Appending Log Entries

The append method receives new log entries and a completion callback. The contract requires that callback.io_completed() is invoked only after entries are durably written:

async fn append<I>(&mut self, entries: I, callback: IOFlushed<C>) -> Result<(), io::Error>
where I: IntoIterator<Item = C::Entry> + OptionalSend;

This callback mechanism allows OpenRaft to pipeline log appends while ensuring durability before committing entries to the state machine.

Optional Commit Index Persistence

OpenRaft optionally allows storing the last committed log index via save_committed:

async fn save_committed(&mut self, _committed: Option<LogIdOf<C>>) -> Result<(), io::Error> {
    Ok(())
}

Persisting the committed index enables a node to resume with accurate RaftMetrics.last_applied after restart and prevents re-applying stale membership configurations. The memstore implementation demonstrates toggling this feature via an AtomicBool:

pub fn enable_save_committed(&self, enable: bool) {
    self.enable_saving_committed.store(enable, Ordering::Relaxed);
}

State Machine Persistence and Snapshots

While the log storage requires immediate durability, the state machine offers more flexibility through the RaftStateMachine trait.

Applied State Tracking

The applied_state method returns the last applied log ID and membership configuration, allowing OpenRaft to determine where to resume applying logs after restart:

async fn applied_state(&mut self) -> Result<(Option<LogIdOf<C>>, StoredMembership<C>), io::Error>;

Applying Entries

The apply method receives a stream of committed entries. Implementations can choose their persistence strategy:

• Immediate persistence: Flush to disk during each apply call (slower but simpler recovery).
• Deferred persistence: Batch writes in memory, relying on snapshots for crash recovery (higher throughput).

Snapshot Lifecycle

Snapshots provide full state-machine backups for recovery and log compaction. The lifecycle involves four key methods:

1. try_create_snapshot_builder: Initiates snapshot creation (e.g., copying RocksDB files).
2. begin_receiving_snapshot: Returns a Cursor for writing incoming snapshot data from the leader.
3. install_snapshot: Persists the received snapshot to disk and updates state.
4. get_current_snapshot: Returns the current snapshot for replication to lagging nodes.

The RocksDB example in examples/rocksstore/src/state_machine.rs demonstrates snapshot-only persistence, where the state machine writes to RocksDB but relies on periodic snapshots for crash recovery rather than WAL (Write-Ahead Logging) for every update.

Key Source Files for OpenRaft Persistence

File	Purpose
openraft/src/storage/v2/raft_log_storage.rs	Defines RaftLogStorage trait with save_vote, append, and save_committed methods.
openraft/src/storage/v2/raft_state_machine.rs	Defines RaftStateMachine trait for applied state, entry application, and snapshot management.
stores/memstore/src/lib.rs	In-memory reference implementation showing optional save_committed toggling and callback-based append.
examples/rocksstore/src/state_machine.rs	Production-grade RocksDB implementation demonstrating snapshot-based persistence and state machine recovery.
openraft/src/docs/data/log_pointers.md	Documentation explaining the relationship between flushed, committed, and applied indices.

Summary

• OpenRaft delegates all persistence to user-implemented traits, separating log storage (RaftLogStorage) from state-machine storage (RaftStateMachine).
• Log durability requires synchronous flushing of votes before RPC returns and callback confirmation after append completes.
• State machines can choose between immediate persistence (every apply flushes) or snapshot-based recovery (periodic snapshots with transient state).
• Optional save_committed allows nodes to persist the last committed index, improving metric accuracy and preventing membership re-application after crashes.
• Snapshots provide the primary recovery mechanism for state machines, with install_snapshot called automatically on startup if a snapshot exists.

Frequently Asked Questions

What is the difference between RaftLogStorage and RaftStateMachine?

RaftLogStorage persists the consensus log—votes, log entries, and optionally the committed index—ensuring Raft safety properties survive crashes. RaftStateMachine persists application data and handles snapshots, allowing flexibility in how user data is stored (memory, RocksDB, etc.) while ensuring the state machine can recover to the last applied log index.

Is the committed index required to be persisted?

No, persisting the committed index via RaftLogStorage::save_committed is optional. OpenRaft provides a default no-op implementation. However, persisting it improves recovery by allowing RaftMetrics.last_applied to survive restarts and prevents re-applying stale membership configurations. The memstore example shows how to toggle this feature at runtime.

How does OpenRaft recover after a crash?

Recovery follows a specific sequence: First, OpenRaft restores the log from RaftLogStorage, recovering the vote and log entries up to the last append. If save_committed was implemented, it also knows the last committed index. Then, it calls RaftStateMachine::install_snapshot with any existing snapshot file to restore application state. Finally, it replays any log entries after the snapshot’s last applied index via apply, bringing the state machine to the current committed state.

Can I use different storage backends for logs and state machine?

Yes, the trait-based architecture explicitly supports mixing storage backends. For example, you can store the Raft log in a high-performance write-ahead log (WAL) or dedicated disk while running the state machine in memory for speed (relying on snapshots for recovery), or vice versa. The examples/rocksstore demonstrates using RocksDB for both, while stores/memstore shows pure in-memory operation, illustrating the flexibility of the persistence model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}