OpenRaft Storage Durability Guarantees: How Committed Entries Survive Node Crashes

OpenRaft guarantees that every Raft log entry deemed committed is durably persisted to underlying storage before application to the state machine or acknowledgment to clients, enforced through serialized write-I/O operations, three-stage I/O progress tracking, and commit-index advancement only after explicit flush confirmation.

OpenRaft is a high-performance Raft consensus engine written in Rust that powers distributed systems requiring strong consistency. Understanding OpenRaft storage durability guarantees is critical for building production-grade storage backends that prevent committed data loss during crashes or network partitions. This article examines the specific mechanisms—serialized writes, IOProgress tracking, and flush-conditional commit logic—as implemented in the databendlabs/openraft repository.

Serialized Write-I/O for Vote and Log Ordering

The foundation of OpenRaft's durability model rests on strict write serialization. The storage layer must process all write requests—both votes and log entries—in the exact order issued by the Raft core, without reordering or parallelizing dependent operations.

The RaftLogStorage Trait Contract

The durability requirement is explicitly documented in the RaftLogStorage trait definition at openraft/src/storage/v2/raft_log_storage.rs (lines 27–33):

/// - All write‑IO must be serialized, i.e. the internal implementation must **NOT**
///   apply a latter write request before a former write request is completed.
///   This rule applies to both `vote` and `log` IO. E.g., Saving a vote and
///   appending a log entry must be serialized too.

This contract ensures that when a leader persists a vote for term T, all subsequent log entries from term T are physically written to disk only after that vote completes. If an implementation respects this ordering (such as the RocksDB-backed store in the rocksstore example), the term metadata will always precede associated log entries, preventing the "log-entry-reordering" problem detailed in the IO Ordering documentation.

Three-Stage I/O Progress Tracking

OpenRaft tracks every I/O operation through an explicit state machine using the IOProgress<T> struct, defined in openraft/src/raft_state/io_state/io_progress.rs (lines 9–14). This design decouples operation submission from durability confirmation.

Accepted, Submitted, and Flushed Cursors

The struct maintains three monotonically-increasing cursors:

/// Tracks the progress of I/O operations through three stages:
/// accepted → submitted → flushed.
pub(crate) struct IOProgress<T> {
    accepted: Option<T>,   // highest operation accepted by RaftCore
    submitted: Option<T>,  // highest operation handed to storage
    flushed:   Option<T>,  // highest operation confirmed durable
}

• Accepted: The Raft core has scheduled the operation (e.g., append or save_vote).
• Submitted: The storage layer has taken ownership, typically by queueing a batch.
• Flushed: The storage implementation has invoked the IOFlushed callback, confirming data resides on persistent media.

The code enforces the invariant flushed ≤ submitted ≤ accepted using validation macros. Notably, the flush method tolerates out-of-order notifications, allowing storage engines that complete operations non-monotonically to still satisfy the model as long as the flushed cursor eventually advances.

Commit Advancement Only After Flush

The Raft core updates the commit index exclusively for log entries whose flushed cursor has advanced past their LogIOId. This logic, described in openraft/src/docs/data/log_io_progress.md (lines 24–28), ensures that an entry is never considered committed until it is physically durable.

Because the commit index serves as a global ordering guarantee, this design automatically propagates durability to all followers. Once a leader has flushed an entry, any follower receiving that entry can apply it safely, knowing the associated term is already persisted on the leader's storage.

Preventing Committed Data Loss

The durability guarantees prevent data loss in specific failure scenarios that would compromise safety in weaker implementations.

Crash Recovery Safety

If a crash occurs after a log entry persists but before its associated term vote persists, the node could restart with an entry lacking term metadata. According to openraft/src/docs/protocol/io_ordering.md, this gap allows an older leader to overwrite the entry, effectively erasing a previously committed operation. Serialized writes and flush-aware commits eliminate this window of vulnerability.

Network Partition Handling

During network partitions, followers only apply entries that are both replicated and flushed on the leader. This prevents a leader from falsely claiming commitment for entries that might be lost before reaching persistent storage, ensuring that any entry reported as committed to a client will survive subsequent leader elections and crashes.

Implementing Durable Storage

Storage backends must implement the RaftLogStorage trait while respecting the serialization and callback contracts to satisfy OpenRaft storage durability guarantees.

Minimal In-Memory Example

The following Rust code demonstrates a minimal implementation that respects the durability contract. Note the await on save_vote ensuring serialization and the invocation of callback only after the simulated persistence:

use openraft::storage::{IOFlushed, RaftLogStorage};
use openraft::{RaftTypeConfig, Vote};

pub struct InMemoryStore<C> {
    pub votes: Vec<Vote<C::NodeId>>,
    pub logs: Vec<C::Entry>,
}

#[async_trait::async_trait]
impl<C> RaftLogStorage<C> for InMemoryStore<C>
where
    C: RaftTypeConfig,
{
    type LogReader = ();

    async fn get_log_state(&mut self) -> Result<openraft::storage::LogState<C>, std::io::Error> {
        # Ok(openraft::storage::LogState::default())

    }

    async fn get_log_reader(&mut self) -> Self::LogReader {
        ()
    }

    /// Persist the vote **before** returning. The `await` ensures the operation is serialized.
    async fn save_vote(&mut self, vote: &Vote<C::NodeId>) -> Result<(), std::io::Error> {
        self.votes.push(vote.clone());
        // In a real store you would call `fsync`/`flush` here.
        Ok(())
    }

    /// Append log entries and invoke the callback once they are flushed.
    async fn append<I>(&mut self, entries: I, callback: IOFlushed<C>) -> Result<(), std::io::Error>
    where
        I: IntoIterator<Item = C::Entry> + openraft::OptionalSend,
        I::IntoIter: openraft::OptionalSend,
    {
        for entry in entries {
            self.logs.push(entry);
        }
        // Simulate durable persistence.
        callback(Ok(()));
        Ok(())
    }
}

Production RocksDB Implementation

For production deployments, the rocksstore example provides a reference implementation:

use openraft_rocksstore::RocksStore;
use openraft::Raft;

let store = RocksStore::new("/var/lib/openraft")?;
let raft = Raft::new(config, network, store.clone(), store);

The RocksDB implementation at examples/rocksstore/src/log_store.rs guarantees durability by:

• Persisting votes to a dedicated column family (vote_cf) before any log entry from that term.
• Using a write-ahead log (WAL) that is explicitly flushed (db.flush()) before invoking the IOFlushed callback.

This ensures the flushed cursor in IOProgress only advances after data is physically written to disk.

Summary

• OpenRaft requires serialized write-I/O for both votes and log entries, ensuring save_vote operations complete before subsequent log appends from the same term.
• The IOProgress<T> struct tracks operations through accepted, submitted, and flushed stages, with the flushed cursor indicating durability.
• The Raft core advances the commit index only after the flushed cursor moves forward, guaranteeing committed entries survive crashes.
• Storage implementations must invoke the IOFlushed callback only after data is physically persisted (e.g., after fsync or WAL flush).

Frequently Asked Questions

What happens if a storage implementation does not serialize write-I/O?

If writes are reordered so that a log entry from term T persists before the vote for term T, a crash could leave the node with an entry lacking its associated term metadata. According to the IO Ordering documentation in openraft/src/docs/protocol/io_ordering.md, this allows an older leader to potentially overwrite the entry, violating safety guarantees and causing committed data loss.

How does OpenRaft track when a write becomes durable?

OpenRaft uses the IOProgress<T> struct defined in openraft/src/raft_state/io_state/io_progress.rs to monitor each operation through three monotonic stages: accepted (scheduled by RaftCore), submitted (handed to storage), and flushed (confirmed durable by storage). The invariant flushed ≤ submitted ≤ accepted is strictly enforced.

Can the storage layer report I/O completions out of order?

Yes. The IOProgress::flush method tolerates out-of-order notifications, allowing storage engines that complete operations non-monotonically to still satisfy the durability model. The Raft core only requires that the flushed cursor eventually advances past a given IOId, not that notifications arrive in sequence.

Where can I find a production-ready storage implementation example?

The rocksstore example in examples/rocksstore/src/log_store.rs demonstrates a production-grade implementation using RocksDB. It persists votes to a dedicated column family and invokes the IOFlushed callback only after calling db.flush() on the write-ahead log, ensuring OpenRaft storage durability guarantees are met.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}