# OpenRaft Storage Durability Guarantees: How Committed Entries Survive Node Crashes

> Understand OpenRaft storage durability guarantees. Learn how committed entries survive node crashes with serialized I/O, progress tracking, and commit-index confirmation.

- Repository: [Databend Labs/openraft](https://github.com/databendlabs/openraft)
- Tags: internals
- Published: 2026-02-28

---

**OpenRaft guarantees that every Raft log entry deemed committed is durably persisted to underlying storage before application to the state machine or acknowledgment to clients, enforced through serialized write-I/O operations, three-stage I/O progress tracking, and commit-index advancement only after explicit flush confirmation.**

OpenRaft is a high-performance Raft consensus engine written in Rust that powers distributed systems requiring strong consistency. Understanding OpenRaft storage durability guarantees is critical for building production-grade storage backends that prevent committed data loss during crashes or network partitions. This article examines the specific mechanisms—serialized writes, `IOProgress` tracking, and flush-conditional commit logic—as implemented in the `databendlabs/openraft` repository.

## Serialized Write-I/O for Vote and Log Ordering

The foundation of OpenRaft's durability model rests on **strict write serialization**. The storage layer must process all write requests—both votes and log entries—in the exact order issued by the Raft core, without reordering or parallelizing dependent operations.

### The RaftLogStorage Trait Contract

The durability requirement is explicitly documented in the `RaftLogStorage` trait definition at [`openraft/src/storage/v2/raft_log_storage.rs`](https://github.com/databendlabs/openraft/blob/main/openraft/src/storage/v2/raft_log_storage.rs) (lines 27–33):

```rust
/// - All write‑IO must be serialized, i.e. the internal implementation must **NOT**
///   apply a latter write request before a former write request is completed.
///   This rule applies to both `vote` and `log` IO. E.g., Saving a vote and
///   appending a log entry must be serialized too.

```

This contract ensures that when a leader persists a vote for term `T`, all subsequent log entries from term `T` are physically written to disk only after that vote completes. If an implementation respects this ordering (such as the RocksDB-backed store in the `rocksstore` example), the term metadata will always precede associated log entries, preventing the "log-entry-reordering" problem detailed in the **IO Ordering** documentation.

## Three-Stage I/O Progress Tracking

OpenRaft tracks every I/O operation through an explicit state machine using the `IOProgress<T>` struct, defined in [`openraft/src/raft_state/io_state/io_progress.rs`](https://github.com/databendlabs/openraft/blob/main/openraft/src/raft_state/io_state/io_progress.rs) (lines 9–14). This design decouples operation submission from durability confirmation.

### Accepted, Submitted, and Flushed Cursors

The struct maintains three monotonically-increasing cursors:

```rust
/// Tracks the progress of I/O operations through three stages:
/// accepted → submitted → flushed.
pub(crate) struct IOProgress<T> {
    accepted: Option<T>,   // highest operation accepted by RaftCore
    submitted: Option<T>,  // highest operation handed to storage
    flushed:   Option<T>,  // highest operation confirmed durable
}

```

- **Accepted**: The Raft core has scheduled the operation (e.g., `append` or `save_vote`).
- **Submitted**: The storage layer has taken ownership, typically by queueing a batch.
- **Flushed**: The storage implementation has invoked the `IOFlushed` callback, confirming data resides on persistent media.

The code enforces the invariant `flushed ≤ submitted ≤ accepted` using validation macros. Notably, the `flush` method tolerates out-of-order notifications, allowing storage engines that complete operations non-monotonically to still satisfy the model as long as the `flushed` cursor eventually advances.

## Commit Advancement Only After Flush

The Raft core updates the **commit index** exclusively for log entries whose `flushed` cursor has advanced past their `LogIOId`. This logic, described in [`openraft/src/docs/data/log_io_progress.md`](https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/data/log_io_progress.md) (lines 24–28), ensures that an entry is never considered committed until it is physically durable.

Because the commit index serves as a global ordering guarantee, this design automatically propagates durability to all followers. Once a leader has flushed an entry, any follower receiving that entry can apply it safely, knowing the associated term is already persisted on the leader's storage.

## Preventing Committed Data Loss

The durability guarantees prevent data loss in specific failure scenarios that would compromise safety in weaker implementations.

### Crash Recovery Safety

If a crash occurs after a log entry persists but before its associated term vote persists, the node could restart with an entry lacking term metadata. According to [`openraft/src/docs/protocol/io_ordering.md`](https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/io_ordering.md), this gap allows an older leader to overwrite the entry, effectively erasing a previously committed operation. Serialized writes and flush-aware commits eliminate this window of vulnerability.

### Network Partition Handling

During network partitions, followers only apply entries that are both replicated **and** flushed on the leader. This prevents a leader from falsely claiming commitment for entries that might be lost before reaching persistent storage, ensuring that any entry reported as committed to a client will survive subsequent leader elections and crashes.

## Implementing Durable Storage

Storage backends must implement the `RaftLogStorage` trait while respecting the serialization and callback contracts to satisfy OpenRaft storage durability guarantees.

### Minimal In-Memory Example

The following Rust code demonstrates a minimal implementation that respects the durability contract. Note the `await` on `save_vote` ensuring serialization and the invocation of `callback` only after the simulated persistence:

```rust
use openraft::storage::{IOFlushed, RaftLogStorage};
use openraft::{RaftTypeConfig, Vote};

pub struct InMemoryStore<C> {
    pub votes: Vec<Vote<C::NodeId>>,
    pub logs: Vec<C::Entry>,
}

#[async_trait::async_trait]
impl<C> RaftLogStorage<C> for InMemoryStore<C>
where
    C: RaftTypeConfig,
{
    type LogReader = ();

    async fn get_log_state(&mut self) -> Result<openraft::storage::LogState<C>, std::io::Error> {
        # Ok(openraft::storage::LogState::default())

    }

    async fn get_log_reader(&mut self) -> Self::LogReader {
        ()
    }

    /// Persist the vote **before** returning. The `await` ensures the operation is serialized.
    async fn save_vote(&mut self, vote: &Vote<C::NodeId>) -> Result<(), std::io::Error> {
        self.votes.push(vote.clone());
        // In a real store you would call `fsync`/`flush` here.
        Ok(())
    }

    /// Append log entries and invoke the callback once they are flushed.
    async fn append<I>(&mut self, entries: I, callback: IOFlushed<C>) -> Result<(), std::io::Error>
    where
        I: IntoIterator<Item = C::Entry> + openraft::OptionalSend,
        I::IntoIter: openraft::OptionalSend,
    {
        for entry in entries {
            self.logs.push(entry);
        }
        // Simulate durable persistence.
        callback(Ok(()));
        Ok(())
    }
}

```

### Production RocksDB Implementation

For production deployments, the `rocksstore` example provides a reference implementation:

```rust
use openraft_rocksstore::RocksStore;
use openraft::Raft;

let store = RocksStore::new("/var/lib/openraft")?;
let raft = Raft::new(config, network, store.clone(), store);

```

The RocksDB implementation at [`examples/rocksstore/src/log_store.rs`](https://github.com/databendlabs/openraft/blob/main/examples/rocksstore/src/log_store.rs) guarantees durability by:

- Persisting votes to a dedicated column family (`vote_cf`) **before** any log entry from that term.
- Using a write-ahead log (WAL) that is explicitly flushed (`db.flush()`) before invoking the `IOFlushed` callback.

This ensures the `flushed` cursor in `IOProgress` only advances after data is physically written to disk.

## Summary

- OpenRaft requires **serialized write-I/O** for both votes and log entries, ensuring `save_vote` operations complete before subsequent log appends from the same term.
- The **`IOProgress<T>`** struct tracks operations through **accepted**, **submitted**, and **flushed** stages, with the `flushed` cursor indicating durability.
- The Raft core advances the **commit index only after** the `flushed` cursor moves forward, guaranteeing committed entries survive crashes.
- Storage implementations must invoke the **`IOFlushed`** callback only after data is physically persisted (e.g., after `fsync` or WAL flush).

## Frequently Asked Questions

### What happens if a storage implementation does not serialize write-I/O?

If writes are reordered so that a log entry from term T persists before the vote for term T, a crash could leave the node with an entry lacking its associated term metadata. According to the **IO Ordering** documentation in [`openraft/src/docs/protocol/io_ordering.md`](https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/io_ordering.md), this allows an older leader to potentially overwrite the entry, violating safety guarantees and causing committed data loss.

### How does OpenRaft track when a write becomes durable?

OpenRaft uses the **`IOProgress<T>`** struct defined in [`openraft/src/raft_state/io_state/io_progress.rs`](https://github.com/databendlabs/openraft/blob/main/openraft/src/raft_state/io_state/io_progress.rs) to monitor each operation through three monotonic stages: **accepted** (scheduled by RaftCore), **submitted** (handed to storage), and **flushed** (confirmed durable by storage). The invariant `flushed ≤ submitted ≤ accepted` is strictly enforced.

### Can the storage layer report I/O completions out of order?

Yes. The `IOProgress::flush` method tolerates out-of-order notifications, allowing storage engines that complete operations non-monotonically to still satisfy the durability model. The Raft core only requires that the `flushed` cursor eventually advances past a given `IOId`, not that notifications arrive in sequence.

### Where can I find a production-ready storage implementation example?

The **`rocksstore`** example in [`examples/rocksstore/src/log_store.rs`](https://github.com/databendlabs/openraft/blob/main/examples/rocksstore/src/log_store.rs) demonstrates a production-grade implementation using RocksDB. It persists votes to a dedicated column family and invokes the `IOFlushed` callback only after calling `db.flush()` on the write-ahead log, ensuring OpenRaft storage durability guarantees are met.