Transaction Support in OpenRaft Storage: Implementing Atomic Raft Log Operations

OpenRaft requires storage implementations to provide transactional atomicity through the "no-hole" rule, ensuring that operations like appending entries and truncating logs maintain consecutive log continuity without gaps.

Transaction support in OpenRaft storage is not exposed through a generic transaction API, but rather enforced through strict durability guarantees that implementers must provide in the databendlabs/openraft repository. The storage layer relies on two core traits—RaftLogStorage and RaftStateMachine—that are deliberately stateless from the engine's perspective, placing the responsibility for atomicity entirely on the backend implementation.

Core Storage Traits and Stateless Design

OpenRaft's storage architecture separates concerns between log persistence and state machine management. The RaftLogStorage trait handles Raft log entries, votes, and log-related metadata, while the RaftStateMachine trait manages application state and snapshot handling. Both traits are designed to be stateless from the engine's viewpoint, meaning the implementer must supply all durability guarantees through the underlying storage mechanism.

This design choice requires that every method implementation treat multi-step updates as single atomic units. When the Raft engine calls truncate_after() during snapshot installation, the storage layer must ensure that both the snapshot persistence and log truncation occur together, or not at all.

The "No-Hole" Rule: Foundation of Transactional Safety

The defining characteristic of transaction support in OpenRaft is the "no-hole" rule, which mandates absolute log continuity at all times. This rule creates two critical constraints for storage implementations:

• Log continuity: After any storage operation, the log must remain a consecutive sequence without missing indices. OpenRaft only examines the last log ID to ensure correctness, making gaps fatal to consensus safety.
• Atomicity of multi-step updates: Operations touching multiple state pieces—such as installing a snapshot while truncating old logs—must appear as a single atomic step to the engine.

In openraft/src/storage/v2/raft_log_storage.rs, the documentation explicitly warns implementers about this requirement:

/// - It must not leave a **hole** in logs. Because Raft only examines the last log id to ensure
///   correctness.
#[since(version = "0.10.0")]
async fn truncate_after(&mut self, last_log_id: Option<LogIdOf<C>>) -> Result<(), io::Error>;

Source: openraft/src/storage/v2/raft_log_storage.rs – lines 113‑122

Required Atomic Operations

OpenRaft expects four specific operations to be transaction-safe. Each method signature implies atomic guarantees that the implementation must satisfy:

append() and Callback Guarantees

The append method must persist entries to disk before invoking the IOFlushed callback. As documented in openraft/src/storage/v2/raft_log_storage.rs (lines 86‑102), the entries must be readable immediately after the method returns, and the callback signals only after durable persistence is confirmed.

Typical implementation: Use a write-batch (e.g., RocksDB WriteBatch) to group entries, flush the batch to disk, then trigger the callback.

truncate_after() and Snapshot Coordination

The truncate_after method presents the highest risk for creating holes. When OpenRaft installs a snapshot, it must truncate logs older than the snapshot's last log ID. If truncation succeeds but the snapshot write fails—or vice versa—the log chain becomes inconsistent.

Implementation requirement: Use a single database transaction that deletes the log range and writes snapshot metadata atomically.

purge() and Range Deletion

The purge method removes obsolete logs up to a specific log_id. Like truncation, this must delete a continuous range in a single batch to prevent gaps that would break the "last-log-id" invariant.

save_committed() for Transient State Machines

For transient state machines that do not immediately persist applied entries, save_committed ensures the committed index survives restarts. This prevents the engine from replaying logs that have already been applied after a crash.

Detection and Recovery Mechanisms

OpenRaft implements defensive checks in StorageHelper::get_initial_state to detect storage implementations that violate transactional guarantees.

Recovery from Non-Transactional Implementations

During node startup, the helper loads the last persisted log ID, last applied state, and snapshot metadata. If last_log_id < last_applied, indicating that logs were removed without updating the applied index (a "hole"), the helper automatically purges logs up to last_applied to restore consistency:

if last_log_id < last_applied {
    self.log_store.purge(last_applied.clone().unwrap()).await?;
}

Source: openraft/src/storage/helper.rs – lines 149‑162

This recovery mechanism appears in the change log as a warning for non-transactional implementations: "This may be caused by a non-transactional impl of the store, e.g. installing snapshot and removing logs are not atomic." (Source: change-log.md, line 2299)

While this auto-recovery prevents crashes, it incurs performance penalties and risks data loss in edge cases. A correct implementation eliminates the need for this cleanup.

Implementation Patterns

Production storage backends typically implement these transactional requirements through three main approaches:

In-Memory Stores (MemStore)

The reference MemStore implementation achieves atomicity through Rust's ownership model. All operations occur within a single mutable struct, where the borrow checker prevents data races and ensures that updates happen as unified memory operations.

RocksDB-Backed Stores

RocksDB implementations leverage rocksdb::WriteBatch to group multiple operations:

1. Call put for new log entries or snapshot metadata
2. Call delete_range for log truncation
3. Execute db.write(batch) to commit atomically
4. Signal the IOFlushed callback only after successful commit

File-Based Storage

File-based stores use atomic rename operations. Write snapshot data to a temporary file, use fs::rename to move it into place, and delete old log files within the same logical transaction step—or wrap both actions in a higher-level transaction manager.

Code Example: Atomic Write-Batch Implementation

The following pattern from the RocksDB-backed store demonstrates how to implement atomic append and truncate_after operations:

impl<C> RaftLogStorage<C> for RocksStore<C>
where
    C: RaftTypeConfig,
{
    // Append entries atomically
    async fn append<I>(&mut self, entries: I, callback: IOFlushed<C>) -> Result<(), io::Error>
    where
        I: IntoIterator<Item = C::Entry> + OptionalSend,
        I::IntoIter: OptionalSend,
    {
        let mut batch = rocksdb::WriteBatch::default();
        for entry in entries {
            batch.put(entry_key(entry.log_id()), serialize(&entry)?) ;
        }
        // Flush to disk, then signal callback
        self.db.write(batch)?;
        callback.signal(Ok(()));
        Ok(())
    }

    // Install snapshot + truncate logs atomically
    async fn truncate_after(&mut self, last_log_id: Option<LogIdOf<C>>) -> Result<(), io::Error> {
        let mut batch = rocksdb::WriteBatch::default();
        // Delete log range beyond `last_log_id`
        batch.delete_range(log_prefix(), range_start(last_log_id));
        // Write snapshot meta (already persisted by state machine)
        batch.put(snapshot_meta_key(), serialize(&self.pending_snapshot_meta)?);
        self.db.write(batch)?;
        Ok(())
    }
}

Testing Atomicity with Blocking Helpers

To verify that your implementation properly waits for persistence, use the RaftLogStorageExt::blocking_append helper. This converts the async append into a blocking call that waits on the IOFlushed callback:

let (tx, rx) = C::oneshot();
let callback = IOFlushed::<C>::signal(tx);
self.append(entries, callback).await?;
rx.await?;

Source: openraft/src/storage/v2/raft_log_storage_ext.rs – lines 24‑33

Summary

• OpenRaft does not provide a generic transaction API; instead, it requires storage implementations to guarantee atomicity for specific operations.
• The "no-hole" rule mandates that log entries remain consecutive at all times, with no missing indices between the first and last log ID.
• Four operations require atomic implementation: append (with callback guarantees), truncate_after (coordinated with snapshots), purge (continuous range deletion), and save_committed (for transient state machines).
• Detection mechanisms in StorageHelper::get_initial_state can identify and recover from non-transactional implementations, but correct atomicity prevents the need for costly cleanup.
• Implementation techniques vary by backend: Rust's borrow checker for in-memory stores, WriteBatch for RocksDB, and atomic renames for file-based storage.

Frequently Asked Questions

Does OpenRaft provide a built-in transaction API for storage implementations?

No, OpenRaft does not expose a generic "begin transaction" or "commit" API. Instead, the engine expects the RaftLogStorage and RaftStateMachine implementations to handle atomicity internally using the underlying storage engine's transaction capabilities, such as RocksDB's WriteBatch or database transactions.

What happens if my storage implementation is not transactional?

OpenRaft detects inconsistencies during startup via StorageHelper::get_initial_state. If it finds that last_log_id is less than last_applied—indicating logs were deleted without proper state updates—it automatically purges the inconsistent range and logs a warning. While this prevents immediate crashes, it can cause performance degradation and potential data loss in edge cases.

How do I test that my storage implementation provides proper atomicity?

Use the RaftLogStorageExt::blocking_append helper to convert async append operations into blocking calls that wait for the IOFlushed callback. This allows you to verify that entries are actually persisted to disk before the callback fires. Additionally, simulate crashes between multi-step operations (like snapshot installation and log truncation) to ensure the system maintains consistency.

Can I use a distributed transaction manager with OpenRaft storage?

Yes, you can implement the storage traits using any transaction manager that provides atomic commits. The traits only require that methods like append and truncate_after appear atomic from the Raft engine's perspective. Whether you use a local WriteBatch, a distributed transaction coordinator, or file-system atomic operations depends entirely on your storage backend architecture.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}