OpenRaft Storage Performance Tuning: A Complete Guide to Configuration and Optimization

OpenRaft storage performance tuning centers on adjusting batch sizes, snapshot policies, and persistence settings in the Config struct to match your disk I/O and network characteristics.

OpenRaft is a high-performance Raft consensus implementation maintained by databendlabs/openraft that separates log storage from state machine storage through the RaftLogStorage and RaftStateMachine traits. By tuning parameters in openraft/src/config/config.rs and selecting appropriate store implementations, you can optimize throughput for SSD-backed production clusters or minimize latency for in-memory benchmarks.

Understanding OpenRaft Storage Architecture

OpenRaft delegates all persistence to two pluggable interfaces: the log store (RaftLogStorage) for Raft log entries, and the state machine store (RaftStateMachine) for snapshots and applied state. The repository provides an in-memory implementation (MemLogStore/MemStateMachine) in stores/memstore/src/lib.rs for testing, and a RocksDB-backed implementation (RocksLogStore) in examples/rocksstore/src/log_store.rs for production.

Log Store Implementation Details

The RaftLogStorage trait requires methods like append, purge, and get_log_state. In stores/memstore/src/lib.rs, MemLogStore stores entries in a BTreeMap<u64, String> with JSON serialization, invoking the IOFlushed callback immediately after insertion. Purging removes entries up to a given LogId and can be artificially delayed via BlockOperation::PurgeLog.

In examples/rocksstore/src/log_store.rs, RocksLogStore persists log entries in a RocksDB column family. Logs are stored with big-endian binary keys (id_to_bin) to preserve ordering. The append method writes each entry as a JSON payload, then spawns a thread that flushes the WAL and invokes the callback (callback.io_completed(res)). The purge method writes the last-purged log id as metadata (meta::LastPurged) and deletes the key range in one atomic range delete.

Both stores expose a get_log_state method that returns the last_log_id and the last_purged_log_id, which the Raft core uses to decide when to trigger snapshots or log compaction.

State Machine and Snapshot Lifecycle

Snapshots are triggered by Config::snapshot_policy (e.g., LogsSinceLast(N)) when the committed index advances N entries beyond the last snapshot. The RaftSnapshotBuilder implementation in MemStateMachine serializes the entire state machine via serde_json::to_vec, reporting the payload size for observability. The snapshot_max_chunk_size parameter controls how this payload is fragmented for network transmission, while max_in_snapshot_log_to_keep determines how many logs persist after snapshot installation to balance recovery speed against disk usage.

Key Configuration Parameters for Storage Performance

All tunables reside in openraft/src/config/config.rs within the Config struct. Adjusting these values allows you to optimize the trade-off between write latency, replication throughput, and snapshot frequency.

Batching and Network Tuning

max_append_entries (Option<usize>, default 4096) sets the upper bound on entries written in a single I/O operation. Increasing this value to Some(8192) or higher reduces disk sync overhead on fast SSDs but increases latency for individual entries.

max_payload_entries (usize, default 300) controls how many entries a leader ships in a single AppendEntries RPC. Raising this to 500–1000 reduces round-trips on reliable, high-bandwidth networks.

Snapshot Management

replication_lag_threshold (u64, default 10000) defines the log gap that triggers snapshot-based catch-up. Lowering this value forces snapshots earlier, which is beneficial when logs grow large, but increases snapshot build frequency.

snapshot_policy (SnapshotPolicy, default LogsSinceLast(5000)) determines when to trigger automatic snapshots. Setting this to LogsSinceLast(2000) for high-write workloads prevents unbounded log growth.

snapshot_max_chunk_size (u64, default 1MB) limits the byte size of snapshot chunks transmitted to followers. Increasing this to 8MB or 16MB reduces RPC count for large state machines.

Log Retention and Purging

max_in_snapshot_log_to_keep (u64, default 1000) specifies how many logs to retain after a snapshot. Smaller values (e.g., 500) speed up purging and reduce disk usage, but may increase recovery time after a crash.

purge_batch_size (u64, default 256) sets the minimum number of applied logs removed in one batch. Larger batches (e.g., 512 or 1024) reduce the overhead of frequent purge operations.

Persistence Optimization

enable_saving_committed (memstore only, default true) controls whether MemLogStore persists the committed index. Disabling this via log_store.enable_saving_committed.store(false, std::sync::atomic::Ordering::Relaxed) eliminates a write per commit, useful only for pure in-memory benchmarks where durability is not required.

Production Tuning Examples

Optimizing RocksDB for SSD Workloads

For SSD-backed deployments using RocksLogStore, combine OpenRaft configuration with aggressive batching:

use openraft::{Config, SnapshotPolicy};
use std::sync::Arc;

let mut cfg = Config::default();
cfg.max_append_entries = Some(8192);          // Large batches for SSD
cfg.max_payload_entries = 1_000;             // Reduce network round-trips
cfg.replication_lag_threshold = 5_000;        // Early snapshot trigger
cfg.snapshot_policy = SnapshotPolicy::LogsSinceLast(2_000);
cfg.snapshot_max_chunk_size = 8 * 1024 * 1024; // 8 MiB chunks
cfg.max_in_snapshot_log_to_keep = 500;
cfg.purge_batch_size = 200;

let cfg = Arc::new(cfg.validate()?);
// Initialize RocksDB log_store...

This configuration minimizes disk sync frequency while ensuring snapshots prevent unbounded log growth.

High-Performance In-Memory Benchmarking

When measuring raw Raft overhead without persistence costs:

use openraft::{Config, SnapshotPolicy};
use std::sync::Arc;

let mut cfg = Config::default();
cfg.max_append_entries = Some(4_096);
cfg.max_payload_entries = 300;
cfg.snapshot_policy = SnapshotPolicy::Never; // Prevent automatic snapshots
let cfg = Arc::new(cfg.validate()?);

let (log_store, state_machine) = openraft::stores::memstore::new_mem_store();
// Disable committed index persistence for pure speed
log_store.enable_saving_committed.store(false, std::sync::atomic::Ordering::Relaxed);

Disabling enable_saving_committed removes the final write barrier per commit, yielding lower latency in transient benchmarks.

Simulating Slow Storage with BlockConfig

The test harness provides BlockConfig to inject artificial latency:

use openraft::stores::memstore::{BlockConfig, BlockOperation};
use std::time::Duration;

let block = BlockConfig::default();
block.set_blocking(BlockOperation::BuildSnapshot, Duration::from_millis(200));
block.set_blocking(BlockOperation::PurgeLog, Duration::from_millis(50));

let (log_store, state_machine) = 
    openraft::stores::memstore::new_mem_store_with_block(block);

This simulates slow disks to validate Raft behavior under I/O pressure without hardware changes.

Critical Source Files and Implementation Details

Path	Purpose	Key Components
openraft/src/config/config.rs	Runtime configuration	Config struct, max_append_entries, snapshot_policy
stores/memstore/src/lib.rs	In-memory storage reference	MemLogStore, MemStateMachine, BlockConfig
examples/rocksstore/src/log_store.rs	Production RocksDB backend	RocksLogStore, WAL flushing, range deletes
benchmarks/minimal/src/store.rs	Benchmark harness store	Minimal in-memory implementation for throughput tests
tests/tests/replication/t62_follower_clear_restart_recover.rs	Recovery testing	Demonstrates enable_saving_committed behavior
tests/tests/state_machine/t10_total_order_apply.rs	Ordering guarantees	Validates state machine apply order under load

Best Practices for Production Deployments

• Start with conservative defaults – The default max_append_entries = 4096 and max_payload_entries = 300 suit most SSD and Ethernet environments.

• Measure commit latency first – If latency spikes occur under load, reduce max_append_entries to avoid long write stalls rather than increasing batch sizes further.

• Balance snapshot frequency – Set snapshot_policy = LogsSinceLast(N) where N reflects your write rate; pair with replication_lag_threshold to prevent excessive snapshot builds for minor follower lag.

• Minimize log retention – Use max_in_snapshot_log_to_keep = 500 (or lower) to reduce disk usage, accepting slightly longer recovery times after crashes.

• Disable persistence only for benchmarks – Keep enable_saving_committed = true in production; disable it only in MemLogStore for transient performance testing.

• Tune RocksDB independently – OpenRaft issues simple put/delete operations; optimize RocksDB's write buffer, compression, and compaction settings to match your hardware for dominant performance gains.

• Monitor with tracing – Enable tracing subscribers to observe log compaction complete events and snapshot build times, adjusting configuration iteratively based on real-time metrics.

Summary

OpenRaft storage performance tuning revolves around configuring batch sizes, snapshot policies, and persistence settings in openraft/src/config/config.rs to align with your hardware capabilities. Key adjustments include increasing max_append_entries for SSD throughput, tuning snapshot_policy and replication_lag_threshold to balance log growth against snapshot overhead, and managing retention via max_in_snapshot_log_to_keep and purge_batch_size. For production RocksDB deployments, these parameters work alongside database-level optimization, while in-memory benchmarks can disable enable_saving_committed to eliminate persistence overhead.

Frequently Asked Questions

What is the optimal max_append_entries value for SSD storage?

For SSD-backed storage, setting max_append_entries to Some(8192) or higher in openraft/src/config/config.rs maximizes throughput by amortizing disk sync costs across many entries. However, if you observe increased commit latency or write stalls, reduce this value to Some(4096) or Some(2048) to prevent long-running I/O operations from blocking the Raft core. The optimal value depends on your specific SSD's IOPS capabilities and the latency requirements of your application.

How does snapshot_policy affect cluster performance?

The snapshot_policy parameter, typically configured as SnapshotPolicy::LogsSinceLast(N) in Config::snapshot_policy, controls how frequently the state machine is serialized to disk. Aggressive policies with low N values prevent unbounded log growth and reduce recovery time for lagging followers, but consume CPU and I/O resources for frequent snapshot builds. Conservative policies reduce snapshot overhead but may cause excessive network traffic when synchronizing followers that have fallen far behind, requiring careful tuning alongside replication_lag_threshold.

When should I disable enable_saving_committed?

You should only disable enable_saving_committed when using MemLogStore for pure in-memory benchmarking where durability is not required and you need to measure minimal commit latency. This is accomplished by calling log_store.enable_saving_committed.store(false, std::sync::atomic::Ordering::Relaxed) as shown in stores/memstore/src/lib.rs. Never disable this in production deployments or when using persistent storage like RocksDB, as it risks losing committed log positions during process restarts and violates Raft safety guarantees.

How do I tune OpenRaft for high-throughput benchmarking?

For high-throughput benchmarking, configure aggressive batching and disable automatic snapshots to measure raw replication overhead. Set max_append_entries to Some(4096) and max_payload_entries to 300 or higher, and use SnapshotPolicy::Never to prevent automatic snapshot interruptions. Initialize MemLogStore and disable enable_saving_committed to eliminate persistence overhead. Additionally, ensure enable_heartbeat and enable_elect are configured appropriately for your test topology (often disabled for single-node benchmarks) to isolate the storage layer performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}