How r-nacos Handles Concurrent Write Requests and Ensures Data Consistency in Cluster Mode

r-nacos uses the Raft consensus algorithm via the async-raft-ext crate to serialize all write operations, ensuring linearizable consistency by requiring majority replication before committing configuration changes.

The r-nacos project is a high-performance Rust implementation of the Nacos service discovery and configuration management platform. When operating in cluster mode, the system must coordinate concurrent write requests across multiple nodes while maintaining strong data consistency. The architecture achieves this through a carefully implemented Raft consensus layer that transforms parallel client requests into a strictly ordered, replicated log.

Raft Consensus Foundation

At the core of r-nacos's cluster consistency model lies the Raft consensus algorithm, implemented through the async-raft-ext crate. This algorithm elects a single leader node that acts as the serialization point for all state-changing operations.

ClientRequest Enum Structure

All mutating requests are strictly typed through the ClientRequest enum defined in src/raft/store/mod.rs. This enum encapsulates every possible state change, ensuring type-safe serialization before Raft processing.

pub enum ClientRequest {
    ConfigSet { key, value, history_id, history_table_id, … },
    ConfigRemove { key },
    // … other request types
}

The ClientRequest enum variants carry all necessary metadata for configuration operations, including history tracking identifiers and operation timestamps.

Write Request Flow in Cluster Mode

The system provides multiple entry points for write operations, but all converge on the same Raft serialization pipeline.

HTTP and gRPC Entry Points

External clients interact with the cluster through HTTP or gRPC handlers defined in src/raft/network/management.rs. These handlers construct ClientWriteRequest objects and submit them to the Raft node.

app.raft.client_write(ClientWriteRequest::new(ClientRequest::NodeAddr {
    id: node_id,
    addr,
})).await?;

The client_write method returns a future that resolves only after the entry has been safely committed to a majority of nodes.

Configuration Actor Integration

For configuration data specifically, the ConfigActor in src/config/core.rs acts as the bridge between the application layer and the Raft cluster. When processing a ConfigAdd command, the actor builds a ConfigSet request and forwards it through the Raft infrastructure.

let req = ClientRequest::ConfigSet {
    key: key.build_key(),
    value,
    config_type,
    desc,
    history_id,
    history_table_id,
    op_time: now_millis_i64(),
    op_user,
};
Self::send_raft_request(&raft, req).await.ok();

The send_raft_request method (lines 532-540 in src/config/core.rs) handles the asynchronous submission to the Raft leader.

Serialization and Replication Mechanism

Raft guarantees consistency by transforming concurrent requests into a strictly ordered log that must be replicated across the cluster before application.

Leader-Based Log Replication

When the Raft leader receives a ClientWriteRequest, it appends the request as a new log entry. The entry is then replicated to follower nodes through the Raft protocol. The client_write future does not resolve until the entry achieves committed status, meaning it has been written to a majority of nodes in the cluster.

This mechanism ensures that even with concurrent write requests arriving from multiple clients simultaneously, the leader queues and appends each request to the log in a specific total order.

State Machine Application

Once a log entry becomes committed, the Raft state machine applies the change. The replicate_to_state_machine_range function in src/raft/store/innerstore.rs processes the EntryPayload::Normal variants and dispatches configuration commands.

ClientRequest::ConfigSet { key, value, history_id, history_table_id } => {
    let cmd = ConfigRaftCmd::ConfigAdd {
        key,
        value,
        history_id,
        history_table_id,
    };
    self.do_send_to_config(cmd);
}

The do_send_to_config method forwards the command to the ConfigActor, which updates both the in-memory cache and the persistent sled database. Because every node applies committed entries in the same order, all replicas converge to identical states deterministically.

Handling Concurrent Access

Multiple clients may issue writes concurrently, but Raft's architecture inherently prevents conflicts. The leader node queues incoming client_write futures and processes them sequentially, appending each to the log in strict order.

Follower nodes never accept write requests directly; they only apply entries received from the leader. This design eliminates write-write conflicts entirely, as the single leader acts as the natural bottleneck that serializes all operations.

Durability and Failure Recovery

The system implements multiple safeguards to ensure data survives node failures and network partitions.

Commit Guarantees and Majority Quorums

An operation is considered durable only after persisting on a majority of nodes. If the leader crashes before committing an entry, the uncommitted data is lost on that node but persists on the majority. A newly elected leader will replay the entry from the remaining logs to ensure consistency.

The ShutdownError enum in src/raft/store/mod.rs defines an UnsafeStorageError variant to guard against unsafe state changes during shutdown scenarios.

Snapshot and Log Compaction

To prevent unbounded log growth, r-nacos periodically creates snapshots through the do_log_compaction method in src/raft/store/innerstore.rs. These snapshots capture the full configuration state and are stored via src/raft/filestore/raftsnapshot.rs.

New nodes joining the cluster receive the latest snapshot rather than replaying the entire log history, ensuring they start with a consistent baseline even after long periods of write activity.

Summary

• Raft consensus via async-raft-ext serializes all write operations through a single elected leader
• ClientRequest enum in src/raft/store/mod.rs strictly types all mutating operations
• Majority replication ensures committed entries survive node failures
• Total ordering of the Raft log guarantees all nodes apply changes in identical sequence
• State machine application in src/raft/store/innerstore.rs dispatches commands to the configuration actor
• Snapshotting prevents log unbounded growth while maintaining consistency for new cluster members

Frequently Asked Questions

What happens if the Raft leader crashes during a write operation?

If the leader crashes before an entry is committed, the uncommitted entry is lost on that node but remains persisted on the majority of followers. Once a new leader is elected, it will detect the uncommitted entry in the followers' logs and replicate it to achieve commitment before processing new client requests.

Can follower nodes accept write requests directly?

No, follower nodes reject write requests and forward them to the current leader. This design ensures that all writes flow through a single serialization point, preventing divergent states and write-write conflicts across the cluster.

How does r-nacos handle log storage for Raft entries?

The system uses persistent append-only log storage implemented in src/raft/filestore/raftlog/mod.rs. Entries are written to disk before acknowledgment, ensuring durability. Periodic compaction creates snapshots stored via src/raft/filestore/raftsnapshot.rs to manage storage growth.

What consistency level does r-nacos provide for configuration data?

r-nacos provides linearizable consistency for configuration data. This means that once a write operation completes (the client_write future resolves), the change is guaranteed to be visible to all subsequent reads across the entire cluster, and the operation appears to have executed atomically at a single point in time.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}