How the Sled Database Powers r-nacos: Embedded Storage Architecture and Data Organization

The sled database serves as the embedded, ordered key-value engine in r-nacos, persisting all durable data—including Raft state, configuration history, and user tables—to local disk without requiring an external database server.

r-nacos is a Rust implementation of the Nacos service discovery and configuration management platform. Unlike traditional deployments that rely on external MySQL or PostgreSQL instances, r-nacos uses the sled database as its single source of truth for persistent storage, embedding durability directly into the binary through a multi-tree architecture.

Role of the Sled Database in r-nacos

sled is an embedded, lock-free key-value store written in Rust that provides ordered map semantics. In r-nacos, it fulfills the role of the primary persistence layer, ensuring that every piece of state required to survive a restart is stored durably on local disk.

All data resides in a directory controlled by the RNACOS_DATA_DIR environment variable, defaulting to /var/lib/r-nacos/data. When the server initializes in src/main.rs, it opens a single sled::Db instance at this path and injects it as an Arc<sled::Db> into the actor system. This design eliminates network dependencies and simplifies deployment to a single binary.

Multi-Tree Data Organization

r-nacos organizes data using sled’s multi-tree model: a single database contains multiple independent Tree objects, each identified by a unique string name. This logical separation allows different subsystems to maintain isolated keyspaces while sharing the same underlying storage engine.

Values are stored as binary blobs, typically encoded using Protocol Buffers (protobuf) or raw byte arrays. Each tree serves a specific domain, such as Raft consensus, configuration management, or user data.

Raft Persistence Layer (raft_store and raft_logs)

The consensus engine persists its state across two dedicated trees defined in src/raft/store/innerstore.rs.

The raft_store tree holds the hard state, membership configuration, and snapshot metadata. It uses fixed byte keys:

• b"hs" → Hard state (JSON-encoded)
• b"membership" → Current membership set
• b"snapshot" → Last snapshot info
• b"last_applied_log" → Last applied log index

The raft_logs tree stores the actual Raft log entries in an append-only fashion. Each entry is keyed by its log index (as bytes) and holds a protobuf-encoded Entry<ClientRequest>. The append_entry_to_log and replicate_to_log methods batch these writes for atomicity, while delete_logs_from handles log truncation.

Configuration Storage (config and config_history)

The configuration subsystem, implemented in src/config/config_sled.rs, uses a hybrid approach to store both current and historical data.

The config tree stores the latest version of every configuration. Keys are composite protobuf structures encoding tenant, group, and dataId, while values are protobuf-encoded Config objects.

For history tracking, r-nacos creates dynamic trees named config_history_<hash>, where <hash> is derived from the config key. Each history tree stores previous versions of a specific configuration, keyed by a monotonically increasing u64 (big-endian encoded). The TableSequence helper generates these IDs, ensuring every config change receives a unique, ordered identifier.

Table Management and Sequences

Beyond configuration, r-nacos supports generic key-value tables through the TableManager in src/raft/db/table.rs. These tables use dynamically named trees (such as t_user or user) to store application data.

To coordinate distributed ID generation, the table_sequence tree acts as a global sequence generator. The TableSequence struct in src/common/sled_utils.rs implements three distinct strategies:

1. Simple cached batch (next_id) – Reserves a block of IDs in the tree, then serves them from memory to reduce I/O (used by Config history).
2. Compare-and-swap batch (next_id_by_compare) – Uses atomic compare_and_swap operations to safely generate IDs across concurrent actors.
3. Stateful batch (next_state) – Returns both the next ID and the new high-water mark, designed for Raft snapshot integration where the sequence state must be persisted alongside the snapshot.

Data Integrity and Snapshots

r-nacos guarantees consistency through atomic batch operations and comprehensive snapshotting. All mutations that span multiple keys use sled::Batch to ensure atomic writes. For example, log replication and history cleanup batch deletions and insertions into a single transaction.

When the Raft engine triggers a snapshot, InnerStore::build_snapshot_data iterates over every tree in the database—config, config_history_*, table_sequence, table definitions, and user tables—and serializes each entry into a SnapshotItem. These items are streamed through a SnapshotWriterActor, producing a complete, portable backup of the entire state.

During recovery, finalize_snapshot_installation reverses the process: it clears existing trees and repopulates them using install_snapshot_data, ensuring a restarted node reconstructs its exact state from the local sled files.

Key Source Files and Implementation Details

The sled integration is centralized in these modules:

File	Purpose
src/main.rs	Initializes the sled::Db and injects it into the actor system
src/common/sled_utils.rs	TableSequence implementation for global ID generation
src/raft/store/innerstore.rs	Raft persistence: hard-state, logs, and snapshot handling
src/config/config_sled.rs	Configuration storage and version history management
src/raft/db/table.rs	Generic table management and snapshot aggregation

A minimal initialization example from the codebase:

// src/main.rs – Database initialization
let db = sled::Config::new()
    .path(data_dir)  // Controlled by RNACOS_DATA_DIR
    .open()?;

// Injection into the actor system
bean_factory.register(Arc::new(db).into());

Summary

• sled acts as the embedded, ordered key-value engine that powers all persistent storage in r-nacos, eliminating external database dependencies.
• Data is organized into multiple named trees within a single sled database, separating concerns between Raft consensus, configuration, history, and user data.
• Critical trees include raft_store (hard-state), raft_logs (entries), config (current configs), config_history_<hash> (versions), and table_sequence (ID generation).
• TableSequence provides three ID generation strategies—simple cache, compare-and-swap, and stateful batches—ensuring monotonic identifiers across distributed components.
• Atomic batches and comprehensive snapshots guarantee consistency; the entire database state can be serialized via build_snapshot_data and restored via install_snapshot_data.

Frequently Asked Questions

How does r-nacos handle concurrent ID generation without external databases?

r-nacos uses the TableSequence helper in src/common/sled_utils.rs to generate monotonic IDs. It leverages sled's compare_and_swap primitive in the next_id_by_compare method to atomically reserve batches of IDs, allowing concurrent actors to safely generate unique identifiers without external coordination or lock contention.

What happens if the r-nacos server restarts—how is state recovered?

Upon restart, r-nacos initializes the sled database from the directory specified by RNACOS_DATA_DIR (default /var/lib/r-nacos/data). The Raft engine replays any persisted logs from the raft_logs tree and loads the hard-state from raft_store. If a snapshot was previously taken, finalize_snapshot_installation restores all trees—including configuration, history, and user tables—to their exact state at the time of the snapshot.

Can I inspect or migrate data from the sled storage manually?

While sled stores data in a binary format optimized for performance, the directory structure is accessible via the sled::Db API. Each logical domain resides in a separate tree (e.g., config, raft_store). For migrations, r-nacos provides snapshot functionality via build_snapshot_data which serializes the entire state into portable SnapshotItem objects, allowing complete backup and restore across nodes without manual byte-level manipulation.

How does r-nacos ensure data consistency during writes?

r-nacos utilizes sled's transactional batch API (sled::Batch) to group multiple write operations into atomic units. For example, when replicating Raft logs via replicate_to_log or cleaning up history via delete_hisotry_since, all deletions and insertions are collected into a single batch and applied atomically. This prevents partial writes during crashes and maintains consistency across related trees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}