# How the Sled Database Powers r-nacos: Embedded Storage Architecture and Data Organization

> Discover how the sled database powers r-nacos with its embedded storage architecture. Learn about efficient local data organization for Raft state, config history, and more.

- Repository: [Nacos Group/r-nacos](https://github.com/nacos-group/r-nacos)
- Tags: internals
- Published: 2026-03-07

---

**The sled database serves as the embedded, ordered key-value engine in r-nacos, persisting all durable data—including Raft state, configuration history, and user tables—to local disk without requiring an external database server.**

r-nacos is a Rust implementation of the Nacos service discovery and configuration management platform. Unlike traditional deployments that rely on external MySQL or PostgreSQL instances, r-nacos uses the **sled database** as its single source of truth for persistent storage, embedding durability directly into the binary through a multi-tree architecture.

## Role of the Sled Database in r-nacos

sled is an embedded, lock-free key-value store written in Rust that provides ordered map semantics. In r-nacos, it fulfills the role of the **primary persistence layer**, ensuring that every piece of state required to survive a restart is stored durably on local disk.

All data resides in a directory controlled by the `RNACOS_DATA_DIR` environment variable, defaulting to `/var/lib/r-nacos/data`. When the server initializes in [`src/main.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/main.rs), it opens a single `sled::Db` instance at this path and injects it as an `Arc<sled::Db>` into the actor system. This design eliminates network dependencies and simplifies deployment to a single binary.

## Multi-Tree Data Organization

r-nacos organizes data using sled’s **multi-tree model**: a single database contains multiple independent `Tree` objects, each identified by a unique string name. This logical separation allows different subsystems to maintain isolated keyspaces while sharing the same underlying storage engine.

Values are stored as binary blobs, typically encoded using Protocol Buffers (protobuf) or raw byte arrays. Each tree serves a specific domain, such as Raft consensus, configuration management, or user data.

### Raft Persistence Layer (raft_store and raft_logs)

The consensus engine persists its state across two dedicated trees defined in [`src/raft/store/innerstore.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/raft/store/innerstore.rs).

The **`raft_store`** tree holds the hard state, membership configuration, and snapshot metadata. It uses fixed byte keys:
- `b"hs"` → Hard state (JSON-encoded)
- `b"membership"` → Current membership set
- `b"snapshot"` → Last snapshot info
- `b"last_applied_log"` → Last applied log index

The **`raft_logs`** tree stores the actual Raft log entries in an append-only fashion. Each entry is keyed by its log index (as bytes) and holds a protobuf-encoded `Entry<ClientRequest>`. The `append_entry_to_log` and `replicate_to_log` methods batch these writes for atomicity, while `delete_logs_from` handles log truncation.

### Configuration Storage (config and config_history)

The configuration subsystem, implemented in [`src/config/config_sled.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/config/config_sled.rs), uses a hybrid approach to store both current and historical data.

The **`config`** tree stores the latest version of every configuration. Keys are composite protobuf structures encoding tenant, group, and dataId, while values are protobuf-encoded `Config` objects.

For history tracking, r-nacos creates dynamic trees named **`config_history_<hash>`**, where `<hash>` is derived from the config key. Each history tree stores previous versions of a specific configuration, keyed by a monotonically increasing `u64` (big-endian encoded). The `TableSequence` helper generates these IDs, ensuring every config change receives a unique, ordered identifier.

### Table Management and Sequences

Beyond configuration, r-nacos supports generic key-value tables through the `TableManager` in [`src/raft/db/table.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/raft/db/table.rs). These tables use dynamically named trees (such as `t_user` or `user`) to store application data.

To coordinate distributed ID generation, the **`table_sequence`** tree acts as a global sequence generator. The `TableSequence` struct in [`src/common/sled_utils.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/common/sled_utils.rs) implements three distinct strategies:

1. **Simple cached batch** (`next_id`) – Reserves a block of IDs in the tree, then serves them from memory to reduce I/O (used by Config history).
2. **Compare-and-swap batch** (`next_id_by_compare`) – Uses atomic `compare_and_swap` operations to safely generate IDs across concurrent actors.
3. **Stateful batch** (`next_state`) – Returns both the next ID and the new high-water mark, designed for Raft snapshot integration where the sequence state must be persisted alongside the snapshot.

## Data Integrity and Snapshots

r-nacos guarantees consistency through atomic batch operations and comprehensive snapshotting. All mutations that span multiple keys use `sled::Batch` to ensure atomic writes. For example, log replication and history cleanup batch deletions and insertions into a single transaction.

When the Raft engine triggers a snapshot, `InnerStore::build_snapshot_data` iterates over every tree in the database—`config`, `config_history_*`, `table_sequence`, table definitions, and user tables—and serializes each entry into a `SnapshotItem`. These items are streamed through a `SnapshotWriterActor`, producing a complete, portable backup of the entire state.

During recovery, `finalize_snapshot_installation` reverses the process: it clears existing trees and repopulates them using `install_snapshot_data`, ensuring a restarted node reconstructs its exact state from the local sled files.

## Key Source Files and Implementation Details

The sled integration is centralized in these modules:

| File | Purpose |
|------|---------|
| [`src/main.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/main.rs) | Initializes the `sled::Db` and injects it into the actor system |
| [`src/common/sled_utils.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/common/sled_utils.rs) | `TableSequence` implementation for global ID generation |
| [`src/raft/store/innerstore.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/raft/store/innerstore.rs) | Raft persistence: hard-state, logs, and snapshot handling |
| [`src/config/config_sled.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/config/config_sled.rs) | Configuration storage and version history management |
| [`src/raft/db/table.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/raft/db/table.rs) | Generic table management and snapshot aggregation |

A minimal initialization example from the codebase:

```rust
// src/main.rs – Database initialization
let db = sled::Config::new()
    .path(data_dir)  // Controlled by RNACOS_DATA_DIR
    .open()?;

// Injection into the actor system
bean_factory.register(Arc::new(db).into());

```

## Summary

- **sled** acts as the embedded, ordered key-value engine that powers all persistent storage in r-nacos, eliminating external database dependencies.
- Data is organized into **multiple named trees** within a single sled database, separating concerns between Raft consensus, configuration, history, and user data.
- Critical trees include `raft_store` (hard-state), `raft_logs` (entries), `config` (current configs), `config_history_<hash>` (versions), and `table_sequence` (ID generation).
- **TableSequence** provides three ID generation strategies—simple cache, compare-and-swap, and stateful batches—ensuring monotonic identifiers across distributed components.
- **Atomic batches** and **comprehensive snapshots** guarantee consistency; the entire database state can be serialized via `build_snapshot_data` and restored via `install_snapshot_data`.

## Frequently Asked Questions

### How does r-nacos handle concurrent ID generation without external databases?

r-nacos uses the `TableSequence` helper in [`src/common/sled_utils.rs`](https://github.com/nacos-group/r-nacos/blob/main/src/common/sled_utils.rs) to generate monotonic IDs. It leverages sled's `compare_and_swap` primitive in the `next_id_by_compare` method to atomically reserve batches of IDs, allowing concurrent actors to safely generate unique identifiers without external coordination or lock contention.

### What happens if the r-nacos server restarts—how is state recovered?

Upon restart, r-nacos initializes the sled database from the directory specified by `RNACOS_DATA_DIR` (default `/var/lib/r-nacos/data`). The Raft engine replays any persisted logs from the `raft_logs` tree and loads the hard-state from `raft_store`. If a snapshot was previously taken, `finalize_snapshot_installation` restores all trees—including configuration, history, and user tables—to their exact state at the time of the snapshot.

### Can I inspect or migrate data from the sled storage manually?

While sled stores data in a binary format optimized for performance, the directory structure is accessible via the `sled::Db` API. Each logical domain resides in a separate tree (e.g., `config`, `raft_store`). For migrations, r-nacos provides snapshot functionality via `build_snapshot_data` which serializes the entire state into portable `SnapshotItem` objects, allowing complete backup and restore across nodes without manual byte-level manipulation.

### How does r-nacos ensure data consistency during writes?

r-nacos utilizes sled's transactional batch API (`sled::Batch`) to group multiple write operations into atomic units. For example, when replicating Raft logs via `replicate_to_log` or cleaning up history via `delete_hisotry_since`, all deletions and insertions are collected into a single batch and applied atomically. This prevents partial writes during crashes and maintains consistency across related trees.