# Best Practices for Snapshotting and Log Compaction in Dragonboat: A Complete Guide

> Master Dragonboat snapshotting and log compaction with this guide. Learn to prevent storage growth and maintain fast replication. Explore configuration options for efficient storage management.

- Repository: [lni/dragonboat](https://github.com/lni/dragonboat)
- Tags: best-practices
- Published: 2026-03-06

---

**Dragonboat uses automatic snapshotting and log compaction to prevent unbounded storage growth while maintaining fast replication, configurable through `SnapshotEntries`, `CompactionOverhead`, and manual `RequestSnapshot` or `RequestCompaction` calls.**

Dragonboat, the high-performance Raft consensus library written in Go (`lni/dragonboat`), relies on two core mechanisms for storage management: **snapshotting** and **log compaction**. Properly configuring these features ensures that your Raft log does not grow indefinitely while keeping recovery and replication costs low. This guide explains the internal architecture of these systems and provides production-ready best practices based on the actual source code implementation.

## How Snapshotting Works in Dragonboat

Dragonboat creates snapshots to capture the current state of your state machine, allowing nodes to recover without replaying the entire Raft log.

### Automatic and Manual Snapshot Triggers

Snapshots are triggered in two ways. First, automatically every *N* applied entries, controlled by the `Config.SnapshotEntries` field defined in [`config/config.go`](https://github.com/lni/dragonboat/blob/main/config/config.go). Second, manually via the `NodeHost.RequestSnapshot` method in [`nodehost.go`](https://github.com/lni/dragonboat/blob/main/nodehost.go). When requesting a manual snapshot, you can pass a `SnapshotOption` struct, which is validated by `SnapshotOption.Validate` to ensure parameters like `CompactionOverhead` and `CompactionIndex` are legal ([nodehost.go lines 21-35](https://github.com/lni/dragonboat/blob/master/nodehost.go#L21-L35)).

### The Snapshot Generation Pipeline

The actual snapshot creation is handled by the `snapshotter` component in [`snapshotter.go`](https://github.com/lni/dragonboat/blob/main/snapshotter.go). The process follows this pipeline:

1.  **Write**: State machine data is written to a temporary directory.
2.  **Compress**: The data is compressed based on `SnapshotCompressionType` settings.
3.  **Commit**: Metadata is committed to the LogDB via `snapshotter.Save`.

The full implementation of the save pipeline is found in `snapshotter.Save` ([snapshotter.go lines 104-146](https://github.com/lni/dragonboat/blob/master/snapshotter.go#L104-L146)).

### Metadata Registration

After a successful write, the snapshot is registered in the LogDB through `logdb.SaveSnapshots`. This metadata allows the system to track which snapshots are available for recovery and compaction.

## How Log Compaction Works in Dragonboat

Log compaction removes obsolete Raft log entries that have been superseded by a snapshot, reclaiming disk space.

### Automatic Compaction Triggers

When a snapshot is successfully persisted and `Config.DisableAutoCompactions` is false (the default), the node automatically calls `node.requestCompaction()` in [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go). This method obtains the *compact-to* index from the `snapshotState` and invokes `LogDB.CompactEntriesTo` ([node.go lines 72-86](https://github.com/lni/dragonboat/blob/master/node.go#L72-L86)).

### The Compaction Flow

The compaction process involves two main operations:

1.  **Log Entry Removal**: `LogDB.RemoveEntriesTo` deletes Raft log entries up to the *compact-to* index.
2.  **Snapshot Directory Cleanup**: `snapshotter.Compact` removes stale snapshot directories older than the target index. This is implemented in `snapshotter.Compact` ([snapshotter.go lines 221-236](https://github.com/lni/dragonboat/blob/master/snapshotter.go#L221-L236)).

### Manual Compaction Requests

If you disable automatic compactions to control I/O bursts, you must manually trigger compaction using `NodeHost.RequestCompaction`. This method forwards the request to the node's internal `requestCompaction` method ([nodehost.go lines 79-94](https://github.com/lni/dragonboat/blob/master/nodehost.go#L79-L94)).

## Configuration Knobs for Storage Management

The `config.Config` struct in [`config/config.go`](https://github.com/lni/dragonboat/blob/main/config/config.go) ([lines 99-136](https://github.com/lni/dragonboat/blob/master/config/config.go#L99-L136)) defines the primary fields for controlling snapshotting and log compaction:

| Field | Purpose | Typical Values |
|-------|---------|----------------|
| `SnapshotEntries` | Frequency of automatic snapshots (entries per snapshot) | `0` disables auto-snapshot; otherwise `50000` to `1000000` depending on workload |
| `CompactionOverhead` | Number of recent log entries kept after compaction | `0` compacts up to snapshot index; `500` keeps recent entries for fast follower catch-up |
| `DisableAutoCompactions` | Disables automatic compaction after snapshots | `false` for normal operation; `true` only for explicit I/O control |
| `SnapshotOption.OverrideCompactionOverhead` | Per-snapshot override of compaction behavior | Use for critical snapshots requiring custom history retention |

## Snapshot State Tracking

The `snapshotState` struct in [`snapshotstate.go`](https://github.com/lni/dragonboat/blob/main/snapshotstate.go) manages the indices required for compaction and ensures only one snapshot operation runs concurrently. Key fields include `compactLogTo` and `compactedTo`, which track the target and completed compaction indices respectively ([snapshotstate.go lines 65-70](https://github.com/lni/dragonboat/blob/master/snapshotstate.go#L65-L70)).

Access to these fields is synchronized via atomic helpers such as `hasCompactLogTo`, `getCompactLogTo`, and `setCompactedTo` ([snapshotstate.go lines 133-151](https://github.com/lni/dragonboat/blob/master/snapshotstate.go#L133-L151)).

## Best Practices for Production Workloads

Follow these guidelines to optimize storage management in Dragonboat:

- **Set a sensible `SnapshotEntries` value** – Choose a value that balances snapshot size versus frequency. Very large numbers cause the Raft log to grow unbounded; very small numbers increase snapshot I/O overhead.

- **Tune `CompactionOverhead`** – Keep a modest overhead (e.g., 200–1000 entries) so followers can catch up by log replication rather than downloading an entire snapshot, reducing network traffic.

- **Leave auto-compaction enabled unless necessary** – The node automatically compacts the log right after a snapshot, reclaiming disk space promptly. Disabling it adds management burden.

- **When disabling auto-compaction, always call `RequestCompaction`** – Otherwise the log will never be trimmed and storage will explode.

- **Use `SnapshotOption.OverrideCompactionOverhead` for outlier snapshots** – For critical snapshots you may want to keep more recent entries (or none) – supply a custom `SnapshotOption` to `RequestSnapshot`.

- **Monitor snapshot and compaction events** – Dragonboat emits `SnapshotCompacted`, `LogCompacted`, and `LogDBCompacted` events (see `raftio.Listener`). Hook into these for observability or alerting.

- **Periodically clean up old snapshots** – Even after log compaction, old snapshot directories remain. Use `snapshotter.Remove` (called internally) or invoke `snapshotter.Compact` with the newest index to purge older snapshots.

- **Avoid overlapping snapshot requests** – The `snapshotState` logic ensures only one snapshot is in flight; if you receive `ErrSystemBusy` from `RequestSnapshot`, back-off and retry.

- **Test with realistic payload sizes** – Compression (`SnapshotCompressionType`) can affect snapshot size and compaction speed; experiment with `Snappy` versus `NoCompression` in a staging environment.

## Code Examples

### Configuring NodeHost for Balanced Storage

```go
cfg := config.NodeHostConfig{
    // … other required fields …
    RaftAddress:   "127.0.0.1:63001",
    LogDBFactory:  logdb.New,
    SnapshotEntries: 50000,          // create a snapshot every 50 k entries
    CompactionOverhead: 500,        // keep the last 500 entries after compaction
    // DisableAutoCompactions: false // default – keep auto‑compaction
}
nh, err := dragonboat.NewNodeHost(cfg)
if err != nil { /* handle error */ }

```

*Links:* `SnapshotEntries` and `CompactionOverhead` fields are defined in [`config/config.go`](https://github.com/lni/dragonboat/blob/main/config/config.go) ([lines 99-136](https://github.com/lni/dragonboat/blob/master/config/config.go#L99-L136)).

### Requesting Manual Snapshots with Custom Compaction Overhead

```go
opt := dragonboat.SnapshotOption{
    OverrideCompactionOverhead: true,
    CompactionOverhead:         0,           // compact completely up to snapshot index
}
rs, err := nh.RequestSnapshot(shardID, opt, 5*time.Second)
if err != nil { /* handle error */ }
// wait for completion
result := <-rs.ResultC()
fmt.Printf("snapshot created at index %d\n", result.Value)

```

*Link:* `SnapshotOption.Validate` ensures the override is legal ([nodehost.go lines 21-35](https://github.com/lni/dragonboat/blob/master/nodehost.go#L21-L35)).

### Triggering Manual Log Compaction

```go
// Assuming you previously disabled auto compaction in the config:
_, err = nh.RequestCompaction(shardID, replicaID)
if err != nil && err != dragonboat.ErrRejected {
    // ErrRejected means there was nothing to compact
    log.Fatalf("compaction failed: %v", err)
}

```

*Link:* `NodeHost.RequestCompaction` forwards the request to the node's internal `requestCompaction` method ([nodehost.go lines 79-94](https://github.com/lni/dragonboat/blob/master/nodehost.go#L79-L94)).

### Observing Compaction Events

```go
type myListener struct{ dragonboat.SysEventListener }
func (l *myListener) LogCompacted(info raftio.EntryInfo) {
    fmt.Printf("log compacted up to %d\n", info.Index)
}
nh.AddEventListener(&myListener{})

```

*Link:* The listener interface is defined in [`raftio/listener.go`](https://github.com/lni/dragonboat/blob/main/raftio/listener.go) ([listener definitions](https://github.com/lni/dragonboat/blob/master/raftio/listener.go#L73-L75)).

## Key Source Files Reference

| File | Role | Direct Link |
|------|------|-------------|
| [`config/config.go`](https://github.com/lni/dragonboat/blob/main/config/config.go) | Holds global configuration (`SnapshotEntries`, `CompactionOverhead`, `DisableAutoCompactions`) | <https://github.com/lni/dragonboat/blob/master/config/config.go> |
| [`snapshotstate.go`](https://github.com/lni/dragonboat/blob/main/snapshotstate.go) | Tracks snapshot indices, compaction requests, and flags (`compactLogTo`, `compactedTo`) | <https://github.com/lni/dragonboat/blob/master/snapshotstate.go> |
| [`snapshotter.go`](https://github.com/lni/dragonboat/blob/main/snapshotter.go) | Implements snapshot creation, saving, loading, and compacting of old snapshot files | <https://github.com/lni/dragonboat/blob/master/snapshotter.go> |
| [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go) | Core Raft node logic – applies updates, triggers snapshotting, and performs log compaction (`requestCompaction`) | <https://github.com/lni/dragonboat/blob/master/node.go> |
| [`nodehost.go`](https://github.com/lni/dragonboat/blob/main/nodehost.go) | Public API for snapshot & compaction requests (`RequestSnapshot`, `RequestCompaction`) | <https://github.com/lni/dragonboat/blob/master/nodehost.go> |
| [`raftio/logdb.go`](https://github.com/lni/dragonboat/blob/main/raftio/logdb.go) | Interface to the underlying log storage; defines `CompactEntriesTo` and `RemoveEntriesTo` | <https://github.com/lni/dragonboat/blob/master/raftio/logdb.go> |
| [`raftio/listener.go`](https://github.com/lni/dragonboat/blob/main/raftio/listener.go) | System‑event listener hooks for snapshot/compaction notifications | <https://github.com/lni/dragonboat/blob/master/raftio/listener.go> |

## Summary

- **Snapshotting and log compaction in Dragonboat** work together to prevent unbounded storage growth while maintaining replication efficiency.
- Configure `SnapshotEntries` to balance snapshot frequency against log growth, typically between 50,000 and 1,000,000 entries depending on workload.
- Set `CompactionOverhead` to retain recent entries (e.g., 500-1000) so followers can catch up via log replication rather than full snapshot transfers.
- Keep `DisableAutoCompactions` set to `false` unless you need explicit I/O control, in which case you must manually invoke `RequestCompaction`.
- Use `SnapshotOption.OverrideCompactionOverhead` for specific snapshots that require custom retention policies.
- Monitor system health by implementing the `raftio.Listener` interface to receive `LogCompacted` and `SnapshotCompacted` events.

## Frequently Asked Questions

### What is the difference between snapshotting and log compaction in Dragonboat?

**Snapshotting** captures the current state of your state machine and saves it to disk via the `snapshotter.Save` method in [`snapshotter.go`](https://github.com/lni/dragonboat/blob/main/snapshotter.go), creating a point-in-time image that can be used for recovery. **Log compaction** removes obsolete Raft log entries that are already covered by a snapshot through `LogDB.RemoveEntriesTo` and `snapshotter.Compact`, reclaiming disk space. While snapshotting creates recovery points, compaction cleans up the log entries that preceded those points.

### How often should I configure SnapshotEntries for optimal performance?

The ideal value for `SnapshotEntries` in [`config/config.go`](https://github.com/lni/dragonboat/blob/main/config/config.go) depends on your write throughput and state machine size. For most workloads, values between **50,000 and 1,000,000** entries work well. Setting it too high (e.g., millions) causes the Raft log to grow large, increasing recovery time and disk usage. Setting it too low (e.g., thousands) generates excessive I/O from frequent snapshot creation. Test with realistic payload sizes using `Snappy` or `NoCompression` to find the balance for your specific hardware.

### Can I disable automatic log compaction in Dragonboat?

Yes, you can disable automatic compaction by setting `DisableAutoCompactions` to `true` in your `config.Config`. This is useful when you need explicit control over I/O bursts or want to coordinate compaction with maintenance windows. However, **you must manually call `NodeHost.RequestCompaction`** periodically if you disable auto-compaction; otherwise, the log will never be trimmed and storage will grow unbounded. Always check for `dragonboat.ErrRejected` when calling this method, as it indicates there was nothing to compact.

### How do I monitor snapshot and compaction activity in production?

Dragonboat emits system events through the `raftio.Listener` interface defined in [`raftio/listener.go`](https://github.com/lni/dragonboat/blob/main/raftio/listener.go). Implement this interface to receive notifications when compaction occurs:

- **`LogCompacted`** fires when entries are removed from the log.
- **`SnapshotCompacted`** fires when old snapshot files are cleaned up.
- **`LogDBCompacted`** provides low-level storage compaction events.

Attach your listener using `NodeHost.AddEventListener()`. Additionally, monitor for `ErrSystemBusy` when calling `RequestSnapshot`, as the `snapshotState` logic in [`snapshotstate.go`](https://github.com/lni/dragonboat/blob/main/snapshotstate.go) ensures only one snapshot runs at a time, and busy errors indicate you need to back off and retry.