How GreptimeDB WAL Ensures Data Durability: A Deep Dive into the Write-Ahead Log Implementation

GreptimeDB uses a Write-Ahead Log (WAL) to guarantee that every mutation is persisted to durable storage before becoming visible to queries, implemented as a pluggable wrapper around RaftEngine files or Kafka topics with atomic batch persistence and recovery capabilities.

GreptimeDB's storage engine relies on the Write-Ahead Log (WAL) as the foundation of its data durability guarantees. Before any data modification becomes queryable, the system serializes the mutation and persists it through a pluggable log store interface. This architecture ensures that even in the event of a crash, committed transactions can be recovered by replaying the durable log entries.

Atomic Entry Creation and Protobuf Serialization

The durability pipeline begins with atomic entry creation in src/mito2/src/wal.rs. Each mutation is encapsulated in a WalEntry struct and serialized using Protocol Buffers via wal_entry.encode_to_vec(). The resulting byte vector is wrapped in a store_api::logstore::Entry object that carries critical metadata including the region ID, a monotonically increasing entry ID, and the selected WAL provider identifier.

let data = wal_entry.encode_to_vec();
let entry = self.store.entry(data, entry_id, region_id, provider)
    .context(BuildEntrySnafu { region_id })?;

This encoding happens in the Wal::writer() method implementation at lines 87-93, ensuring that every entry is immutable and self-contained before reaching the storage layer. The monotonic entry ID assignment guarantees ordering, which is essential for deterministic recovery during region startup.

Batch Persistence for Durable Storage

Individual entries are accumulated in a WalWriter buffer and flushed atomically using append_batch. This batching strategy amortizes the cost of durable writes while maintaining atomicity semantics. When write_to_wal() is invoked, the underlying log store guarantees that the entire batch is fsynced to local disk (for RaftEngine) or acknowledged by a quorum of Kafka brokers before the future resolves.

let entries = mem::take(&mut self.entries);
self.store.append_batch(entries).await
    .context(WriteWalSnafu)?;

As implemented in src/mito2/src/wal.rs (lines 99-108), this batch commit ensures that partial writes cannot occur—either all entries in the batch become durable, or none do. This all-or-nothing behavior is critical for maintaining consistency between the WAL and the region's memtables.

Obsolete Entry Handling and Log Cleanup

To prevent unbounded log growth while preserving recovery capabilities, GreptimeDB implements a log cleanup mechanism through the Wal::obsolete method. After data compaction moves mutations to immutable files, older WAL entries are marked for deletion. The log store's obsolete implementation truncates or removes persisted data up to the specified entry ID.

self.store.obsolete(provider, region_id, last_id).await
    .context(DeleteWalSnafu { region_id })?;

This cleanup logic, found at lines 151-158 of src/mito2/src/wal.rs, ensures that only entries not yet flushed to permanent storage are retained. The background pruning procedure in src/meta-srv/src/procedure/wal_prune.rs periodically invokes this mechanism to reclaim disk space without compromising durability for active regions.

Provider-Agnostic Durability Architecture

GreptimeDB's WAL abstraction supports multiple storage backends selected per-region via WalOptions. The WalProvider::alloc_batch method in src/common/meta/src/wal_provider.rs (lines 76-94) allocates the appropriate provider and encodes it as JSON metadata during region creation.

RaftEngine Provider uses local file-based storage where writes are persisted to disk with explicit fsync calls. The append_batch operation ensures data reaches physical storage media before returning, providing single-node durability against process crashes.

Kafka Provider produces entries to a replicated topic for distributed deployments. Durability is achieved through Kafka's acks=all configuration, requiring acknowledgment from a quorum of brokers before the write is considered complete. This shifts durability responsibility to the Kafka cluster's replication protocol.

Noop Provider skips persistence entirely and is restricted to testing scenarios where durability is intentionally disabled.

Recovery and State Replay Mechanism

During region startup, GreptimeDB reconstructs the Wal instance and invokes Wal::scan to read all entries from the last committed entry ID. Because the WAL guarantees that mutations are durable before application to in-memory state, replaying these entries restores the exact pre-crash state.

The scan implementation at lines 31-42 of src/mito2/src/wal.rs returns a stream of (EntryId, WalEntry) tuples that the region applies sequentially. This deterministic replay ensures that even if the node crashed during an append_batch operation, the recovered state reflects only fully persisted entries.

Practical Code Examples

Writing entries to a region using the RaftEngine provider:

let wal = /* obtain Wal<RaftEngineLogStore> */;
let region_id = RegionId::new(1, 42);
let mut writer = wal.writer();

let entry = WalEntry {
    mutations: vec![/* … */],
    bulk_entries: vec![],
};

writer
    .add_entry(
        region_id,
        1,                               // entry id
        &entry,
        &Provider::raft_engine_provider(region_id.as_u64()),
    )
    .unwrap();

writer.write_to_wal().await.unwrap();   // persisted atomically

Scanning entries for recovery after restart:

let wal = /* re‑create Wal */;
let provider = Provider::raft_engine_provider(region_id.as_u64());

// Scan from entry id 1
let stream = wal.scan(region_id, 1, &provider).unwrap();
let entries: Vec<(EntryId, WalEntry)> = stream.try_collect().await.unwrap();

Marking obsolete entries for cleanup:

wal.obsolete(region_id, /* last valid id */ 5, &provider).await.unwrap();

Summary

• Atomic serialization: Every mutation is encoded as an immutable WalEntry with monotonic IDs before reaching storage.
• Batch durability: The append_batch operation guarantees atomic persistence to either local disk (RaftEngine) or distributed brokers (Kafka).
• Pluggable architecture: Region-level WalOptions determine whether to use local files, remote Kafka topics, or skip WAL for testing.
• Automatic cleanup: The obsolete mechanism truncates logs after compaction prevents unbounded growth while preserving recoverability.
• Deterministic recovery: Startup scans replay durable entries from the last checkpoint, ensuring exactly-once state reconstruction.

Frequently Asked Questions

How does GreptimeDB ensure data is not lost during a power failure?

GreptimeDB relies on the underlying log store's fsync guarantees. When using RaftEngine, the append_batch call issues filesystem-level sync commands before acknowledging the write. For Kafka deployments, the system waits for acknowledgment from a quorum of brokers, ensuring the data exists on multiple physical machines before proceeding.

Can GreptimeDB run without the WAL for better performance?

Yes, but only through the Noop provider which is restricted to testing environments. This configuration provides no durability guarantees—data loss is guaranteed on process restart. Production deployments must use either RaftEngine or Kafka providers.

What happens if Kafka is unavailable in a remote WAL configuration?

Write operations will block or fail depending on the configured timeout and retry policy. Since the WAL must confirm durability before making data visible, the system prioritizes consistency over availability when the log store is unreachable. The region will reject new writes until Kafka connectivity is restored or the cluster is reconfigured.

How does GreptimeDB prevent the WAL from consuming unlimited disk space?

The Wal::obsolete method truncates logs after data is compacted into immutable files. A background procedure in src/meta-srv/src/procedure/wal_prune.rs periodically identifies regions where the WAL has been fully persisted to storage and invokes the obsolete operation, allowing the log store to reclaim physical storage space.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}