How MiniDB Handles Snapshot and WAL Persistence in Kimi Code's Embedded JSON Document Store

MiniDB combines an append-only Write-Ahead Log (WAL) for durability with periodic index generation snapshots for fast recovery and compact storage.

MiniDB powers document persistence in Kimi Code, MoonshotAI's AI coding assistant. Understanding its snapshot and WAL persistence architecture is essential for developers building reliable embedded applications. This article examines the implementation in packages/minidb, tracing how writes become durable and how the system recovers from crashes.

Write-Ahead Log (WAL) Architecture

Every mutation in MiniDB—set, delete, or batch operations—flows through an append-only WAL before touching the in-memory store.

WAL Implementation and Configuration

The WAL lives in src/wal.ts and is exposed to the core database via this.wal in MiniDb. Writes follow this sequence:

1. MiniDb.writePath serializes the operation
2. The in-memory Store updates
3. WAL.append persists to disk

import { MiniDb } from '@moonshot-ai/minidb';

const db = await MiniDb.open({
  dir: './mydb',
  fsyncPolicy: 'everysec',  // flush WAL every second (default)
});

The fsyncPolicy controls durability trade-offs:

Policy	Behavior
always	Flush on every write (safest, slowest)
everysec	Flush once per second (default)
never	Let OS schedule flushes (fastest, riskiest)

WAL Growth and Triggering Snapshots

The WAL grows unbounded until a generation build threshold triggers compaction. In src/mini-db.ts (lines 1020–1026), maybeAutoGenerationBuild checks:

// Simplified from src/mini-db.ts
if (this.generationStale()) {
  // GEN_BUILD_WAL_DELTA_BYTES ≈ 4 MiB threshold exceeded
  this.maintenanceScheduler.schedule(() => this.buildGeneration('auto'));
}

Once the WAL delta exceeds 4 MiB, the background scheduler initiates snapshot creation.

Snapshot (Index Generation) Persistence

Snapshots in MiniDB are called index generations: complete, self-contained materializations of database state.

Generation Structure and Atomic Publishing

A generation is a directory (generations/g-NNNNNN/) containing:

• Binary key-value store image
• Secondary indexes
• DT-column indexes
• Full-text index artifacts

The build process in src/generation-builder.ts ensures crash safety through atomic directory operations:

1. Write to temporary directory: g-N.tmp-<random>
2. Rename to final name: g-NNNNNN
3. Update CURRENT symlink
4. Truncate WAL to checkpoint offset

This atomic publish pattern—documented in packages/minidb/AGENTS.md (lines 11–13)—means readers never see partial states, even if the process crashes mid-build.

Post-Snapshot WAL Truncation

After successful generation publish, src/mini-db.ts truncates the WAL to the checkpoint offset stored in the generation manifest. This keeps the WAL small and recovery fast.

If generation loading fails—corruption, format version mismatch—the system executes fallback full rebuild: re-indexing the entire store from scratch using only WAL replay.

Database Lifecycle and Recovery Flow

The src/lifecycle.ts module orchestrates open, close, and recovery through openMiniDb.

Opening a Database: Recovery Steps

// Called internally by MiniDb.open()
async function openMiniDb(options) {
  // 1. Create/lock database directory
  await acquireDirectoryLock(dir);
  
  // 2. Load latest generation if present
  const generation = await loadLatestGeneration();
  
  // 3. Replay WAL delta to catch up
  await catchUpWalAsync(wal, store, generation?.checkpointOffset);
  
  return new MiniDb(...);
}

This three-phase open ensures fast startup (via generation) plus durability (via WAL replay).

Read-Only Replica Catch-Up

Read-only instances create isolated scratch directories and replay the WAL without interfering with the writer. From src/mini-db.ts (lines 135–142):

const readonlyDb = await MiniDb.open({
  dir: './mydb',
  readOnly: true,  // Creates <dir>.ro-scratch/<pid>-<random>/
});

// Transparently includes all writes flushed to WAL
const doc = await readonlyDb.getAsync('user:123');

Manual Control and Debugging

For testing or operational checkpoints, trigger snapshots explicitly:

// Force immediate snapshot generation
await db.buildGeneration('manual');

// Check if generation is stale (exposed for monitoring)
const needsSnapshot = db.generationStale();

Summary

• WAL persistence in src/wal.ts provides append-only durability with configurable fsyncPolicy flushing
• Automatic snapshot triggering occurs when WAL delta exceeds GEN_BUILD_WAL_DELTA_BYTES (≈ 4 MiB), handled in src/mini-db.ts
• Atomic generation publishing via src/generation-builder.ts uses temp directories and symlink swapping for crash safety
• Recovery in src/lifecycle.ts loads the latest generation then replays WAL deltas for consistency
• Read-only replicas replay WAL into scratch directories without blocking the writer

Frequently Asked Questions

What happens if the WAL grows very large?

The maybeAutoGenerationBuild method detects when WAL size exceeds 4 MiB and schedules a background generation build. Once published, the WAL truncates to zero, preventing unbounded growth. If automatic builds are disabled, manual buildGeneration() calls or restart recovery will eventually compact the log.

How does MiniDB prevent data loss during crashes?

The fsyncPolicy determines durability guarantees. With everysec (default), at most one second of writes may be lost on OS crash; with always, every write is durable but slower. The atomic generation publish ensures snapshots are never partial—readers see either the old or new generation, never a corrupted intermediate.

Can multiple processes open the same MiniDB database?

Only one writer can hold the database lock at a time. Multiple read-only instances are supported: each creates an isolated *.ro-scratch/ directory (src/mini-db.ts lines 135–142) and replays WAL deltas independently without interfering with the writer or each other.

What causes the "fallback full rebuild" path?

Generation loading fails due to checksum mismatch, disk corruption, or an incompatible storage format version. In these cases, src/lifecycle.ts discards the unusable generation and rebuilds all indexes by replaying the complete WAL from offset zero—slower but guaranteed to produce a valid state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}