Turso Database Recovery Process: Automatic Crash Recovery After Power Failure

Turso automatically restores database consistency after a crash or power failure by replaying the Write-Ahead Log (WAL) and MVCC logical log, validating headers with checksums, and atomically back-filling pages before truncating the WAL.

The Turso database engine (tursodatabase/turso) implements a robust recovery process for Turso databases that ensures ACID durability even after sudden power loss or operating system crashes. Built on a SQLite-compatible storage layer, the system leverages a three-phase protocol that detects incomplete writes and reconstructs committed transaction states without manual intervention.

How Crash Recovery Works in Turso

When an operating system crashes or power is cut, Turso may be left with a partially-written Write-Ahead Log (WAL) and an out-of-date on-disk B-tree. The engine resolves this by replaying the WAL through the MVCC logical log the next time the database is opened. This recovery flow is built into the core storage layer and orchestrated by the OpenSharedWal::poll state machine defined in core/storage/wal.rs.

The Three Phases of Recovery

Turso's recovery protocol consists of tightly-coupled phases that execute sequentially when Database::open is invoked.

1. WAL Header Validation and Restart

The engine begins by reading the WAL header (WAL_HEADER_SIZE bytes) to verify the magic number, page size, and checksum. If the header is missing, corrupted, or indicates an interrupted checkpoint, the engine executes a restart sequence.

In core/storage/wal.rs, the prepare_wal_header function performs initial validation, while begin_restart rewrites the header to a clean state and clears stale read-marks. This follows the SQLite-compatible recovery protocol, ensuring the log is in a known good state before replay begins.

2. MVCC Logical Log Replay

Turso uses a separate MVCC logical log (an on-disk file storing MVCC frames) to replay operations committed to the WAL but not yet back-filled into the main database file. The recovery logic iterates over each frame from the last known max_frame to the current file size.

For every frame encountered, the engine:

• Verifies the CRC checksum to detect corruption
• Deserializes the payload
• Applies the page updates to the in-memory B-tree via LogicalLog::apply_frame
• Updates the running CRC state to resume subsequent writes

This logic resides in core/mvcc/persistent_storage/logical_log.rs within the recover and apply_frame functions.

3. Checkpoint and Cleanup

Once logical-log replay completes, Turso performs an implicit checkpoint. The engine back-fills newly-replayed pages into the database file and updates the WAL's max_frame and nbackfills counters.

If a TRUNCATE checkpoint was pending, Wal::truncate_wal (also in core/storage/wal.rs) shrinks the WAL to zero bytes. Critically, truncation occurs only after the database file has been synced to disk, preserving durability guarantees.

Step-by-Step Crash Recovery Walkthrough

The following sequence occurs when reopening a database after an unexpected shutdown:

1. Crash occurs – The process terminates abruptly, potentially leaving the WAL with a partially-written frame or unfinished checkpoint.
2. Database reopened – Database::open constructs a WAL implementation (SharedWalCoordination → InProcessWalCoordination or ShmWalCoordination for cross-process use).
3. Header preparation – prepare_wal_header validates the existing header or initializes a fresh one if uninitialized.
4. Poll execution – OpenSharedWal::poll runs iteratively:
   • Reads the WAL header and checks magic numbers and checksums
   • Invokes begin_restart if the header indicates a restart is required
   • Scans WAL frames, calling LogicalLog::apply_frame for each
5. Logical-log replay – Frame payloads apply to the B-tree while LogicalLog::running_crc updates.
6. Implicit checkpoint – The engine back-fills pages via Wal::checkpoint and potentially calls Wal::truncate_wal.
7. Ready state – The Database object returns to the caller with a fully-recovered, consistent state.

Automatic Recovery Example

No special API calls are required to trigger recovery. Simply reopening the database executes the full recovery protocol:

use turso::Database;

// Open database (creates WAL if missing)
let mut db = Database::open("my.db", Default::default())?;

// Write transaction - WAL flushes but we don't checkpoint
db.execute("INSERT INTO demo (id, txt) VALUES (1, 'first')")?;

// Simulate crash by exiting without graceful shutdown
std::process::exit(0);   // Power failure simulation

// After process restart
let db = Database::open("my.db", Default::default())?;

// Committed data is visible despite the crash
let row: (i64, String) = db.query_row("SELECT id, txt FROM demo")?;
assert_eq!(row, (1, "first".to_string()));

Manual Recovery Control for Testing

For integration testing or custom tooling, you can drive the recovery loop manually using the low-level API:

use turso::storage::wal::OpenSharedWal;

// Assume `wal` is a `Box<dyn Wal>` obtained from `Database::open`
let mut poll = OpenSharedWal::new(&wal);
loop {
    match poll.poll()? {
        turso::storage::wal::PollResult::Done => break,
        turso::storage::wal::PollResult::Pending => continue,
    }
}

This pattern allows tests to inspect intermediate states or verify idempotency—that running recovery twice produces the same result.

Durability Guarantees and Verification

Turso guarantees Atomicity, Consistency, and Durability after sudden failure through:

• Checksum validation detecting incomplete WAL writes via header CRCs
• Sequential replay of every committed WAL frame through the MVCC logical log
• Safe truncation of the WAL only after the database file sync completes

The correctness of this recovery path is continuously verified in:

• tests/integration/query_processing/test_transactions.rs – Simulates crashes by closing the DB without checkpointing, then re-opening to assert committed rows remain visible
• testing/simulator/runner/memory/mvcc_recovery.rs – Stress-tests recovery idempotency by running the loop repeatedly

Summary

• Turso's recovery process for Turso databases runs automatically on Database::open, requiring no manual intervention.
• The protocol follows three phases: WAL header validation (prepare_wal_header), MVCC logical log replay (LogicalLog::apply_frame), and checkpoint truncation (truncate_wal).
• Checksum validation at the header and frame level detects corruption from partial writes.
• Durability is ensured by syncing the database file before truncating the WAL.
• Recovery is idempotent and safe to run multiple times, as verified by the simulator tests.

Frequently Asked Questions

Is the Turso database recovery process automatic?

Yes. When you call Database::open after a crash, the engine automatically creates an OpenSharedWal instance and drives the recovery state machine via repeated poll calls until completion. No separate recovery command or configuration is required.

What happens if the WAL header is corrupted?

If prepare_wal_header detects a missing magic number, invalid page size, or checksum mismatch, the engine invokes begin_restart to reset the WAL header to a clean, initialized state. This follows SQLite-compatible recovery semantics and allows the database to start fresh while preserving back-filled data in the main database file.

How does Turso ensure data durability during recovery?

Durability is guaranteed through a specific ordering of operations: the engine writes and syncs all back-filled pages to the main database file before calling truncate_wal to delete WAL segments. Additionally, every WAL frame and logical log entry includes CRC checksums that LogicalLog::apply_frame validates before applying updates.

Can I run recovery manually for testing purposes?

Yes. You can instantiate OpenSharedWal directly and manually drive the poll loop, as shown in the code example above. This is useful for integration tests that need to verify specific intermediate states or confirm that the recovery process is idempotent—running it twice on the same WAL produces identical results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}