How Turso's Write-Ahead Log (WAL) Ensures Durability and Concurrency

Turso's WAL implementation combines explicit filesystem synchronization, atomic checkpointing, and fine-grained reader-writer locks to guarantee that committed transactions survive crashes while allowing multiple concurrent readers and a single writer to operate safely.

The Turso database (tursodatabase/turso) implements a fully-featured, SQLite-compatible Write-Ahead Log (WAL) in Rust that separates durability guarantees from concurrency control. This architecture ensures that every committed transaction persists to disk through explicit fsync operations and back-fill proofs, while a sophisticated locking mechanism enables high-concurrency access patterns. Understanding how Turso's WAL handles both crash recovery and simultaneous read/write operations requires examining the specific implementation details in core/storage/wal.rs.

How Durability Works in Turso's WAL

At its core, durability in Turso's WAL relies on a strict sequence of disk synchronization operations that ensure data survives system crashes. The implementation follows SQLite's protocol but adds Rust-specific safety guarantees and optional cross-process coordination.

WAL Header and Frame Preparation

Before appending the first frame of any transaction, the engine prepares a fresh WAL header containing magic numbers, page size, salts, and checksums. The prepare_wal_header function (lines 1198-1224) creates this structure and returns a Some(Completion) only after the header is safely written to disk.

Each page destined for the WAL is converted into a PreparedFrames struct via prepare_frames (lines 868-886). This structure holds serialized frame buffers, per-frame metadata, and running checksums that verify data integrity throughout the write process.

Commit Publishing and Durable Sync

After frames are written to the WAL file using vectored I/O (pwritev), the commit_prepared_frames function (lines 890-894) updates the in-memory index, advances max_frame, and records the new checksum. This atomic operation makes the frames visible to readers while maintaining consistency.

The actual durability guarantee comes from the sync method (lines 637-677), which issues fsync or fdatasync calls depending on the FileSyncType. An atomic syncing flag prevents concurrent durability operations from interfering with each other.

For checkpoint modes that truncate the WAL, Turso optionally installs a durable back-fill proof via install_durable_backfill_proof (lines 447-456). This platform-specific operation ensures checkpointed pages are flushed to the main database file before the WAL is truncated, preventing partial recovery scenarios.

Checkpointing and Truncation

The checkpoint routine (lines 1314-1330) drives the process of copying frames from the WAL into the main DB file through checkpoint_inner. Protected by a checkpoint lock, this routine calls publish_backfill before optionally invoking truncate_wal (lines 4450-4478).

The truncation operation rewrites the WAL to zero length and performs a final fsync, ensuring that old log bytes cannot be recovered after a crash, effectively sealing the durability guarantee of the checkpointed data.

Concurrency Control Mechanisms

Turso's WAL enables high-concurrency access through a custom read-write lock implementation and strategic use of read-mark slots that implement a form of multi-version concurrency control.

The TursoRwLock Primitive

The foundation of Turso's concurrency model is TursoRwLock (lines 770-824), a 64-bit atomic structure that packs writer flags, reader counts, and a 32-bit embedded value into a single atomic variable. This lock provides read(), write(), upgrade(), downgrade(), and unlock() operations with minimal overhead.

Read-Mark Slots and Multi-Version Concurrency

Turso maintains five read-mark slots (read_locks[0..4]) described at lines 667-673. Slot 0 is special: when held shared, the reader bypasses the WAL entirely and reads directly from the main database file. Slots 1-4 carry frame numbers that limit how far into the WAL each reader may see, creating snapshot isolation.

When acquiring a read transaction via try_begin_read_tx (lines 6990-7020), the system first obtains a vacuum read lock, selects the best read-mark slot, upgrades it to exclusive briefly to write the current max_frame, then downgrades to shared. This sequence ensures readers see a consistent snapshot without blocking writers.

Writer Exclusivity and Transaction Upgrades

Writers must follow SQLite's "read-to-write" rule, first holding a read transaction before upgrading. The begin_write_tx function (lines 5110-5135) verifies the WAL snapshot is current, then calls coordination.try_begin_write_tx() to acquire the write lock exclusively.

The writer flag in TursoRwLock guarantees that only one writer operates at a time; subsequent writers receive Busy errors until the current writer releases the lock via end_write_tx.

Checkpoint and VACUUM Serialization

Checkpointing requires exclusive access to prevent partial reads of back-filled pages. The acquire_checkpoint_guard function (lines 1009-1053) acquires the checkpoint lock (checkpoint_lock.write()), the write lock, and read-mark slot 0 simultaneously, ensuring no reader sees partially checkpointed data.

VACUUM operations use a separate vacuum_lock (also a TursoRwLock) to block new readers and writers during in-place database rewrites. The VacuumLockGuard (lines 887-926) manages this exclusion, preventing any transaction from starting until the full rewrite and WAL truncation complete.

Cross-Process Coordination

When compiled with the host_shared_wal feature, Turso extends these guarantees across process boundaries using ShmWalCoordination (lines 1290-1296). This implementation maps shared memory and uses POSIX-style file locks to coordinate readers, writers, and checkpoint operations between separate processes accessing the same database file.

The WalCoordination trait abstracts whether the system runs in standalone mode (InProcessWalCoordination) or multi-process mode, ensuring that every writer's view of WAL metadata remains consistent with shared state before any durability operation executes.

Practical Implementation Examples

The following Rust examples demonstrate working with Turso's WAL API from the turso-core crate.

Starting a Read Transaction

To begin a read transaction that establishes a snapshot of the database:

use turso_core::storage::wal::Wal;
use turso_core::storage::wal::WalFile;

fn read_tx(wal: &WalFile) -> turso_core::Result<bool> {
    // Returns `Ok(changed)` indicating whether the DB has been
    // modified since the last read transaction.
    wal.begin_read_tx()
}

This calls WalFile::begin_read_tx (lines 6984-7020), which handles read-mark slot selection and locking.

Executing a Write Transaction

The complete workflow for writing a page and committing it durably:

fn write_tx(wal: &WalFile, page_id: u64, page: &[u8]) -> turso_core::Result<()> {
    // 1. Acquire a read transaction first (required by SQLite semantics)
    wal.begin_read_tx()?;
    
    // 2. Upgrade to a write transaction
    wal.begin_write_tx(turso_core::storage::wal::WalAutoActions::all_enabled())?;
    
    // 3. Prepare a single frame
    let prepared = wal.prepare_frames(
        &[wal.buffer_pool.page_ref(page_id)],
        wal.page_size(),
        Some(0), // db size after commit
        None,
    )?;
    
    // 4. Append frames to the WAL file
    let completion = wal.append_frames_vectored(prepared.pages.to_vec(), wal.page_size())?;
    completion.wait()?; // Wait for disk write
    
    // 5. Publish the commit (makes frames visible to other readers)
    wal.commit_prepared_frames(&[prepared]);
    
    // 6. Finalize the pages (marks them clean)
    wal.finalize_committed_pages(&[prepared]);
    
    // 7. End the transaction
    wal.end_write_tx();
    Ok(())
}

Key functions involved: begin_write_tx (lines 5110-5135), prepare_frames (lines 868-886), append_frames_vectored (lines 1000-1005), and commit_prepared_frames (lines 890-894).

Performing a Checkpoint

To execute a full checkpoint that blocks writers and truncates the WAL:

fn checkpoint(wal: &WalFile, pager: &Pager) -> turso_core::Result<()> {
    // Full checkpoint blocks writers and forces WAL restart
    let result = wal.checkpoint(pager, turso_core::storage::wal::CheckpointMode::Full)?;
    // Returns number of frames back-filled and truncate status
    Ok(())
}

This invokes Wal::checkpoint which delegates to WalFile::checkpoint (lines 1314-1330) and automatically handles the durable synchronization.

Handling VACUUM Operations

For exclusive VACUUM operations that require blocking all other access:

fn vacuum(wal: &WalFile, pager: &Pager) -> turso_core::Result<()> {
    // Acquire exclusive VACUUM lock (may return Busy)
    wal.try_begin_vacuum_checkpoint_lock()?;
    
    // Block new readers and writers
    wal.begin_vacuum_blocking_tx()?;
    
    // Run checkpoint using the held lock
    let result = wal.vacuum_checkpoint_with_held_lock(pager)?;
    
    // Release the lock after truncation
    wal.release_vacuum_checkpoint_lock();
    Ok(())
}

Critical calls include try_begin_vacuum_checkpoint_lock (lines 1389-1394), begin_vacuum_blocking_tx (lines 1529-1545), and vacuum_checkpoint_with_held_lock (lines 1266-1274).

Summary

• Turso's WAL implementation in core/storage/wal.rs provides SQLite-compatible durability through explicit fsync operations, commit publishing, and durable back-fill proofs before truncation.
• Checkpointing serializes through exclusive locks and ensures atomic movement of frames from WAL to main database file, followed by WAL truncation.
• Concurrency is managed by TursoRwLock, a compact 64-bit atomic lock supporting reader/writer/upgraded modes, combined with five read-mark slots that implement snapshot isolation.
• Single-writer semantics are enforced by requiring readers to upgrade to writers atomically, while VACUUM operations acquire a separate exclusive lock to block all other transactions.
• Cross-process support extends these guarantees via ShmWalCoordination and shared memory when compiled with host_shared_wal.

Frequently Asked Questions

How does Turso prevent data loss if the system crashes during a write?

Turso ensures durability through a sequence of explicit disk synchronization operations. Before making frames visible via commit_prepared_frames, the system writes data using vectored I/O and calls sync (lines 637-677) to issue fsync or fdatasync on the underlying file descriptor. For checkpoint operations that truncate the WAL, the install_durable_backfill_proof function (lines 447-456) ensures all checkpointed pages are flushed to the main database file before the log is cleared, preventing recovery of partial data.

Can multiple processes write to the same Turso database simultaneously?

No, Turso enforces single-writer semantics regardless of whether running in single-process or multi-process mode. The begin_write_tx function (lines 5110-5135) acquires an exclusive write lock through the TursoRwLock primitive, causing subsequent writers to receive Busy errors until the current transaction completes. However, multiple readers can access the database concurrently from different processes when using ShmWalCoordination with the host_shared_wal feature enabled.

What is the purpose of read-mark slots in Turso's WAL?

The five read-mark slots (lines 667-673) implement snapshot isolation by tracking how far into the WAL each active reader is allowed to see. Slot 0 allows direct reads from the main database file, bypassing the WAL entirely, while slots 1-4 store specific frame numbers that limit the reader's view. When a reader begins a transaction via try_begin_read_tx (lines 6990-7020), it writes the current max_frame into its assigned slot, ensuring it sees a consistent snapshot even as new frames are appended by the writer.

How does checkpointing affect running transactions?

Checkpointing requires exclusive access to ensure consistency. The acquire_checkpoint_guard function (lines 1009-1053) obtains the checkpoint lock, write lock, and read-mark slot 0 simultaneously, blocking new readers and writers while copying frames to the main database file. Existing read transactions continue operating against their established snapshot in the WAL, but no new transactions can start until the checkpoint releases these locks. VACUUM operations use a similar but stricter locking strategy via begin_vacuum_blocking_tx (lines 1529-1545) to perform full database rewrites.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}