# Turso Multi-Process WAL Coordination with .tshm Sidecar: Implementation Guide

> Learn how Turso uses a .tshm sidecar file for multi-process WAL coordination. Achieve safe concurrent reads and writes without scanning WAL

- Repository: [Turso Database/turso](https://github.com/tursodatabase/turso)
- Tags: how-to-guide
- Published: 2026-06-22

---

**Turso uses a memory-mapped `.tshm` sidecar file to coordinate write-ahead-log access across multiple OS processes, enabling safe concurrent reads and writes without per-process WAL scanning.**

The `tursodatabase/turso` repository implements an experimental multi-process write-ahead-log (WAL) coordination mechanism that allows several operating system processes to share a single database file safely. At the heart of this system lies the `.tshm` sidecar, a shared-memory file that stores a compact data structure tracking WAL state, writer ownership, and a shared page-to-frame index. This architecture eliminates the need for each process to scan the entire WAL independently, significantly improving performance in multi-process workloads.

## What Is the .tshm Sidecar?

The `.tshm` sidecar is a **shared-memory coordination file** created alongside the main database when multi-process WAL mode is enabled. When a database is opened with the `--experimental-multiprocess-wal` flag, Turso creates or reuses a file named `<dbname>.db-tshm` in the same directory as the database.

According to the source code in [`core/storage/shared_wal_coordination.rs`](https://github.com/tursodatabase/turso/blob/main/core/storage/shared_wal_coordination.rs), this file is memory-mapped by every participating process and contains:
- A fixed header with magic constants, version numbers, and snapshot metadata (`max_frame`, `checkpoint_seq`, page size)
- Lock bytes for coordinating writers, checkpointers, and readers
- Owner records tracking which OS process holds which slot
- The shared page-to-frame index for fast WAL lookups

## How the Sidecar Coordinates Multi-Process Access

### Shared-Memory Layout and Header Structure

The sidecar file begins with a fixed header that contains a **magic constant** and **version number** to validate the file format. The header stores critical snapshot metadata including the current `max_frame`, `nbackfills`, `checkpoint_seq`, and the database page size. This layout is defined in [`core/storage/shared_wal_coordination.rs`](https://github.com/tursodatabase/turso/blob/main/core/storage/shared_wal_coordination.rs) where the data structures specify the exact byte offsets for each field.

When Turso opens a database, it probes the `.tshm` file. If the file is fresh (size 0), the sidecar initializes with default values. If the file exists, Turso validates the persisted header against expected values. Validation failures behave differently depending on the open mode: under an **Exclusive** open, Turso sanitizes the backfill proof and reinitializes the header, whereas under a **MultiProcess** open, an invalid header results in an error because another live process may rely on that state.

### Lock Byte Layout and Platform-Specific Locking

The sidecar uses a **byte-range locking scheme** to coordinate access between processes. The lock byte layout assigns specific meaning to each byte:
- **Byte 0**: The *lifetime* lock distinguishing **Exclusive** opens (no other process holds the file) from **MultiProcess** opens (another process already holds the sidecar)
- **Byte 1**: The writer lock
- **Byte 2**: The checkpointer lock
- **Bytes 3+**: Individual lock bytes for each reader slot

The implementation differs by platform to handle different `fcntl` semantics. On Linux, Turso uses **open-file-description** locks (`F_OFD_SETLK`) via the `SharedWalOwnershipMode` enum, which are independent per `open()` call and do not conflict with threads in the same process. On macOS and other Unix systems, it falls back to classic `F_SETLK` process-scoped locks, which require additional coordination to avoid conflicts between connections in the same process.

### Ownership Bookkeeping and Process-Local State

To prevent a process from deadlocking against itself when using process-scoped locks, Turso maintains **owner records** in the shared slots containing the PID and instance ID. Inside each process, a `ProcessLocalOwnershipState` mirrors these slots, ensuring that sibling connections within the same process respect each other's ownership without attempting to acquire conflicting locks. This separation is vital because `F_SETLK` locks are per-process, not per-file-descriptor, making internal tracking necessary to avoid self-deadlock.

## Shared Page-to-Frame Index and Snapshot Publishing

### The Append-Only Index Structure

To avoid a full WAL scan on every read, the sidecar maintains an **append-only index** mapping database pages to the most recent WAL frame that modified them. The index grows in fixed-size blocks with a capacity of **4,096 entries per block**, using an open-addressing hash table for fast lookups. This index is protected by a per-process `frame_index_publish_lock` and is lazily memory-mapped as needed to conserve resources.

### Lock-Free Snapshot Reading

When a writer commits a new WAL frame, it updates the shared header inside a **sequence lock** (`snapshot_seq`). Readers spin on this odd/even value to obtain a coherent snapshot of fields such as `max_frame`, `nbackfills`, and `checkpoint_seq`. The `snapshot()` method in [`shared_wal_coordination.rs`](https://github.com/tursodatabase/turso/blob/main/shared_wal_coordination.rs) demonstrates this lock-free read pattern, allowing readers to access current WAL metadata without blocking writers.

```rust
// Accessing the current snapshot from any process
let snapshot = coordination.snapshot();
println!("Current WAL max frame: {}", snapshot.max_frame);

```

## Recovery, Re-opening, and Graceful Shutdown

On process startup, Turso probes the `.tshm` file to determine initialization state. If the file is fresh, it initializes the header; otherwise, it validates the persisted structure. When the last process holding a sidecar mapping exits, Turso detects this condition through the `is_last_process_mapping()` method, which checks if the **lifetime lock** (byte 0) can be reacquired. This mirrors SQLite’s "last connection cleans up shared state" behavior, ensuring that temporary files are properly managed when no processes remain.

```rust
// Detecting the last process before shutdown
if coordination.is_last_process_mapping() {
    // This is the final process holding the .tshm sidecar
    // Perform necessary cleanup before exit
}

```

## Enabling Multi-Process WAL Mode

### Command-Line Interface

To enable the experimental feature when launching the server or CLI REPL, add the flag documented in [`docs/manual.md`](https://github.com/tursodatabase/turso/blob/main/docs/manual.md) at line 158:

```bash
tursodb --experimental-multiprocess-wal path/to/database.db

```

### Rust API Integration

The high-level API in [`core/storage/wal.rs`](https://github.com/tursodatabase/turso/blob/main/core/storage/wal.rs) integrates with the shared coordination when `multiprocess_wal` is enabled in the configuration:

```rust
use turso::Database;
use turso::OpenOptions;

// High-level API
let db = Database::open(
    OpenOptions::new()
        .path("example.db")
        .multiprocess_wal(true)   // enables the .tshm sidecar
)?;

```

For low-level control, the `MappedSharedWalCoordination::create_or_open` method in [`core/storage/shared_wal_coordination.rs`](https://github.com/tursodatabase/turso/blob/main/core/storage/shared_wal_coordination.rs) creates the sidecar directly:

```rust
use std::sync::Arc;
use turso::io::IO;
use turso::storage::shared_wal_coordination::MappedSharedWalCoordination;

fn open_shared_coordination(io: &Arc<dyn IO>, db_path: &std::path::Path) -> anyhow::Result<()> {
    // reader_slot_count is typically a multiple of 64; 256 is a common default
    let coordination = MappedSharedWalCoordination::create_or_open(
        io,
        &db_path.with_extension("db-tshm"),
        256,
    )?;
    
    // The returned coordination can be used to register readers, obtain snapshots, etc.
    Ok(())
}

```

## Summary

- The `.tshm` sidecar is a **memory-mapped coordination file** that replaces SQLite’s per-process WAL scan in multi-process scenarios, located in [`core/storage/shared_wal_coordination.rs`](https://github.com/tursodatabase/turso/blob/main/core/storage/shared_wal_coordination.rs).
- It provides **fast, lock-free snapshots** of WAL metadata and a **shared page-to-frame index** with 4,096-entry blocks to serve readers without scanning the WAL.
- Ownership is maintained via **byte-range locks** (OFD locks on Linux, `F_SETLK` on macOS) and **process-local registries** (`ProcessLocalOwnershipState`) to avoid conflicts between connections in the same process.
- The feature is controlled by the **`--experimental-multiprocess-wal`** flag and is intended for workloads where many short-lived processes need concurrent read/write access to the same Turso database.

## Frequently Asked Questions

### What is the .tshm file format?

The `.tshm` file is a binary shared-memory file containing a fixed header with magic constants and snapshot metadata, followed by lock bytes, owner records (PID + instance ID), and an append-only page-to-frame index. The layout is defined in [`core/storage/shared_wal_coordination.rs`](https://github.com/tursodatabase/turso/blob/main/core/storage/shared_wal_coordination.rs) and uses open-addressing hash tables with 4,096-entry blocks for the index structure.

### How does Turso handle locking on Linux vs macOS?

On Linux, Turso uses **open-file-description locks** (`F_OFD_SETLK`) which are scoped to the file descriptor rather than the process, allowing multiple threads in the same process to hold different locks without conflict. On macOS and other Unix systems, it falls back to classic **process-scoped locks** (`F_SETLK`), requiring the `ProcessLocalOwnershipState` to track which connections within a process hold which slots to prevent self-deadlock.

### What happens when the last process exits?

When the final process holding a mapping to the `.tshm` file exits, Turso detects this via the `is_last_process_mapping()` method, which attempts to reacquire the lifetime lock on byte 0. If successful, the process knows it is the last one and can perform cleanup operations, similar to SQLite’s last-connection-cleanup pattern.

### Is multi-process WAL production ready?

No, multi-process WAL coordination is **experimental**. The feature is guarded by the `--experimental-multiprocess-wal` flag and documented in [`docs/manual.md`](https://github.com/tursodatabase/turso/blob/main/docs/manual.md) and `docs/sql-reference/multiprocess-access.mdx` as not being production-ready. It is intended for testing and specific workloads where many short-lived processes require concurrent database access.