SpacetimeDB Storage Mechanisms: In-Memory vs. Disk Persistence Architecture

SpacetimeDB supports both purely in-memory and persistent on-disk storage modes, selected via the Storage enum at database initialization, with disk mode utilizing commitlogs and snapshots while memory mode uses EmptyHistory.

SpacetimeDB offers flexible storage architectures that allow developers to run databases either ephemerally in RAM or with full disk persistence. According to the clockworklabs/SpacetimeDB source code, this choice is implemented through a three-layer system encompassing database-wide storage modes, external program storage, and durability mechanisms. Understanding these SpacetimeDB storage mechanisms is essential for configuring production deployments and optimizing development workflows.

Database-Wide Storage Mode

The fundamental storage decision is expressed by the Storage enum in crates/core/src/db/mod.rs (lines 20‑26):

/// Whether SpacetimeDB is run in memory, or persists objects and
/// a message log to disk.
#[derive(Clone, Copy)]
pub enum Storage {
    /// The object store is in memory, and no message log is kept.
    Memory,
    /// The object store is persisted to disk, and a message log is kept.
    Disk,
}

When the host controller boots a replica, it matches on this enum to determine persistence behavior (see crates/core/src/host/host_controller.rs, lines 905‑919):

match config.storage {
    db::Storage::Memory => RelationalDB::open(...EmptyHistory::new()...),
    db::Storage::Disk   => {
        // load persisted commitlog, snapshots, durability, etc.
    }
}

Memory mode keeps the object store in RAM with no message log, while Disk mode persists objects to disk and maintains a message log for durability.

To select the storage mode via CLI, use the --in_memory flag found in crates/standalone/src/subcommands/start.rs (lines 26‑30). Omitting this flag defaults to Storage::Disk.

Program (Wasm) Storage with DiskStorage

Modules (Wasm programs) are stored externally from the relational data using the DiskStorage implementation of the ExternalStorage trait. Located in crates/core/src/host/disk_storage.rs (lines 12‑14), this struct manages content-addressed program blobs:

pub struct DiskStorage {
    base: PathBuf,
}

DiskStorage provides four key operations:

• new(base) – Creates the base directory using fs::create_dir_all.
• put(value) – Hashes the bytes, writes to a temporary file with nanosecond timestamp, fsyncs, then atomically renames to a hash-derived location.
• get(key) – Reads the file and verifies the stored hash matches the key.
• prune(key) – Removes corrupted or obsolete objects.

The host controller receives a ProgramStorage (type-aliased to Arc<dyn ExternalStorage>) and calls load_program(storage, hash) in crates/core/src/host/host_controller.rs (lines 48‑53) to fetch Wasm bytes during replica bootstrap.

Persistence, Durability, and the Commitlog

When Storage::Disk is selected, the database uses a persistence provider supplying three services as defined in crates/core/src/db/persistence.rs (lines 29‑46):

pub struct Persistence {
    pub durability: Arc<Durability>,
    pub disk_size: DiskSizeFn,
    pub snapshots: Option<SnapshotWorker>,
    pub runtime: tokio::runtime::Handle,
}

Durability implements spacetimedb_durability::Durability<TxData = Txdata> to append committed transactions to a commitlog on disk. DiskSizeFn returns the current on-disk size for metrics and energy accounting. SnapshotWorker (optional) runs background tasks to capture point-in-time snapshots and compress old commitlog segments.

Local On-Disk Durability

The LocalPersistenceProvider creates durability objects for a given replica directory:

let (durability, disk_size) = relational_db::local_durability(replica_dir, Some(&snapshot_worker)).await?;

The underlying commitlog resides in the spacetimedb_commitlog crate. Its size is computed by SizeOnDisk in crates/commitlog/src/repo/fs.rs (lines 107‑112).

In-Memory Durability

When running in Memory mode, the host supplies EmptyHistory::new()—a no-op durability implementation that allocates no commitlog and performs no disk writes.

How Replica Initialization Works

The startup sequence in crates/core/src/host/host_controller.rs (lines 905‑960) orchestrates the storage mechanisms:

1. Parse CLI to determine Storage (memory vs. disk).
2. HostController builds a Host through try_init.
3. Match on storage:
   • Memory → RelationalDB::open with EmptyHistory (no durability, no snapshots).
   • Disk → Load persisted commitlog, obtain Persistence from LocalPersistenceProvider, open RelationalDB with durability and snapshot services.
4. Program loading: If the database contains a program hash, fetch it from in-process ProgramStorage; otherwise load from external DiskStorage.

Code Examples

Configuring Storage in Rust

use spacetimedb::db::{Config, Storage};

let config = Config {
    storage: Storage::Disk,               // or Storage::Memory
    page_pool_max_size: Some(256 << 20),  // 256 MiB page pool (optional)
};

Using DiskStorage Directly

use spacetimedb::host::DiskStorage;
use spacetimedb_lib::Hash;
use std::path::PathBuf;

// Create storage directory under `/var/lib/spacetimedb/programs`
let base = PathBuf::from("/var/lib/spacetimedb/programs");
let disk = DiskStorage::new(base).await?;

// Store a Wasm module
let wasm_bytes: &[u8] = include_bytes!("my_mod.wasm");
let hash: Hash = disk.put(wasm_bytes).await?;

// Retrieve it later
let maybe_blob = disk.get(&hash).await?;
assert_eq!(maybe_blob.unwrap().as_ref(), wasm_bytes);

Starting Standalone Server in Memory Mode

spacetimedb start --in_memory --data_dir /tmp/spacetime_data

This forces Storage::Memory, causing the server to use EmptyHistory with no disk writes for objects or the commit log.

Disabling Snapshots (Disk Mode)

Snapshots are automatically enabled when using LocalPersistenceProvider. To disable them while retaining disk persistence:

let snapshots = None; // disable
let persistence = Persistence::new(durability_impl, size_on_disk, snapshots, tokio::runtime::Handle::current());

Summary

• SpacetimeDB storage mechanisms operate at three levels: database-wide storage mode, external program storage, and durability services.
• The Storage enum (Memory vs. Disk) in crates/core/src/db/mod.rs controls whether the database runs ephemerally or persists to disk.
• Wasm programs are stored separately via DiskStorage in crates/core/src/host/disk_storage.rs, which provides content-addressed, atomic file storage.
• Disk mode utilizes the Persistence struct to coordinate commitlog durability, disk size metrics, and background snapshots, while memory mode uses EmptyHistory to bypass persistence entirely.
• Replica initialization in crates/core/src/host/host_controller.rs wires these components together based on CLI flags or programmatic configuration.

Frequently Asked Questions

What is the difference between Memory and Disk storage in SpacetimeDB?

Memory mode stores the object store entirely in RAM using EmptyHistory for durability, resulting in no disk writes for data or commit logs. Disk mode persists the object store to disk and maintains a commitlog for transaction durability, optionally utilizing background snapshots for recovery and compaction, as implemented in the Persistence struct.

How does SpacetimeDB ensure atomicity when storing Wasm programs?

The DiskStorage::put method in crates/core/src/host/disk_storage.rs writes program bytes to a temporary file with a nanosecond timestamp, calls fsync to ensure data reaches the filesystem, then performs an atomic rename to the final hash-derived location. This pattern prevents partial writes and ensures content-addressed integrity.

Can I disable snapshots while keeping disk persistence?

Yes. While LocalPersistenceProvider automatically enables snapshots, you can manually construct a Persistence instance using Persistence::new(durability_impl, size_on_disk, None, runtime_handle), passing None for the snapshot parameter. This maintains commitlog durability without background snapshot processing.

Where are commitlog files stored on disk?

Commitlog files are stored within the replica directory structure managed by the spacetimedb_commitlog crate. The SizeOnDisk calculation in crates/commitlog/src/repo/fs.rs traverses these files to report total persistence storage consumption, typically located under the configured data directory alongside the relational database files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}