SpacetimeDB Scaling and Performance: Horizontal and Vertical Optimization Guide

SpacetimeDB achieves horizontal scaling through configurable replica sets with automatic leader election, while vertical scaling relies on strategic index management that trades write latency for read performance.

SpacetimeDB is an open-source relational database optimized for real-time applications, combining the performance of in-memory processing with the durability of persistent storage. Understanding SpacetimeDB scaling and performance requires examining both its distributed architecture for horizontal growth and its indexing strategies for single-node optimization.

Horizontal Scaling Through Replica Management

Replica Architecture and Leader Election

The foundation of horizontal scaling in SpacetimeDB resides in crates/standalone/src/lib.rs, where the Replica struct defines each node in a cluster with database_id, node_id, and a critical leader boolean flag. The control database in crates/standalone/src/control_db.rs tracks leadership via get_leader_replica_by_database, ensuring that only the leader replica processes writes while followers serve read-only traffic.

Dynamic Replica Scheduling

The schedule_replicas function in crates/standalone/src/lib.rs orchestrates cluster topology. When a database is created, the engine invokes schedule_replicas(database_id, num_replicas), where num_replicas defaults to 1 for standalone deployments but accepts any u8 value for clustered configurations.

During module updates, the publish_database function compares desired versus actual replica counts. It triggers insert_replica to scale up or delete_replica (for non-leader nodes) to scale down, emitting transparent log messages: "Scaling up database …" or "Scaling down database …".

Client-Side Configuration

Users configure replication through the client API defined in crates/client-api/src/lib.rs. The DatabaseDef::num_replicas field accepts replication factors via the unstable CLI flag --num-replicas:

spacetime publish \
    --file my_module.wasm \
    --identity my-db \
    --num-replicas 3

This value propagates through the REST/WebSocket API to the backend replication logic in crates/standalone/src/lib.rs.

Vertical Scaling with Index Optimization

Index Architecture and Memory Tracking

Vertical scaling in SpacetimeDB depends on efficient index management within crates/table/src/table.rs. The Table struct maintains index metadata through several diagnostic methods:

• num_indices() returns self.indexes.len()
• num_rows_in_indexes() calculates self.num_rows() * self.indexes.len()
• bytes_used_by_index_keys() iterates all indexes summing idx.num_key_bytes()

These metrics enable precise memory accounting when scaling tables to millions of rows.

Index Implementations and Trade-offs

The crates/table/src/table_index/ directory contains specialized index structures:

• hash_index.rs – O(1) lookups for equality predicates
• unique_direct_index.rs – Optimized single-element lookups with uniqueness constraints
• multimap.rs – Many-to-many relationship support

Selecting the appropriate index shape transforms O(N) table scans into O(log N) or O(1) operations.

Write Path Performance Implications

Every write operation traverses all table indexes. In Table::insert (around line 890), the engine invokes index.check_and_insert for each index:

// Conceptual representation of the write path
for index in &self.indexes {
    index.check_and_insert(row)?;
}

This means write latency grows linearly with the number of indexes on the table. High-throughput applications must balance read optimization against write amplification.

Read Path Optimization

The read path leverages indexes through find_same_row_via_unique_index (around line 945), which bypasses full table scans when serving queries against unique indexed columns. Direct column access via Table::project and RowRef::read_col operates without index overhead when projections don't require indexed lookups.

Observability and Runtime Metrics

Key Performance Indicators

SpacetimeDB exposes Prometheus-compatible metrics in crates/core/src/worker_metrics/mod.rs. Critical indicators for scaling decisions include:

• spacetime_total_incoming_queue_length – WebSocket message backlog indicating client write pressure
• spacetime_total_outgoing_queue_length – Server-to-client message queue depth, relevant during replica synchronization
• spacetime_replay_total_time_seconds – Database recovery duration after restart, scaling with replica count and log size
• spacetime_snapshot_creation_time_total – Snapshot overhead, sensitive to index cardinality
• spacetime_module_create_instance_time_seconds – WASM/V8 instantiation latency during module scaling

Monitoring Replication Health

Metrics automatically tag data with db: Identity and replica_id, enabling per-replica analysis. Grafana dashboards can track replica lag through spacetime_total_outgoing_queue_length per replica, while spacetime_replay_total_time_seconds reveals how long replicas take to catch up after restarts.

Practical Implementation Examples

Configuring Multi-Replica Deployment

Deploy a three-node cluster using the CLI:

spacetime publish \
    --file my_module.wasm \
    --identity production-db \
    --num-replicas 3

This triggers schedule_replicas with num_replicas = 3, creating two follower replicas and one leader in crates/standalone/src/lib.rs.

Optimizing Table Indexes

Add a composite index for high-cardinality queries:

use spacetimedb::{Table, IndexAlgorithm, BTreeAlgorithm};

let mut table = MyTable::table(&db);
let algo = IndexAlgorithm::BTree(BTreeAlgorithm {
    columns: vec!["user_id".into(), "timestamp".into()],
    unique: false,
});
table.add_index(algo)?;

This creates a B-tree index in crates/table/src/table_index/, reducing query complexity from O(N) to O(log N).

Monitoring with Prometheus

Configure scraping for the metrics endpoint:

# prometheus.yml

scrape_configs:
  - job_name: "spacetime"
    static_configs:
      - targets: ["localhost:9090"]

Query for scaling bottlenecks:

# High incoming queue indicates need for more replicas or index optimization

spacetime_total_incoming_queue_length{db="production-db"}

# Snapshot duration correlates with index count

rate(spacetime_snapshot_creation_time_total_seconds[5m])

Summary

• Horizontal scaling is achieved through the Replica struct and schedule_replicas logic in crates/standalone/src/lib.rs, supporting dynamic scaling via the --num-replicas CLI flag.
• Vertical scaling depends on strategic index management in crates/table/src/table.rs, where each index adds linear write overhead but provides logarithmic or constant-time read improvements.
• Observability is provided through Prometheus metrics in crates/core/src/worker_metrics/mod.rs, tracking queue depths, replay times, and snapshot durations to inform scaling decisions.
• Trade-offs between replication (availability/read scaling) and indexing (query performance) must be balanced based on workload characteristics observed through runtime metrics.

Frequently Asked Questions

How does SpacetimeDB handle leader election during horizontal scaling?

SpacetimeDB tracks leadership through the leader boolean flag on the Replica struct in crates/standalone/src/lib.rs. The control database in crates/standalone/src/control_db.rs provides get_leader_replica_by_database to identify the current leader, ensuring only one replica processes writes while followers remain read-only. When scaling up, the existing leader remains unchanged until explicitly failed over.

What is the performance impact of adding indexes in SpacetimeDB?

Each index adds linear write overhead because Table::insert in crates/table/src/table.rs calls index.check_and_insert for every index on the table. However, indexes provide O(log N) or O(1) read performance for indexed columns versus O(N) table scans. The bytes_used_by_index_keys method helps monitor memory consumption as tables scale to millions of rows.

How can I monitor replica lag in a SpacetimeDB cluster?

Monitor the spacetime_total_outgoing_queue_length metric from crates/core/src/worker_metrics/mod.rs, which tracks server-to-client message queues per replica. High values indicate replication lag. Additionally, track spacetime_replay_total_time_seconds to measure how long replicas take to catch up after restarts, which grows with replica count and transaction log size.

Can I change the number of replicas without downtime?

Yes, SpacetimeDB supports dynamic scaling through the publish_database logic in crates/standalone/src/lib.rs. When you publish a module with a different --num-replicas value, the engine compares the desired count with current replicas and automatically invokes insert_replica to scale up or delete_replica (for non-leaders) to scale down without requiring a full cluster restart.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}