Turso Sync Engine: How Local SQLite Replicates with Turso Cloud

The Turso Sync Engine is a Rust component that maintains bidirectional synchronization between local SQLite databases and remote Turso Cloud instances using WAL frame streaming, change data capture, and HTTP-based replication protocols.

The sync engine resides in the tursodatabase/turso repository within the sync/engine crate. Built atop Turso Core’s storage, WAL, and CDC (Change Data Capture) layers, it provides edge-deployed applications with cloud-native replication capabilities that keep data consistent across distributed environments.

Core Architecture and Responsibilities

The sync engine operates as an orchestration layer managing five critical responsibilities:

Bootstrapping and Metadata Initialization

When initializing a database, DatabaseSyncEngine::bootstrap_db (located in sync/engine/src/database_sync_engine.rs at lines 39-78) either downloads an existing snapshot from Turso Cloud or creates a fresh local database. This process writes a metadata file (<db>-info) beside the database file using atomic file operations via the full_write method.

Remote Pull Operations

The engine continuously polls Turso Cloud for new WAL frames through wait_changes_from_remote (lines 92-114 in database_sync_engine.rs). This method constructs a SyncOperationCtx, invokes wal_pull_to_file to download changes to a temporary file, and returns a DbChangesStatus. The subsequent apply_changes_from_remote call replays these frames on a revert connection, updates the schema version, and refreshes the metadata state.

Local Push Operations

For uploading local modifications, push_changes_to_remote reads the CDC table (turso_cdc), optionally applies transformations, and streams serialized changes to the cloud via an HTTP POST /push-changes endpoint. This operation uses push_logical_changes (defined in database_sync_operations.rs) to package the change set.

Checkpointing and Revert Mechanism

After each successful pull, the engine executes checkpoint (lines 76-115) to create a revert WAL—a snapshot of the database state before applying remote changes. This enables fast rollback capabilities and maintains WAL consistency through checkpoint_passive. The revert mechanism ensures that failed synchronizations can be safely undone without corrupting the main database.

Integration with Turso Cloud

The sync engine integrates with Turso Cloud through an abstracted HTTP transport layer and metadata-driven state management.

HTTP Transport via SyncEngineIo

All network operations flow through the SyncEngineIo trait defined in sync/sdk-kit/src/sync_engine_io.rs. This trait implements three core operations: http, full_read, and full_write. Requests are queued as SyncEngineIoRequest structs and processed by C-API bindings in bindings/rust/src/sync.rs, which forward them to Turso Cloud microservices.

The engine specifically targets two endpoints:

• GET /pull-updates for retrieving remote WAL frames
• POST /push-changes for uploading local CDC rows

Metadata Synchronization

Synchronization state persists in a JSON metadata file (<db>-info) managed by the DatabaseMetadata type (defined in sync/engine/src/types.rs). This file stores the client ID, last pulled revision, remote URL, and encryption configuration. On startup, DatabaseSyncEngine::create_db (lines 102-114) reads this metadata to resume from the correct state, ensuring exactly-once semantics across process restarts.

End-to-End Encryption Support

When provided with a remote_encryption_key (Base64-encoded) in DatabaseSyncEngineOpts, the engine attaches this key to the SyncOperationCtx during remote operations. The key transmits as an HTTP header to Turso Cloud, enabling server-side encryption/decryption without exposing cryptographic material to local storage.

Implementation Example

Below is a minimal Rust example demonstrating how to create a sync engine instance and perform bidirectional synchronization:

use turso::sync::engine::{DatabaseSyncEngine, DatabaseSyncEngineOpts};
use turso::sync::sdk_kit::SyncEngineIoQueue;
use turso_core::IO;
use std::sync::Arc;

// 1. Initialize the filesystem IO implementation
let io: Arc<dyn IO> = Arc::new(turso_core::io::fs::FileSystem::new());

// 2. Create the SyncEngineIo queue for HTTP/file operations
let sync_io = SyncEngineIoQueue::<Vec<u8>>::new();

// 3. Configure engine options pointing to Turso Cloud
let opts = DatabaseSyncEngineOpts {
    remote_url: Some("https://api.turso.io".into()),
    client_name: "my-app".into(),
    tables_ignore: vec![],
    use_transform: false,
    wal_pull_batch_size: 32,
    long_poll_timeout: None,
    protocol_version_hint: turso::sync::engine::DatabaseSyncEngineProtocolVersion::V1,
    bootstrap_if_empty: true,
    reserved_bytes: 0,
    db_opts: Default::default(),
    partial_sync_opts: None,
    remote_encryption_key: None,
    push_operations_threshold: None,
    pull_bytes_threshold: None,
};

// 4. Create or open the database with sync capabilities
let engine = DatabaseSyncEngine::create_db(
    &coro,
    io.clone(),
    sync_io.clone(),
    "my_local.db",
    opts,
).await?;

// 5. Perform full bidirectional sync (push then pull)
engine.sync(&coro).await?;

// 6. Alternatively, pull only remote changes
engine.pull_changes_from_remote(&coro).await?;

Summary

• The Turso Sync Engine is a Rust-based replication layer within sync/engine that synchronizes local SQLite with Turso Cloud.
• Bootstrapping downloads initial snapshots via bootstrap_db while maintaining state in JSON metadata files.
• Bidirectional sync operates through wait_changes_from_remote (pull) and push_changes_to_remote (push), coordinated by DatabaseSyncEngine.
• WAL management includes a revert mechanism that creates pre-sync snapshots for safe rollback.
• Cloud integration uses the SyncEngineIo abstraction to communicate over HTTP with endpoints like /pull-updates and /push-changes.
• Encryption supports client-provided keys transmitted via HTTP headers to Turso Cloud.

Frequently Asked Questions

What is the Turso Sync Engine?

The Turso Sync Engine is a Rust component in the tursodatabase/turso repository that runs inside your application process. It manages the replication protocol between a local SQLite database and Turso Cloud, handling conflict resolution, encryption, and network resilience without requiring a separate daemon.

How does the sync engine handle encryption?

When you provide a remote_encryption_key in DatabaseSyncEngineOpts, the engine includes this key in the SyncOperationCtx for all remote operations. The key travels as an HTTP header to Turso Cloud, where it encrypts data at rest. The key never persists in local metadata files; it exists only in memory during the sync operation.

What happens during a full sync operation?

The sync method (lines 132-140 in database_sync_engine.rs) executes push_changes_to_remote followed by pull_changes_from_remote. This sequence ensures local modifications reach the cloud before downloading remote updates, preventing divergence between the edge database and the central replica.

Where does the sync engine store its synchronization state?

The engine maintains state in a JSON metadata file named <database-name>-info located beside the database file. This file stores the DatabaseMetadata struct including client ID, last pulled revision, and remote URL, enabling the engine to resume incremental synchronization after application restarts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}