How Bidirectional Synchronization Works Between Durable Objects and Containers

The sync protocol maintains consistency between Durable Objects and containers using revision cursors and watermarks, executing a pull-then-push tick that batches changes, validates cross-side invariants, and recovers from divergence by resetting cursors to re-establish a consistent baseline.

The Computer project implements a robust sync protocol to keep the SQLite-backed VFS inside a Durable Object (DO) synchronized with the FUSE-exposed VFS running in a container (computerd). When both sides mutate the filesystem independently, the protocol ensures eventual consistency through atomic resume points and watermark tracking defined in packages/rpc/src/interface.ts and implemented in packages/rpc/src/sync-driver.ts.

Core Concepts: Revision Cursors and Watermarks

The protocol tracks progress through watermarks persisted in the _vfs_watermark table on the DO side and the local Database abstraction on the container side.

• pushRev (DO side): The highest DO revision successfully pushed to the container.
• fetchCursor (DO side): The resume point for the next pull, formatted as { rev, path } where path=null indicates all changes up to rev have been offered.
• currentRev: A monotonic revision counter stamped on each mutation by the respective side.
• appliedPushCursor: The cursor reported by the receiver after applying a batch, guaranteeing the receiver has caught up to the sender’s pushRev.
• ChangeCursor: The atomic resume type { rev: number; path: string | null } used by both sides to resume interrupted transfers.

These values enable idempotent transfers and bounded-memory processing through batching.

The Pull Phase: Container to Durable Object

The pullOnce() function in packages/rpc/src/sync-driver.ts (around lines 91-130) handles ingestion of container changes into the DO:

1. The DO reads its stored fetch cursor via readFetchCursor.
2. It calls remote.fetchChanges({ after }) on the container RPC interface. The response includes:
   • currentCursor: The container’s latest revision at stream open.
   • appliedPushCursor: The container’s progress on previous pushes.
   • stream: A ReadableStream<ChangeEntry> of entries after the after cursor.
3. The DO validates the cross-side invariant through assertAppliedPushCursor, ensuring the container has applied everything up to the DO’s pushRev. If this invariant fails, the driver resets cursors and retries once (see lines 110-170).
4. Entries are consumed in batches of PULL_BATCH_SIZE = 256 to bound memory usage. For each batch:
   • Collect required chunk hashes.
   • Call remote.hasObjects to identify missing chunks.
   • Stream missing chunks via remote.fetchObjects and stage them with stageBlob.
   • Apply the batch using applyChanges.
   • Advance the fetch cursor with writeFetchCursorIfAhead.
5. After stream exhaustion, the final fetch cursor is persisted.

The Push Phase: Durable Object to Container

The pushOnce() function (around lines 92-64 in packages/rpc/src/sync-driver.ts) propagates DO changes to the container:

1. Read the local pushRev watermark using readWatermark.
2. Coalesce all local changes since pushRev via coalesceChanges.
3. Probe the container with remote.hasObjects to determine missing chunks, then stream those chunks via remote.pushObjects.
4. Stream the ChangeEntry batch via remote.push({ senderRev, changes }).
5. The container returns appliedPushCursor; the DO asserts this covers its local pushRev (lines 54-58).
6. On successful validation, the DO advances its pushRev to the local revision observed at batch start using writeWatermark.

This phase ensures the container receives all mutations since the last successful push.

Watermark Reconciliation and Divergence Recovery

When a connection is (re)established, the DO calls reconcileWatermarks() (lines 95-133 in the driver) to establish a consistent baseline. This function compares the remote’s watermarks (retrieved via remote.watermarks()) with local values. If cursors diverge—indicating a split-brain or lost update scenario—the driver resets the diverging cursor to revision 0. This forces the next pull or push to re-ship the full state from scratch, guaranteeing consistency over availability during recovery.

Revision Cursor Semantics and Protocol Guarantees

Resume points, not snapshots: A cursor { rev, path } marks a resume position, not a snapshot. When path is null, all changes up to rev are confirmed delivered. A non-null path allows mid-revision resumption if a batch interrupts (see cursorComplete in the driver).

Bidirectional flow: Each side acts as both sender and receiver:

• The DO pushes changes via push → the container receives them and updates its fetch cursor.
• The container exposes changes via fetchChanges → the DO pulls them and updates its fetch cursor.

Safety guarantees:

• Idempotence: Already-applied entries are silently dropped.
• Bounded memory: Fixed batch sizes prevent unbounded buffer growth.
• Echo prevention: The cross-side invariant ensures a side never re-pushes data the peer has already applied, avoiding infinite loops.

Implementation Examples

Run a complete synchronization tick using tick(), which sequences pull before push:

import { Database } from "@cloudflare/dofs";
import { createSyncClient } from "@cloudflare/computer/rpc";
import { tick } from "@cloudflare/computer/rpc/src/sync-driver";

// Initialize the local SQLite DB (process-lifetime DB for the container)
const db: Database = /* ... */;

// Create the RPC stub that talks to the Durable Object
const remote = await createSyncClient({ /* DO address, credentials */ });

// Run a full sync tick – typically called after an exec or on a timer
async function syncLoop() {
  const { pulled, pushed } = await tick(db, remote);
  console.log(`Pulled ${pulled.applied} entries, pushed ${pushed} entries`);
}

// Periodic sync (e.g., every 5 seconds)
setInterval(syncLoop, 5_000);

Execute pull-only synchronization when you only need to ingest remote changes:

import { pullOnce } from "@cloudflare/computer/rpc/src/sync-driver";

async function justPull() {
  const result = await pullOnce(db, remote);
  console.log(`Applied ${result.applied} entries`);
}

Execute push-only synchronization for external orchestrators writing directly to the DO:

import { pushOnce } from "@cloudflare/computer/rpc/src/sync-driver";

async function justPush() {
  const sent = await pushOnce(db, remote);
  console.log(`Pushed ${sent} entries`);
}

Summary

• The sync protocol uses revision cursors ({ rev, path }) as atomic resume points to track synchronization progress between Durable Objects and containers.
• A full synchronization tick consists of pull (container → DO) followed by push (DO → container), implemented in packages/rpc/src/sync-driver.ts.
• Watermarks (pushRev, fetchCursor, appliedPushCursor) persist in the _vfs_watermark table and enable idempotent, bounded-memory transfers using PULL_BATCH_SIZE = 256.
• Divergence recovery via reconcileWatermarks() resets diverging cursors to revision 0, forcing a full state re-ship to re-establish consistency.
• The protocol guarantees idempotence and prevents echo loops through cross-side invariant checks in assertAppliedPushCursor.

Frequently Asked Questions

What happens when the container and Durable Object have divergent revision histories?

When reconnecting, reconcileWatermarks() compares local and remote watermarks retrieved via remote.watermarks(). If cursors diverge, the driver resets them to revision 0, causing the next sync to re-ship the full state and establish a consistent baseline from scratch.

How does the sync protocol handle intermittent network failures during batch transfers?

The cursor format { rev, path } acts as a resume point. If path is non-null, the protocol resumes mid-revision using writeFetchCursorIfAhead; if null, all changes up to rev are confirmed delivered. This allows the driver to retry from the last successful cursor rather than restarting the entire sync.

What prevents infinite loops where changes echo back and forth between sides?

The protocol validates cross-side invariants via assertAppliedPushCursor, ensuring a side never re-pushes data the peer has already applied. Each side tracks appliedPushCursor to confirm the receiver has caught up to the sender's pushRev before proceeding.

Why does the implementation use a pull-then-push ordering instead of simultaneous bidirectional streaming?

The tick() function in packages/rpc/src/sync-driver.ts sequences pull before push to ensure the container has ingested all remote changes before attempting to push local mutations. This ordering simplifies conflict resolution and maintains the invariant that each side acts as both sender and receiver in distinct phases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}