Sync Cursors and Watermarks in Cloudflare Computer: How Incremental Sync Works

Sync cursors and watermarks are lightweight revision markers persisted in the local SQLite database that enable Cloudflare Computer to transmit only delta changes between a local VFS and a remote Durable Object, automatically recovering from network interruptions by resetting state when divergence is detected.

Cloudflare Computer synchronizes a local SQLite-backed virtual file system with a remote peer through an efficient incremental protocol. The system relies on two distinct persistence mechanisms—sync cursors and watermarks—to track which changes have been exchanged across the network boundary. According to the Cloudflare Computer source code in packages/rpc/src/sync-driver.ts, these primitives allow the driver to resume interrupted synchronizations without full tree rescans or redundant data transfers.

Understanding Sync Cursors

A sync cursor (implemented as the ChangeCursor type) represents the remote peer's progress in the change stream. It stores the last revision (rev) and an optional path that the local side has successfully processed from the remote.

The cursor is persisted in the local database through the readFetchCursor and writeFetchCursor functions located in packages/rpc/src/sync-driver.ts (around lines 91-94). During a pull operation, the driver sends this cursor to the remote as the after parameter in remote.fetchChanges({ after }). The remote then streams every ChangeEntry occurring after that revision, ensuring the local side receives only incremental updates rather than the full change history.

Understanding Watermarks

While cursors track fetch progress, watermarks track push progress and cross-side acknowledgment. The system maintains two critical scalar values stored via readWatermark and writeWatermark in packages/rpc/src/sync-driver.ts (around lines 92-94):

• pushRev: The highest revision that the local side has successfully pushed to the remote.
• fetchCursor: The remote's view of the highest senderRev it has applied from the local peer.

These values are exposed over RPC through the SyncRPC.watermarks() interface defined in packages/rpc/src/interface.ts (lines 54-66). Watermarks enable each side to detect state divergence after restarts or network glitches. If the remote's reported revision falls behind the local watermark, the driver resets the appropriate cursor to 0 and replays the missing changes to ensure consistency.

How Incremental Sync Works

The synchronization driver orchestrates bidirectional data flow through a series of bounded operations. According to the implementation in packages/rpc/src/sync-driver.ts, each sync round follows this protocol:

1. Read the local fetch cursor – The driver retrieves the last processed revision using after = readFetchCursor(db, backend).

2. Request remote changes – The driver calls remote.fetchChanges({ after }), which returns three critical pieces of data: currentCursor (the remote's latest snapshot), appliedPushCursor (what the remote has received from us), and a ReadableStream of ChangeEntry objects.

3. Detect divergence – In pullOnceImpl (lines 110-135), the driver compares watermarks. If appliedPushCursor.rev < localPushRev or currentCursor < after, it resets the divergent cursor to 0 and retries the operation (lines 132-168).

4. Process bounded batches – The consumer processes the stream in chunks of PULL_BATCH_SIZE = 256 (lines 47-52), maintaining O(batch) memory usage regardless of total dataset size.

5. Deduplicate blob data – Before fetching file contents, the driver probes the remote via remote.hasObjects and only retrieves missing blobs, staging them locally with stageBlob (lines 114-124).

6. Apply and advance – After applying the batch via applyChanges, the driver advances the fetch cursor using writeFetchCursorIfAhead (lines 45-52 and 80-86).

7. Push local changes – When pushing, the pushOnce function (lines 89-106) transmits missing blobs via pushObjects, streams change entries, verifies the remote acknowledgment through assertAppliedPushCursor (lines 54-58), and updates the local pushRev watermark via writeWatermark (lines 60-62).

Divergence Recovery Mechanisms

Network interruptions and process restarts can cause the local and remote states to drift. The reconcileWatermarks function (lines 79-92 and 99-106 in packages/rpc/src/sync-driver.ts) implements the recovery logic. When a mismatch is detected between the local pushRev and the remote's appliedPushCursor, or between the local fetch cursor and the remote's currentCursor, the driver automatically resets the offending cursor to zero. This forces a replay of changes from the beginning of the divergent window, ensuring consistency without requiring a full synchronization of the entire file tree.

Working with the Sync API

Developers interact with these primitives through the high-level facade in packages/computer/src/workspace.ts. The following examples demonstrate common operations:

// Initialize a sync round after reconnecting
import { tick, reconcileWatermarks } from "@cloudflare/computer/rpc";

async function syncRound(db: Database, remote: SyncRPC) {
  // Ensure watermarks align with remote state
  await reconcileWatermarks(db, remote);
  
  // Execute one incremental pull and push
  const { pulled, pushed } = await tick(db, remote);
  console.log(`Pulled ${pulled.applied} entries, pushed ${pushed} entries`);
}

For debugging or monitoring, you can inspect the current watermarks directly:

// Inspect synchronization state
async function logWatermarks(db: Database, remote: SyncRPC) {
  const localFetch = readFetchCursor(db);
  const localPush = readWatermark(db, "pushRev");
  const remoteWM = await remote.watermarks(); 
  // Returns { currentRev, pushRev, fetchCursor }

  console.info("Local fetch cursor:", localFetch);
  console.info("Local push watermark:", localPush);
  console.info("Remote watermarks:", remoteWM);
}

In rare cases requiring manual intervention, you can force a reset of the synchronization state:

// Force reset of divergent state
function forceReset(db: Database) {
  writeFetchCursor(db, { rev: 0, path: null });
  writeWatermark(db, "pushRev", 0);
}

Summary

• Sync cursors (ChangeCursor) track the last revision fetched from the remote, stored via readFetchCursor and writeFetchCursor in packages/rpc/src/sync-driver.ts.
• Watermarks track the highest pushed revision (pushRev) and the remote's acknowledgment state, enabling divergence detection across network boundaries.
• The incremental sync protocol processes only delta changes since the last cursor, using bounded batches of 256 entries to maintain constant memory usage.
• Automatic recovery via reconcileWatermarks and pullOnceImpl resets cursors to zero when divergence is detected, ensuring consistency without full-tree rescans.
• Blob deduplication via remote.hasObjects and stageBlob prevents redundant data transfer during synchronization rounds.

Frequently Asked Questions

What happens if the network connection drops during an active sync?

The synchronization protocol handles network interruptions gracefully. Because both sync cursors and watermarks are persisted to the local SQLite database immediately after each batch is applied, the driver can resume from the exact revision where it stopped. When the connection returns, reconcileWatermarks compares the local state with the remote's SyncRPC.watermarks() response. If the remote has fallen behind, the driver resets the appropriate cursor to zero and replays the missing changes without transmitting already-processed data.

How does Cloudflare Computer avoid re-transferring unchanged files?

The system implements blob-level deduplication during the pull phase. Before fetching file contents, the driver invokes remote.hasObjects to check which blobs the remote already possesses. Only missing objects are retrieved via stageBlob (as implemented in packages/rpc/src/sync-driver.ts, lines 114-124). Additionally, the sync cursor ensures that only ChangeEntry records after the last processed revision are streamed, eliminating redundant metadata transfer.

What is the difference between a sync cursor and a watermark?

A sync cursor (specifically the fetch cursor) tracks consumption progress on a single side—it marks the last revision the local database has pulled from the remote. A watermark, conversely, tracks acknowledgment across the network boundary: pushRev indicates the highest revision the local side has transmitted, while the remote's fetchCursor indicates what it has successfully applied from us. Watermarks enable bidirectional divergence detection, whereas cursors enable unidirectional incremental streaming.

Where are sync cursors and watermarks physically stored?

Both primitives are stored as scalar values within the local SQLite database backing the VFS. Fetch cursors are persisted through writeFetchCursor and retrieved via readFetchCursor, while watermarks use writeWatermark and readWatermark. These functions are defined in packages/rpc/src/sync-driver.ts (approximately lines 91-94) and operate on simple key-value rows within the database, ensuring durability across process restarts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}