Sync Protocol Building Blocks in @cloudflare/dofs: A Complete Guide

The @cloudflare/dofs sync protocol consists of eleven composable modules—including watermarks, push, fetch, apply, and coalesce—that enable incremental, resumable synchronization between local filesystem views and a remote SQLite backend.

The @cloudflare/dofs package, part of the cloudflare/computer repository, implements a robust sync protocol through a collection of focused, composable modules under packages/dofs/src/sync. These building blocks handle everything from change detection and path normalization to manifest generation and invariant enforcement, creating a reliable pipeline for bidirectional data synchronization.

Core Sync Protocol Modules

The sync implementation is deliberately modular, with each file in packages/dofs/src/sync/ providing a single, testable responsibility.

State Tracking and Validation

• watermarks.ts – Tracks the last-known sync offsets (both inbound and outbound) to allow resumable, incremental pushes and pulls. Watermarks are persisted in the local database and updated atomically with each successful sync operation.
• invariant.ts – Enforces protocol-level invariants (e.g., monotonic watermarks, valid change ordering) before applying any update. This prevents state corruption by rejecting malformed or out-of-sequence sync messages.

Change Representation and Optimization

• changes.ts – Provides a typed model of individual change records (create, modify, delete, chmod, etc.) used throughout the protocol. Each change is strongly typed to ensure consistent handling across the pipeline.
• manifests.ts – Describes a batch of changes in a deterministic JSON manifest, including file creations, deletions, and metadata updates. Manifests serve as the canonical exchange format between client and server.
• coalesce.ts – Merges overlapping or adjacent change sets into a minimal representation to reduce transfer volume. This optimization prevents redundant updates when multiple operations affect the same file.

Data Transfer and Storage

• push.ts – Orchestrates a client-initiated push: gathers local changes, packages them into a manifest, and sends them to the remote side. The push module coordinates with watermarks to ensure incremental transmission.
• fetch.ts – Retrieves remote state (blobs, manifests, or incremental deltas) needed for a pull operation. It handles HTTP negotiation and streaming responses for efficient large-file handling.
• blobs.ts – Handles binary data (file contents) as streamed blobs, including chunking, checksum verification, and deduplication. This module separates content storage from metadata synchronization.

Path Handling and Filtering

• paths.ts – Normalizes and validates filesystem paths that appear in sync messages, handling case-sensitivity and symbolic-link resolution. Consistent path normalization prevents discrepancies between different operating systems.
• ignore.ts – Implements the ignore-rules language used by the sync client to filter out paths that should not be synchronized. It supports glob patterns and directory-specific rules similar to .gitignore.

State Application

• apply.ts – Consumes a manifest (and any associated blobs) and applies the changes to the local SQLite database and in-memory filesystem cache. This module represents the final step in the inbound sync pipeline.

How the Sync Pipeline Works

According to the cloudflare/computer source code, these modules form a deterministic five-stage pipeline:

1. Detect local modifications using filesystem watchers, then filter them through paths and ignore to exclude non-syncable files.
2. Encode the modifications as typed changes, then assemble them into a manifest describing the batch.
3. Coalesce overlapping changes to minimize payload size, then push prepares the incremental payload while updating watermarks to track the sync point.
4. Transfer blobs via fetch when the remote side needs actual file contents, using blobs for chunked streaming.
5. Validate incoming changes with invariant checks, then apply updates the local state and advances the inbound watermarks.

This design keeps the sync logic side-effect-free, testable, and easy to extend (e.g., adding new change types or custom ignore rules).

Implementation Examples

Below are minimal snippets illustrating how consumers interact with these building blocks.

Performing a Client Push

import { push } from '@cloudflare/dofs/src/sync/push';
import { watermarks } from '@cloudflare/dofs/src/sync/watermarks';
import { createManifest } from '@cloudflare/dofs/src/sync/manifests';

// Assume `db` is a Dofs Database instance and `fs` a FileSystem wrapper.
async function performPush() {
  // 1️⃣ Get the last successful outbound watermark.
  const last = await watermarks.getOutbound(db);

  // 2️⃣ Collect local changes since that watermark.
  const changes = await db.listChangesSince(last);

  // 3️⃣ Build a manifest describing those changes.
  const manifest = createManifest(changes);

  // 4️⃣ Push the manifest (and any required blobs) to the remote.
  await push(db, manifest);

  // 5️⃣ Update the outbound watermark on success.
  await watermarks.setOutbound(db, manifest.watermark);
}

Pulling Remote Changes

import { fetch } from '@cloudflare/dofs/src/sync/fetch';
import { apply } from '@cloudflare/dofs/src/sync/apply';
import { watermarks } from '@cloudflare/dofs/src/sync/watermarks';

async function performPull(remoteUrl: string) {
  // 1️⃣ Retrieve the last inbound watermark.
  const last = await watermarks.getInbound(db);

  // 2️⃣ Ask the remote side for any changes newer than `last`.
  const { manifest, blobs } = await fetch(remoteUrl, { since: last });

  // 3️⃣ Validate the manifest against protocol invariants.
  // (apply internally runs `invariant` checks.)

  // 4️⃣ Apply the incoming changes to the local state.
  await apply(db, manifest, blobs);

  // 5️⃣ Record the new inbound watermark.
  await watermarks.setInbound(db, manifest.watermark);
}

Configuring Ignore Rules

import { ignore } from '@cloudflare/dofs/src/sync/ignore';
import { paths } from '@cloudflare/dofs/src/sync/paths';

// Define a simple ignore rule set.
const ignoreRules = ignore.parse(['node_modules', '*.log']);

function shouldSync(filePath: string): boolean {
  const normalized = paths.normalize(filePath);
  return !ignoreRules.test(normalized);
}

Summary

• The @cloudflare/dofs sync protocol is composed of eleven specialized modules located in packages/dofs/src/sync/.
• Watermarks enable resumable synchronization by tracking incremental sync offsets in both directions.
• The pipeline moves from change detection (paths, ignore) through encoding (changes, manifests) and optimization (coalesce) to transmission (push, fetch, blobs) and final application (apply, invariant).
• Each module is independently testable, with corresponding unit tests like watermarks.test.ts and apply.test.ts ensuring protocol reliability.
• Typed change records and deterministic manifests ensure consistent behavior across different platforms and filesystems.

Frequently Asked Questions

What are watermarks in @cloudflare/dofs?

Watermarks are persistent counters stored in the local SQLite database that track the last successfully synchronized offset for both inbound and outbound operations. As implemented in packages/dofs/src/sync/watermarks.ts, they allow the sync protocol to resume interrupted transfers and avoid re-sending or re-processing already-synced changes, making synchronization incremental and efficient.

How does the coalesce module optimize sync performance?

The coalesce module, found in packages/dofs/src/sync/coalesce.ts, merges overlapping or adjacent change sets into a minimal representation before transmission. For example, if a file is modified twice between syncs, coalesce reduces this to a single change record, significantly reducing network payload and processing overhead during high-frequency file operations.

What invariants does the sync protocol enforce?

According to packages/dofs/src/sync/invariant.ts, the protocol enforces critical safety checks including monotonic watermark progression (ensuring sync offsets never move backwards) and valid change ordering (preventing deletion-before-creation sequences). These checks run on both client and server sides before any changes are applied to prevent state corruption.

How are binary files handled during synchronization?

Binary content is managed separately from metadata through the blobs module in packages/dofs/src/sync/blobs.ts. Rather than embedding file contents directly in manifests, the protocol streams binary data as chunked blobs with checksum verification and deduplication. This approach keeps manifests lightweight JSON while supporting efficient transfer of large files via the fetch and push modules.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}