# Sync Protocol Building Blocks in @cloudflare/dofs: A Complete Guide

> Explore the @cloudflare/dofs sync protocol's eleven modules like watermarks, push, and fetch for incremental, resumable synchronization with a remote SQLite backend. Master local filesystem syncing.

- Repository: [Cloudflare/computer](https://github.com/cloudflare/computer)
- Tags: deep-dive
- Published: 2026-09-04

---

**The @cloudflare/dofs sync protocol consists of eleven composable modules—including watermarks, push, fetch, apply, and coalesce—that enable incremental, resumable synchronization between local filesystem views and a remote SQLite backend.**

The @cloudflare/dofs package, part of the cloudflare/computer repository, implements a robust sync protocol through a collection of focused, composable modules under `packages/dofs/src/sync`. These building blocks handle everything from change detection and path normalization to manifest generation and invariant enforcement, creating a reliable pipeline for bidirectional data synchronization.

## Core Sync Protocol Modules

The sync implementation is deliberately modular, with each file in `packages/dofs/src/sync/` providing a single, testable responsibility.

### State Tracking and Validation

- **watermarks.ts** – Tracks the last-known sync offsets (both inbound and outbound) to allow resumable, incremental pushes and pulls. Watermarks are persisted in the local database and updated atomically with each successful sync operation.
- **invariant.ts** – Enforces protocol-level invariants (e.g., monotonic watermarks, valid change ordering) before applying any update. This prevents state corruption by rejecting malformed or out-of-sequence sync messages.

### Change Representation and Optimization

- **changes.ts** – Provides a typed model of individual change records (create, modify, delete, chmod, etc.) used throughout the protocol. Each change is strongly typed to ensure consistent handling across the pipeline.
- **manifests.ts** – Describes a batch of changes in a deterministic JSON manifest, including file creations, deletions, and metadata updates. Manifests serve as the canonical exchange format between client and server.
- **coalesce.ts** – Merges overlapping or adjacent change sets into a minimal representation to reduce transfer volume. This optimization prevents redundant updates when multiple operations affect the same file.

### Data Transfer and Storage

- **push.ts** – Orchestrates a client-initiated push: gathers local changes, packages them into a manifest, and sends them to the remote side. The push module coordinates with watermarks to ensure incremental transmission.
- **fetch.ts** – Retrieves remote state (blobs, manifests, or incremental deltas) needed for a pull operation. It handles HTTP negotiation and streaming responses for efficient large-file handling.
- **blobs.ts** – Handles binary data (file contents) as streamed blobs, including chunking, checksum verification, and deduplication. This module separates content storage from metadata synchronization.

### Path Handling and Filtering

- **paths.ts** – Normalizes and validates filesystem paths that appear in sync messages, handling case-sensitivity and symbolic-link resolution. Consistent path normalization prevents discrepancies between different operating systems.
- **ignore.ts** – Implements the ignore-rules language used by the sync client to filter out paths that should not be synchronized. It supports glob patterns and directory-specific rules similar to `.gitignore`.

### State Application

- **apply.ts** – Consumes a manifest (and any associated blobs) and applies the changes to the local SQLite database and in-memory filesystem cache. This module represents the final step in the inbound sync pipeline.

## How the Sync Pipeline Works

According to the cloudflare/computer source code, these modules form a deterministic five-stage pipeline:

1. **Detect** local modifications using filesystem watchers, then filter them through **paths** and **ignore** to exclude non-syncable files.
2. **Encode** the modifications as typed **changes**, then assemble them into a **manifest** describing the batch.
3. **Coalesce** overlapping changes to minimize payload size, then **push** prepares the incremental payload while updating **watermarks** to track the sync point.
4. **Transfer** blobs via **fetch** when the remote side needs actual file contents, using **blobs** for chunked streaming.
5. **Validate** incoming changes with **invariant** checks, then **apply** updates the local state and advances the inbound **watermarks**.

This design keeps the sync logic side-effect-free, testable, and easy to extend (e.g., adding new change types or custom ignore rules).

## Implementation Examples

Below are minimal snippets illustrating how consumers interact with these building blocks.

### Performing a Client Push

```typescript
import { push } from '@cloudflare/dofs/src/sync/push';
import { watermarks } from '@cloudflare/dofs/src/sync/watermarks';
import { createManifest } from '@cloudflare/dofs/src/sync/manifests';

// Assume `db` is a Dofs Database instance and `fs` a FileSystem wrapper.
async function performPush() {
  // 1️⃣ Get the last successful outbound watermark.
  const last = await watermarks.getOutbound(db);

  // 2️⃣ Collect local changes since that watermark.
  const changes = await db.listChangesSince(last);

  // 3️⃣ Build a manifest describing those changes.
  const manifest = createManifest(changes);

  // 4️⃣ Push the manifest (and any required blobs) to the remote.
  await push(db, manifest);

  // 5️⃣ Update the outbound watermark on success.
  await watermarks.setOutbound(db, manifest.watermark);
}

```

### Pulling Remote Changes

```typescript
import { fetch } from '@cloudflare/dofs/src/sync/fetch';
import { apply } from '@cloudflare/dofs/src/sync/apply';
import { watermarks } from '@cloudflare/dofs/src/sync/watermarks';

async function performPull(remoteUrl: string) {
  // 1️⃣ Retrieve the last inbound watermark.
  const last = await watermarks.getInbound(db);

  // 2️⃣ Ask the remote side for any changes newer than `last`.
  const { manifest, blobs } = await fetch(remoteUrl, { since: last });

  // 3️⃣ Validate the manifest against protocol invariants.
  // (apply internally runs `invariant` checks.)

  // 4️⃣ Apply the incoming changes to the local state.
  await apply(db, manifest, blobs);

  // 5️⃣ Record the new inbound watermark.
  await watermarks.setInbound(db, manifest.watermark);
}

```

### Configuring Ignore Rules

```typescript
import { ignore } from '@cloudflare/dofs/src/sync/ignore';
import { paths } from '@cloudflare/dofs/src/sync/paths';

// Define a simple ignore rule set.
const ignoreRules = ignore.parse(['node_modules', '*.log']);

function shouldSync(filePath: string): boolean {
  const normalized = paths.normalize(filePath);
  return !ignoreRules.test(normalized);
}

```

## Summary

- The **@cloudflare/dofs** sync protocol is composed of eleven specialized modules located in `packages/dofs/src/sync/`.
- **Watermarks** enable resumable synchronization by tracking incremental sync offsets in both directions.
- The pipeline moves from change detection (**paths**, **ignore**) through encoding (**changes**, **manifests**) and optimization (**coalesce**) to transmission (**push**, **fetch**, **blobs**) and final application (**apply**, **invariant**).
- Each module is independently testable, with corresponding unit tests like [`watermarks.test.ts`](https://github.com/cloudflare/computer/blob/main/watermarks.test.ts) and [`apply.test.ts`](https://github.com/cloudflare/computer/blob/main/apply.test.ts) ensuring protocol reliability.
- Typed change records and deterministic manifests ensure consistent behavior across different platforms and filesystems.

## Frequently Asked Questions

### What are watermarks in @cloudflare/dofs?

Watermarks are persistent counters stored in the local SQLite database that track the last successfully synchronized offset for both inbound and outbound operations. As implemented in [`packages/dofs/src/sync/watermarks.ts`](https://github.com/cloudflare/computer/blob/main/packages/dofs/src/sync/watermarks.ts), they allow the sync protocol to resume interrupted transfers and avoid re-sending or re-processing already-synced changes, making synchronization incremental and efficient.

### How does the coalesce module optimize sync performance?

The **coalesce** module, found in [`packages/dofs/src/sync/coalesce.ts`](https://github.com/cloudflare/computer/blob/main/packages/dofs/src/sync/coalesce.ts), merges overlapping or adjacent change sets into a minimal representation before transmission. For example, if a file is modified twice between syncs, coalesce reduces this to a single change record, significantly reducing network payload and processing overhead during high-frequency file operations.

### What invariants does the sync protocol enforce?

According to [`packages/dofs/src/sync/invariant.ts`](https://github.com/cloudflare/computer/blob/main/packages/dofs/src/sync/invariant.ts), the protocol enforces critical safety checks including monotonic watermark progression (ensuring sync offsets never move backwards) and valid change ordering (preventing deletion-before-creation sequences). These checks run on both client and server sides before any changes are applied to prevent state corruption.

### How are binary files handled during synchronization?

Binary content is managed separately from metadata through the **blobs** module in [`packages/dofs/src/sync/blobs.ts`](https://github.com/cloudflare/computer/blob/main/packages/dofs/src/sync/blobs.ts). Rather than embedding file contents directly in manifests, the protocol streams binary data as chunked blobs with checksum verification and deduplication. This approach keeps manifests lightweight JSON while supporting efficient transfer of large files via the **fetch** and **push** modules.