# How AuditScratchpad Provides Per‑Audit Crawl Scratchpad Functionality

> Discover how AuditScratchpad offers per audit crawl scratchpad functionality using Cloudflare Durable Objects and isolated SQLite instances for secure, concurrent SEO audits.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-19

---

**The AuditScratchpad is a Cloudflare Durable Object that implements per‑audit crawl scratchpad functionality by creating an isolated SQLite‑backed instance for every unique `auditId`, ensuring complete data isolation between concurrent SEO audits while maintaining transient crawl state.**

The Open SEO platform by every‑app uses the `AuditScratchpad` to manage the ephemeral state of website crawls without overloading the primary PostgreSQL database. By instantiating one Durable Object per audit, the system isolates crawl queues, link graphs, and page metadata, keeping workflow steps atomic and resource usage bounded.

## Architecture of the Per‑Audit Scratchpad

The scratchpad architecture centers on a Cloudflare Durable Object class that encapsulates an entire audit's transient state in a local SQLite database.

### The AuditScratchpad Durable Object Class

Located in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts), the `AuditScratchpad` class implements the `DurableObject` interface to provide a stateful, persistent compute instance. According to the source code at line 86, this class initializes three SQLite tables—`frontier`, `links`, and `page_mirror`—during construction to store crawl‑specific data.

The constructor also schedules a self‑cleanup mechanism via `ensureCleanupAlarm` (line 30), which sets a 7‑day alarm. If the audit never finalizes, this alarm triggers automatic database deletion to prevent storage leaks and unnecessary billing.

### Database Schema and Storage Tables

The SQLite schema inside each scratchpad instance is optimized for crawl operations:

- **Frontier table** (line 90): Tracks the URL queue with columns for `url`, status (`pending`, `leased`, `crawled`), `depth`, `source` (either `link` or `sitemap`), and an `in_sitemap` boolean flag.
- **Links table** (line 100): Stores directed edges as `source_page_id` and `target_url` pairs, enabling post‑crawl analysis like broken‑link detection without querying the main database.
- **Page‑mirror table** (line 108): Maintains a lightweight copy of crawled page metadata including HTTP status codes, fetch classifications, and redirect targets.

To prevent platform limits from affecting crawl integrity, the implementation enforces a `LINK_STORAGE_BUDGET_BYTES` cap of approximately 500 MiB (line 84). Once the SQLite file reaches this threshold, the DO stops writing new link records while continuing to process frontier updates.

## Per‑Audit Instantiation and Isolation

The system guarantees data isolation by binding exactly one Durable Object instance to each unique audit identifier.

### The getAuditScratchpad Factory Function

To obtain a scratchpad for a specific audit, the codebase uses the `getAuditScratchpad` factory function defined at line 69:

```typescript
// src/server/features/audit/AuditScratchpad.ts
export function getAuditScratchpad(auditId: string) {
  const namespace = env.AUDIT_SCRATCHPAD as unknown as DurableObjectNamespace<AuditScratchpad>;
  return namespace.get(namespace.idFromName(auditId));
}

```

This function derives a deterministic Durable Object ID from the `auditId` string using `idFromName`, ensuring that all operations for a given audit route to the same physical instance. This design naturally enforces **per‑audit isolation**: concurrent audits operate on separate SQLite databases within distinct Durable Objects, eliminating race conditions and data contamination.

## Core API Methods for Crawl Lifecycle

The `AuditScratchpad` exposes a transactional RPC surface where each method executes synchronously within the Durable Object, providing atomic updates to the crawl state.

### Seeding the Crawl

The `seedStart` and `seedSitemapUrls` methods populate the initial frontier. These methods insert the starting URL and optional sitemap URLs into the `frontier` table with appropriate depth and source markers.

### Claiming Crawl Chunks

Workers fetch batches of URLs to crawl using `claimChunk(chunkNo, limit)`. This method implements a leasing pattern: it atomically updates matching rows from `pending` to `leased` status and returns a `ClaimedUrl[]` array. The implementation guarantees idempotency per `chunkNo`, ensuring that retries do not duplicate work.

### Recording Batch Results

After crawling, workers submit results via `recordBatch`, which accepts crawled URLs, page metadata, discovered links, and new frontier entries. This method updates the `page_mirror` table, writes edges to the `links` table (subject to storage budget), and enqueues newly discovered URLs into the `frontier`.

### Finalization and Cleanup

When crawling completes, `runFinalizeChecks` queries the local SQLite database to identify broken links and orphan pages by analyzing the `links` and `page_mirror` tables. Finally, `destroy()` (or `destroyForErasure()` for GDPR compliance) removes the cleanup alarm and clears all storage, immediately freeing resources.

## Resource Management and Self‑Cleanup

The scratchpad implements defensive mechanisms to prevent resource exhaustion and orphaned data.

- **Automatic deletion**: The `ensureCleanupAlarm` schedules a 7‑day deletion alarm during construction. If the audit workflow fails to complete or crashes, the Durable Object automatically wipes its SQLite database after this period.
- **Storage budgeting**: By monitoring `LINK_STORAGE_BUDGET_BYTES`, the system stops persisting link edges once the database approaches Cloudflare’s 1 GiB platform limit, ensuring the crawl can continue even when processing link‑heavy sites.

## Practical Implementation Examples

### Obtaining a Scratchpad for an Audit

```typescript
import { getAuditScratchpad } from "@/server/features/audit/AuditScratchpad";

const auditId = "audit-123";
const scratchpad = getAuditScratchpad(auditId);
await scratchpad.seedStart("https://example.com");

```

### Claiming a Batch of URLs

```typescript
const chunkNo = 1;
const limit = 20;
const urls = await scratchpad.claimChunk(chunkNo, limit);
// Returns: [{ url: "https://example.com/page", depth: 0, inSitemap: false }, ...]

```

### Recording Crawl Results

```typescript
await scratchpad.recordBatch({
  crawledUrls: ["https://example.com/page1"],
  pages: [
    {
      pageId: "p1",
      url: "https://example.com/page1",
      statusCode: 200,
      fetchClass: "ok",
      redirectUrl: null,
    },
  ],
  links: [
    {
      sourcePageId: "p1",
      sourceUrl: "https://example.com/page1",
      targetUrl: "https://example.com/page2",
      anchor: "Read more",
      isNofollow: false,
    },
  ],
  discovered: [{ url: "https://example.com/page2", depth: 1 }],
});

```

### Running Finalization Checks

```typescript
const { brokenLinks, orphanPages } = await scratchpad.runFinalizeChecks({
  startUrl: "https://example.com",
  crawlCompleted: true,
});

```

### Cleanup After Completion

```typescript
// Normal completion path
await scratchpad.destroy();

// GDPR erasure flow
await scratchpad.destroyForErasure();

```

## Integration with the Open SEO Workflow

The scratchpad integrates with the broader application through several key files:

- **[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)**: Orchestrates crawl phases and instantiates the scratchpad via `getAuditScratchpad`.
- **[`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)**: High‑level service that initializes the scratchpad, executes the crawl workflow, runs final checks, and triggers destruction upon completion.
- **[`src/server/gdpr/storage-erasure.ts`](https://github.com/every-app/open-seo/blob/main/src/server/gdpr/storage-erasure.ts)**: Handles compliance flows by invoking `destroyForErasure` to eliminate audit data on user request.

## Summary

- **Per‑audit isolation** is achieved by binding one Cloudflare Durable Object instance to each unique `auditId` via `getAuditScratchpad`.
- **SQLite‑backed storage** in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts) maintains the frontier queue, link graph, and page mirror locally to the DO instance.
- **Transactional API** methods (`claimChunk`, `recordBatch`, `runFinalizeChecks`) provide atomic updates to crawl state without hitting the primary PostgreSQL database.
- **Automatic cleanup** via 7‑day alarms and explicit `destroy()` calls prevents storage leaks and controls costs.
- **Storage budgets** (500 MiB link limit) ensure crawls survive on large sites without hitting platform limits.

## Frequently Asked Questions

### How does AuditScratchpad ensure data isolation between concurrent audits?

Each audit receives a unique Durable Object instance derived from its `auditId` using `idFromName`. Because Cloudflare Durable Objects provide separate storage namespaces per instance, concurrent audits operate on independent SQLite databases, preventing cross‑contamination of crawl queues or link graphs.

### What happens if an audit crashes and never completes?

The `ensureCleanupAlarm` method schedules a 7‑day deletion alarm during scratchpad construction. If the audit workflow fails to call `destroy()`, this alarm automatically triggers and wipes the SQLite database, preventing orphaned storage and unexpected billing.

### Why does the scratchpad use a 500 MiB storage budget for links?

The `LINK_STORAGE_BUDGET_BYTES` constant (approximately 500 MiB) acts as a safety margin below Cloudflare’s 1 GiB Durable Object storage limit. Once the SQLite file reaches this threshold, the scratchpad stops writing new link records while continuing to process the frontier, ensuring the crawl can complete even on sites with massive link graphs.

### Can the scratchpad persist data beyond the crawl lifecycle?

No. The scratchpad is designed for transient crawl state only. Upon successful completion, `AuditService` explicitly calls `destroy()`, which deletes all tables and cancels cleanup alarms. For GDPR compliance, `destroyForErasure()` provides a separate path for immediate data removal.