How AuditScratchpad Provides Per‑Audit Crawl Scratchpad Functionality

The AuditScratchpad is a Cloudflare Durable Object that implements per‑audit crawl scratchpad functionality by creating an isolated SQLite‑backed instance for every unique auditId, ensuring complete data isolation between concurrent SEO audits while maintaining transient crawl state.

The Open SEO platform by every‑app uses the AuditScratchpad to manage the ephemeral state of website crawls without overloading the primary PostgreSQL database. By instantiating one Durable Object per audit, the system isolates crawl queues, link graphs, and page metadata, keeping workflow steps atomic and resource usage bounded.

Architecture of the Per‑Audit Scratchpad

The scratchpad architecture centers on a Cloudflare Durable Object class that encapsulates an entire audit's transient state in a local SQLite database.

The AuditScratchpad Durable Object Class

Located in src/server/features/audit/AuditScratchpad.ts, the AuditScratchpad class implements the DurableObject interface to provide a stateful, persistent compute instance. According to the source code at line 86, this class initializes three SQLite tables—frontier, links, and page_mirror—during construction to store crawl‑specific data.

The constructor also schedules a self‑cleanup mechanism via ensureCleanupAlarm (line 30), which sets a 7‑day alarm. If the audit never finalizes, this alarm triggers automatic database deletion to prevent storage leaks and unnecessary billing.

Database Schema and Storage Tables

The SQLite schema inside each scratchpad instance is optimized for crawl operations:

• Frontier table (line 90): Tracks the URL queue with columns for url, status (pending, leased, crawled), depth, source (either link or sitemap), and an in_sitemap boolean flag.
• Links table (line 100): Stores directed edges as source_page_id and target_url pairs, enabling post‑crawl analysis like broken‑link detection without querying the main database.
• Page‑mirror table (line 108): Maintains a lightweight copy of crawled page metadata including HTTP status codes, fetch classifications, and redirect targets.

To prevent platform limits from affecting crawl integrity, the implementation enforces a LINK_STORAGE_BUDGET_BYTES cap of approximately 500 MiB (line 84). Once the SQLite file reaches this threshold, the DO stops writing new link records while continuing to process frontier updates.

Per‑Audit Instantiation and Isolation

The system guarantees data isolation by binding exactly one Durable Object instance to each unique audit identifier.

The getAuditScratchpad Factory Function

To obtain a scratchpad for a specific audit, the codebase uses the getAuditScratchpad factory function defined at line 69:

// src/server/features/audit/AuditScratchpad.ts
export function getAuditScratchpad(auditId: string) {
  const namespace = env.AUDIT_SCRATCHPAD as unknown as DurableObjectNamespace<AuditScratchpad>;
  return namespace.get(namespace.idFromName(auditId));
}

This function derives a deterministic Durable Object ID from the auditId string using idFromName, ensuring that all operations for a given audit route to the same physical instance. This design naturally enforces per‑audit isolation: concurrent audits operate on separate SQLite databases within distinct Durable Objects, eliminating race conditions and data contamination.

Core API Methods for Crawl Lifecycle

The AuditScratchpad exposes a transactional RPC surface where each method executes synchronously within the Durable Object, providing atomic updates to the crawl state.

Seeding the Crawl

The seedStart and seedSitemapUrls methods populate the initial frontier. These methods insert the starting URL and optional sitemap URLs into the frontier table with appropriate depth and source markers.

Claiming Crawl Chunks

Workers fetch batches of URLs to crawl using claimChunk(chunkNo, limit). This method implements a leasing pattern: it atomically updates matching rows from pending to leased status and returns a ClaimedUrl[] array. The implementation guarantees idempotency per chunkNo, ensuring that retries do not duplicate work.

Recording Batch Results

After crawling, workers submit results via recordBatch, which accepts crawled URLs, page metadata, discovered links, and new frontier entries. This method updates the page_mirror table, writes edges to the links table (subject to storage budget), and enqueues newly discovered URLs into the frontier.

Finalization and Cleanup

When crawling completes, runFinalizeChecks queries the local SQLite database to identify broken links and orphan pages by analyzing the links and page_mirror tables. Finally, destroy() (or destroyForErasure() for GDPR compliance) removes the cleanup alarm and clears all storage, immediately freeing resources.

Resource Management and Self‑Cleanup

The scratchpad implements defensive mechanisms to prevent resource exhaustion and orphaned data.

• Automatic deletion: The ensureCleanupAlarm schedules a 7‑day deletion alarm during construction. If the audit workflow fails to complete or crashes, the Durable Object automatically wipes its SQLite database after this period.
• Storage budgeting: By monitoring LINK_STORAGE_BUDGET_BYTES, the system stops persisting link edges once the database approaches Cloudflare’s 1 GiB platform limit, ensuring the crawl can continue even when processing link‑heavy sites.

Practical Implementation Examples

Obtaining a Scratchpad for an Audit

import { getAuditScratchpad } from "@/server/features/audit/AuditScratchpad";

const auditId = "audit-123";
const scratchpad = getAuditScratchpad(auditId);
await scratchpad.seedStart("https://example.com");

Claiming a Batch of URLs

const chunkNo = 1;
const limit = 20;
const urls = await scratchpad.claimChunk(chunkNo, limit);
// Returns: [{ url: "https://example.com/page", depth: 0, inSitemap: false }, ...]

Recording Crawl Results

await scratchpad.recordBatch({
  crawledUrls: ["https://example.com/page1"],
  pages: [
    {
      pageId: "p1",
      url: "https://example.com/page1",
      statusCode: 200,
      fetchClass: "ok",
      redirectUrl: null,
    },
  ],
  links: [
    {
      sourcePageId: "p1",
      sourceUrl: "https://example.com/page1",
      targetUrl: "https://example.com/page2",
      anchor: "Read more",
      isNofollow: false,
    },
  ],
  discovered: [{ url: "https://example.com/page2", depth: 1 }],
});

Running Finalization Checks

const { brokenLinks, orphanPages } = await scratchpad.runFinalizeChecks({
  startUrl: "https://example.com",
  crawlCompleted: true,
});

Cleanup After Completion

// Normal completion path
await scratchpad.destroy();

// GDPR erasure flow
await scratchpad.destroyForErasure();

Integration with the Open SEO Workflow

The scratchpad integrates with the broader application through several key files:

• src/server/workflows/siteAuditWorkflowPhases.ts: Orchestrates crawl phases and instantiates the scratchpad via getAuditScratchpad.
• src/server/features/audit/services/AuditService.ts: High‑level service that initializes the scratchpad, executes the crawl workflow, runs final checks, and triggers destruction upon completion.
• src/server/gdpr/storage-erasure.ts: Handles compliance flows by invoking destroyForErasure to eliminate audit data on user request.

Summary

• Per‑audit isolation is achieved by binding one Cloudflare Durable Object instance to each unique auditId via getAuditScratchpad.
• SQLite‑backed storage in src/server/features/audit/AuditScratchpad.ts maintains the frontier queue, link graph, and page mirror locally to the DO instance.
• Transactional API methods (claimChunk, recordBatch, runFinalizeChecks) provide atomic updates to crawl state without hitting the primary PostgreSQL database.
• Automatic cleanup via 7‑day alarms and explicit destroy() calls prevents storage leaks and controls costs.
• Storage budgets (500 MiB link limit) ensure crawls survive on large sites without hitting platform limits.

Frequently Asked Questions

How does AuditScratchpad ensure data isolation between concurrent audits?

Each audit receives a unique Durable Object instance derived from its auditId using idFromName. Because Cloudflare Durable Objects provide separate storage namespaces per instance, concurrent audits operate on independent SQLite databases, preventing cross‑contamination of crawl queues or link graphs.

What happens if an audit crashes and never completes?

The ensureCleanupAlarm method schedules a 7‑day deletion alarm during scratchpad construction. If the audit workflow fails to call destroy(), this alarm automatically triggers and wipes the SQLite database, preventing orphaned storage and unexpected billing.

Why does the scratchpad use a 500 MiB storage budget for links?

The LINK_STORAGE_BUDGET_BYTES constant (approximately 500 MiB) acts as a safety margin below Cloudflare’s 1 GiB Durable Object storage limit. Once the SQLite file reaches this threshold, the scratchpad stops writing new link records while continuing to process the frontier, ensuring the crawl can complete even on sites with massive link graphs.

Can the scratchpad persist data beyond the crawl lifecycle?

No. The scratchpad is designed for transient crawl state only. Upon successful completion, AuditService explicitly calls destroy(), which deletes all tables and cancels cleanup alarms. For GDPR compliance, destroyForErasure() provides a separate path for immediate data removal.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}