How the Audit Scratchpad Durable Object Manages Crawl State in Open SEO

The audit scratchpad Durable Object is a SQLite-backed, per-audit instance that maintains the entire crawl state—including frontier URLs, in-flight claims, and processed results—providing atomic operations for claiming work chunks and resilient recovery from worker failures.

The every-app/open-seo repository implements a robust site auditing system using Cloudflare Durable Objects to orchestrate distributed crawling. At the heart of this system lies the audit scratchpad Durable Object, a per-audit instance that serves as the single source of truth for crawl progress. According to the source code in src/server/features/audit/AuditScratchpad.ts, this SQLite-backed object manages the complete lifecycle of a site audit from initial URL seeding through finalization.

Architecture and SQLite Storage Model

The audit scratchpad is instantiated once per unique auditId, creating isolated storage for each site audit. In src/server/features/audit/AuditScratchpad.ts, the Durable Object leverages a local SQLite database to guarantee ACID semantics for incremental updates while maintaining high throughput for concurrent crawling operations.

The object exposes a proxy interface via getAuditScratchpad(auditId), which returns a stub that serializes method calls to the specific Durable Object instance. This design ensures that all crawl state modifications—whether claiming work or recording results—execute atomically on the SQLite backend.

State Tables: Frontier, In-Flight, and Processed

The SQLite schema divides crawl state into three logical tables:

• frontier – URLs waiting to be crawled.
• in-flight – URLs currently claimed by a worker but not yet processed.
• processed – URLs already crawled with associated page metadata.

This separation enables safe concurrent claims through the claimChunk method and automatic reclamation of lost work via releaseUrls when workers fail.

Core API Methods for Crawl Management

The AuditScratchpad class provides explicit methods for each phase of the audit workflow, as implemented in src/server/features/audit/AuditScratchpad.ts.

Seeding and Initialization

seedStart(startUrl) inserts the initial audit origin into the frontier table. This method is invoked in src/server/workflows/siteAuditWorkflowPhases.ts when the audit begins.

seedSitemapUrls(urls) batches additional URLs from submitted sitemaps into the frontier, called immediately after seedStart in the same workflow phase.

import { getAuditScratchpad } from '@/server/features/audit/AuditScratchpad';

const scratchpad = getAuditScratchpad(auditId);

// Initialize the audit with the starting URL
await scratchpad.seedStart('https://example.com');

// Add URLs discovered from sitemap submission
await scratchpad.seedSitemapUrls([
  'https://example.com/page1',
  'https://example.com/page2'
]);

Distributed Work Claiming

claimChunk(chunkNo, claimLimit) is the primary mechanism for distributing work across crawl workers. Implemented in src/server/workflows/siteAuditWorkflowCrawl.ts, workers request a chunk of URLs (up to CHUNK_TARGET_PAGES) which the DO atomically moves from the frontier table to the in-flight table.

releaseUrls(urls) returns URLs to the frontier when a worker encounters network errors or crashes, ensuring no URLs are lost during distributed processing.

// Worker requests a batch of URLs to crawl
const { urls } = await scratchpad.claimChunk(0, 50);

try {
  const results = await crawlUrls(urls);
  await scratchpad.recordBatch(results);
} catch (error) {
  // Return unprocessed URLs to the queue
  await scratchpad.releaseUrls(urls);
}

Result Persistence and Statistics

recordBatch(results) persists crawl results—including discovered URLs and page data—updating the processed table and inserting new entries into the frontier for newly discovered pages. This method runs after each worker completes its chunk in src/server/workflows/siteAuditWorkflowCrawl.ts.

getStats() reads aggregate statistics (pages discovered, crawled, errors) from the SQLite database, providing real-time progress updates to the UI throughout the workflow.

// Persist crawl results and update counters
await scratchpad.recordBatch(crawlResults);

// Query live statistics for progress reporting
const stats = await scratchpad.getStats();
console.log(`Discovered: ${stats.discoveredPages}, Crawled: ${stats.crawledPages}`);

Finalization and Cleanup

runFinalizeChecks() executes after the frontier empties and all in-flight URLs return. This method, called from src/server/workflows/siteAuditWorkflowPhases.ts, performs static analysis including broken-link detection and orphan-page identification, returning structured results.

destroy() (implicit) triggers when the audit is deleted via src/server/features/audit/services/AuditService.ts, destroying the Durable Object instance and freeing its SQLite storage.

// Run final validation once crawling completes
const { brokenLinks, orphanPages } = await scratchpad.runFinalizeChecks();

Implementation Example

The following workflow demonstrates the complete interaction pattern between crawl workers and the scratchpad Durable Object:

import { getAuditScratchpad } from '@/server/features/audit/AuditScratchpad';

async function runDistributedAudit(auditId: string, startUrl: string) {
  const scratchpad = getAuditScratchpad(auditId);
  
  // Phase 1: Seeding
  await scratchpad.seedStart(startUrl);
  
  // Phase 2: Distributed crawling
  const workerId = 0;
  while (true) {
    const { urls } = await scratchpad.claimChunk(workerId, 50);
    if (urls.length === 0) break;
    
    try {
      const results = await fetchAndParseUrls(urls);
      await scratchpad.recordBatch(results);
    } catch (error) {
      await scratchpad.releaseUrls(urls);
    }
  }
  
  // Phase 3: Finalization
  const finalReport = await scratchpad.runFinalizeChecks();
  return finalReport;
}

Summary

• The audit scratchpad Durable Object in src/server/features/audit/AuditScratchpad.ts provides a SQLite-backed, per-audit instance that serves as the authoritative source for crawl state.
• Three logical tables—frontier, in-flight, and processed—enable resilient work distribution and recovery from worker failures.
• The claimChunk and recordBatch methods coordinate distributed crawling in src/server/workflows/siteAuditWorkflowCrawl.ts, while runFinalizeChecks handles post-crawl analysis.
• Real-time statistics via getStats support UI progress tracking throughout the audit lifecycle.

Frequently Asked Questions

How does the audit scratchpad handle worker crashes during crawling?

When a worker fails to process a batch of URLs, the releaseUrls method returns those URLs from the in-flight table back to the frontier table. Because the Durable Object maintains the in-flight state separately from processed, unclaimed work automatically becomes available for other workers to claim via subsequent claimChunk calls, ensuring no URLs are lost to transient failures.

Why is SQLite used instead of a key-value store for the crawl state?

According to the implementation in src/server/features/audit/AuditScratchpad.ts, SQLite provides ACID semantics necessary for atomic operations like claimChunk, which must simultaneously read the frontier, move URLs to in-flight, and commit the transaction without race conditions. The relational structure also enables efficient queries for statistics and finalization checks that would require multiple KV lookups or complex coordination logic.

How is the Durable Object instance addressed for a specific audit?

The system uses the getAuditScratchpad(auditId) factory function to obtain a proxy stub. This stub routes method calls to the specific Durable Object instance identified by the auditId parameter, ensuring that all workers and workflow phases interact with the same SQLite database file for a given site audit.

What happens to the SQLite data when an audit is deleted?

The destroy lifecycle hook (implicitly invoked via src/server/features/audit/services/AuditService.ts) destroys the Durable Object instance, which drops the associated SQLite storage. While the cleanup is best-effort, this mechanism prevents storage accumulation for completed or cancelled audits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}