# How the Audit Scratchpad Durable Object Manages Crawl State in Open SEO

> Discover how the audit scratchpad Durable Object in Open SEO manages crawl state with SQLite. Learn about atomic operations and resilient recovery for efficient crawling.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-17

---

**The audit scratchpad Durable Object is a SQLite-backed, per-audit instance that maintains the entire crawl state—including frontier URLs, in-flight claims, and processed results—providing atomic operations for claiming work chunks and resilient recovery from worker failures.**

The every-app/open-seo repository implements a robust site auditing system using Cloudflare Durable Objects to orchestrate distributed crawling. At the heart of this system lies the **audit scratchpad Durable Object**, a per-audit instance that serves as the single source of truth for crawl progress. According to the source code in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts), this SQLite-backed object manages the complete lifecycle of a site audit from initial URL seeding through finalization.

## Architecture and SQLite Storage Model

The audit scratchpad is instantiated once per unique `auditId`, creating isolated storage for each site audit. In [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts), the Durable Object leverages a local SQLite database to guarantee ACID semantics for incremental updates while maintaining high throughput for concurrent crawling operations.

The object exposes a proxy interface via `getAuditScratchpad(auditId)`, which returns a stub that serializes method calls to the specific Durable Object instance. This design ensures that all crawl state modifications—whether claiming work or recording results—execute atomically on the SQLite backend.

### State Tables: Frontier, In-Flight, and Processed

The SQLite schema divides crawl state into three logical tables:

- **frontier** – URLs waiting to be crawled.
- **in-flight** – URLs currently claimed by a worker but not yet processed.
- **processed** – URLs already crawled with associated page metadata.

This separation enables safe concurrent claims through the `claimChunk` method and automatic reclamation of lost work via `releaseUrls` when workers fail.

## Core API Methods for Crawl Management

The `AuditScratchpad` class provides explicit methods for each phase of the audit workflow, as implemented in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts).

### Seeding and Initialization

**`seedStart(startUrl)`** inserts the initial audit origin into the frontier table. This method is invoked in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) when the audit begins.

**`seedSitemapUrls(urls)`** batches additional URLs from submitted sitemaps into the frontier, called immediately after `seedStart` in the same workflow phase.

```typescript
import { getAuditScratchpad } from '@/server/features/audit/AuditScratchpad';

const scratchpad = getAuditScratchpad(auditId);

// Initialize the audit with the starting URL
await scratchpad.seedStart('https://example.com');

// Add URLs discovered from sitemap submission
await scratchpad.seedSitemapUrls([
  'https://example.com/page1',
  'https://example.com/page2'
]);

```

### Distributed Work Claiming

**`claimChunk(chunkNo, claimLimit)`** is the primary mechanism for distributing work across crawl workers. Implemented in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), workers request a chunk of URLs (up to `CHUNK_TARGET_PAGES`) which the DO atomically moves from the *frontier* table to the *in-flight* table.

**`releaseUrls(urls)`** returns URLs to the frontier when a worker encounters network errors or crashes, ensuring no URLs are lost during distributed processing.

```typescript
// Worker requests a batch of URLs to crawl
const { urls } = await scratchpad.claimChunk(0, 50);

try {
  const results = await crawlUrls(urls);
  await scratchpad.recordBatch(results);
} catch (error) {
  // Return unprocessed URLs to the queue
  await scratchpad.releaseUrls(urls);
}

```

### Result Persistence and Statistics

**`recordBatch(results)`** persists crawl results—including discovered URLs and page data—updating the *processed* table and inserting new entries into the *frontier* for newly discovered pages. This method runs after each worker completes its chunk in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts).

**`getStats()`** reads aggregate statistics (pages discovered, crawled, errors) from the SQLite database, providing real-time progress updates to the UI throughout the workflow.

```typescript
// Persist crawl results and update counters
await scratchpad.recordBatch(crawlResults);

// Query live statistics for progress reporting
const stats = await scratchpad.getStats();
console.log(`Discovered: ${stats.discoveredPages}, Crawled: ${stats.crawledPages}`);

```

### Finalization and Cleanup

**`runFinalizeChecks()`** executes after the frontier empties and all in-flight URLs return. This method, called from [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), performs static analysis including broken-link detection and orphan-page identification, returning structured results.

**`destroy()`** (implicit) triggers when the audit is deleted via [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts), destroying the Durable Object instance and freeing its SQLite storage.

```typescript
// Run final validation once crawling completes
const { brokenLinks, orphanPages } = await scratchpad.runFinalizeChecks();

```

## Implementation Example

The following workflow demonstrates the complete interaction pattern between crawl workers and the scratchpad Durable Object:

```typescript
import { getAuditScratchpad } from '@/server/features/audit/AuditScratchpad';

async function runDistributedAudit(auditId: string, startUrl: string) {
  const scratchpad = getAuditScratchpad(auditId);
  
  // Phase 1: Seeding
  await scratchpad.seedStart(startUrl);
  
  // Phase 2: Distributed crawling
  const workerId = 0;
  while (true) {
    const { urls } = await scratchpad.claimChunk(workerId, 50);
    if (urls.length === 0) break;
    
    try {
      const results = await fetchAndParseUrls(urls);
      await scratchpad.recordBatch(results);
    } catch (error) {
      await scratchpad.releaseUrls(urls);
    }
  }
  
  // Phase 3: Finalization
  const finalReport = await scratchpad.runFinalizeChecks();
  return finalReport;
}

```

## Summary

- The **audit scratchpad Durable Object** in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts) provides a SQLite-backed, per-audit instance that serves as the authoritative source for crawl state.
- Three logical tables—**frontier**, **in-flight**, and **processed**—enable resilient work distribution and recovery from worker failures.
- The **`claimChunk`** and **`recordBatch`** methods coordinate distributed crawling in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), while **`runFinalizeChecks`** handles post-crawl analysis.
- Real-time statistics via **`getStats`** support UI progress tracking throughout the audit lifecycle.

## Frequently Asked Questions

### How does the audit scratchpad handle worker crashes during crawling?

When a worker fails to process a batch of URLs, the **`releaseUrls`** method returns those URLs from the *in-flight* table back to the *frontier* table. Because the Durable Object maintains the *in-flight* state separately from *processed*, unclaimed work automatically becomes available for other workers to claim via subsequent **`claimChunk`** calls, ensuring no URLs are lost to transient failures.

### Why is SQLite used instead of a key-value store for the crawl state?

According to the implementation in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts), SQLite provides **ACID semantics** necessary for atomic operations like `claimChunk`, which must simultaneously read the frontier, move URLs to in-flight, and commit the transaction without race conditions. The relational structure also enables efficient queries for statistics and finalization checks that would require multiple KV lookups or complex coordination logic.

### How is the Durable Object instance addressed for a specific audit?

The system uses the **`getAuditScratchpad(auditId)`** factory function to obtain a proxy stub. This stub routes method calls to the specific Durable Object instance identified by the `auditId` parameter, ensuring that all workers and workflow phases interact with the same SQLite database file for a given site audit.

### What happens to the SQLite data when an audit is deleted?

The **`destroy`** lifecycle hook (implicitly invoked via [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)) destroys the Durable Object instance, which drops the associated SQLite storage. While the cleanup is best-effort, this mechanism prevents storage accumulation for completed or cancelled audits.