How OpenSEO Manages Crawl Frontiers for Site Audits in Cloudflare Durable Objects

OpenSEO manages crawl frontiers for site audits using a Cloudflare Durable Object called AuditScratchpad that maintains a lightweight SQLite database to queue, lease, and track the state of every URL during large-scale crawls.

Crawl frontiers are the backbone of any site audit engine—they determine which pages to fetch, in what order, and how to recover from failures. In OpenSEO, this complexity is encapsulated in a single Durable Object that persists state across workflow steps, enabling resumable, fault-tolerant crawls without bloating message payloads.

What Is a Crawl Frontier in OpenSEO?

A crawl frontier is the set of URLs discovered but not yet processed. In src/server/features/audit/AuditScratchpad.ts, OpenSEO implements this as three SQLite tables:

• frontier — tracks every URL's depth, source (link or sitemap), and current state (pending, leased, crawled)
• page_mirror — stores crawled page content and metadata
• links — records discovered internal links between pages

The frontier table schema at lines 90-96 ensures each URL is unique and queryable by state:

CREATE TABLE IF NOT EXISTS frontier (
  url TEXT PRIMARY KEY,
  depth INTEGER,
  source TEXT,        -- 'link' or 'sitemap'
  state TEXT,         -- 'pending', 'leased', 'crawled'
  chunk_no INTEGER    -- which batch leased this URL
)

Seeding the Frontier with Start URLs and Sitemaps

Every audit begins with seeding. The seedStart() and seedSitemapUrls() methods (lines 25-41) populate the frontier with initial URLs:

// Seed from the manually provided start URL
await scratchpad.seedStart('https://example.com/');

// Seed from parsed sitemap.xml
await scratchpad.seedSitemapUrls([
  'https://example.com/page1',
  'https://example.com/page2'
]);

Sitemap URLs are upserted with in_sitemap = 1, allowing the frontier to distinguish between link-discovered and sitemap-only entries. This distinction drives crawl prioritization.

Leasing URL Chunks with Priority Ordering

The claimChunk() method (lines 50-75) implements the core frontier traversal logic. Each workflow step requests a batch of URLs by chunk number:

1. Idempotent retry check — if this chunk already leased URLs, return them again
2. Priority query — prefer source='link' over source='sitemap' to prioritize editorially linked pages
3. FIFO within class — maintain crawl order for fairness
4. State transition — pending → leased with chunk number assignment

const chunkNo = 1;
const urlsToCrawl = await scratchpad.claimChunk(chunkNo, 100);
// Returns: [{url: 'https://example.com/', depth: 0, source: 'link'}, ...]

This design ensures exactly-once processing per chunk attempt—critical for Cloudflare Workflows where steps may retry.

Recording Batch Results and Discovering New URLs

After crawling, recordBatch() (lines 85-99) persists results and expands the frontier:

await scratchpad.recordBatch({
  crawledUrls: urlsToCrawl.map(u => u.url),
  pages: [{url: 'https://example.com/', status: 200, title: '...'}],
  links: [{from: 'https://example.com/', to: 'https://example.com/about'}],
  discovered: ['https://example.com/about', 'https://example.com/contact']
});

The method performs four operations atomically:

• Updates frontier rows to state='crawled'
• Inserts or updates page_mirror rows
• Stores link edges (respecting storage budget caps)
• Enqueues newly discovered URLs as pending with source='link'

Handling Failures Through Lease Releases

When a chunk exceeds its soft deadline, uncompleted URLs must return to the queue. The releaseUrls() method (lines 32-40) resets leased entries to pending:

await scratchpad.releaseUrls(['https://example.com/slow-page']);

This prevents orphaned leases that would stall the crawl indefinitely.

Monitoring Frontier Health with Statistics

OpenSEO exposes frontier metrics via getStats() (lines 42-48), enabling the workflow to detect completion or abort conditions:

Metric	Meaning
attempted	URLs that reached recordBatch()
pending	URLs awaiting first lease
seen	Total unique URLs in frontier

The workflow in src/server/workflows/siteAuditWorkflowCrawl.ts polls these stats to decide when no work remains.

Automatic Cleanup and Resource Limits

Every AuditScratchpad instance schedules a 7-day destruction alarm in its constructor (lines 17-23). This guarantees no persistent crawl state leaks after completion, even if workflows fail to signal cleanup.

Additionally, link storage operates under a budget cap to prevent SQLite size explosions on densely interlinked sites—a practical safeguard absent from simpler queue-based frontier implementations.

Summary

• Localized state: SQLite in a Durable Object avoids large workflow payloads
• Idempotent leasing: Chunk-based retries prevent duplicate crawling
• Smart prioritization: Link-discovered URLs precede sitemap-only entries
• Fault recovery: Lease releases and statistics enable graceful degradation
• Resource safety: Storage budgets and automatic cleanup prevent runaway growth

Frequently Asked Questions

How does OpenSEO prevent duplicate crawling when workflow steps retry?

claimChunk() checks for existing leases by chunk number before selecting new URLs. If the same chunk is claimed twice, it returns the already-leased set, making the operation idempotent.

Why does OpenSEO prioritize link-discovered URLs over sitemap URLs?

Links indicate editorial importance and crawlability—pages with more inbound links are typically more significant for SEO. Sitemap entries may include orphaned or low-value pages, so they receive secondary priority while still being crawled eventually.

What happens if a crawl is interrupted mid-batch?

Any URLs still in leased state from incomplete chunks can be released via releaseUrls() and re-claimed by subsequent chunks. The frontier remains consistent, and recordBatch() can be safely re-invoked with updated results.

Where does URL policy enforcement occur before frontier insertion?

The src/server/lib/audit/url-policy.ts module validates same-origin constraints and robots.txt rules before discovered URLs reach recordBatch(), ensuring only policy-compliant URLs enter the frontier queue.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}