How OpenSEO Crawls Websites for Site Audits: A Deep Dive into the Cloudflare Workers Architecture

OpenSEO crawls websites using a multi-stage, Durable Object-backed workflow that processes pages in bounded chunks, keeping memory usage constant even for sites with thousands of pages.

The every-app/open-seo repository implements a site audit crawler designed specifically for the constraints of Cloudflare Workers. Its architecture prioritizes memory safety, incremental persistence, and fault tolerance over raw speed. This article breaks down exactly how the crawl phase works, from the initial API call to the final database write.

---

Overview of the Crawl Architecture

OpenSEO's crawler is not a monolithic script. It distributes work across Durable Objects for state management and uses chunked execution to respect the 128 MB memory limit of Cloudflare Workers.

Component	Purpose	Source File
runSiteAuditTool	Public API entry point that validates input and launches workflows	src/server/mcp/tools/site-audit-tools.ts
SiteAuditWorkflow	Orchestrates three phases: discovery → crawling → reporting	src/server/workflows/SiteAuditWorkflow.ts
runCrawlPhase	Loops over crawl chunks until the page budget is exhausted	src/server/workflows/siteAuditWorkflowCrawl.ts
runCrawlChunk	Fetches a batch of URLs concurrently, with adaptive rate limiting	src/server/workflows/siteAuditWorkflowCrawl.ts
crawlPage	Performs the actual HTTP fetch with custom headers, redirect handling, and body size caps	src/server/workflows/site-audit-workflow-helpers.ts
page-analyzer	Dynamically imported HTML parser for extracting SEO signals	src/server/lib/audit/page-analyzer.ts
AuditScratchpad	Durable Object holding the crawl frontier and visited set	src/server/features/audit/AuditScratchpad.ts
AuditRepository	Persists crawled pages and derived SEO issues to the database	src/server/features/audit/repositories/AuditRepository.ts

---

Starting an Audit: The Entry Point

All audits begin with the run_site_audit MCP tool. The handler validates the start URL, optional maxPages budget, and Lighthouse flag, then delegates to AuditService.startAudit.

// Example: Starting a site audit via the MCP client
await client.run_site_audit({
  projectId: "proj_123",
  url: "https://example.com",
  maxPages: 200,
  runLighthouse: true,
});

This creates an Audit database row and triggers a Cloudflare Workflow instance with workflowInstanceId.

---

Workflow Orchestration: Three Distinct Phases

The SiteAuditWorkflow runs as a series of durable steps—each step can suspend and resume without losing state. The crawling phase is invoked like this:

const crawl = await runCrawlPhase(step, {
  auditId,
  workflowInstanceId,
  origin: new URL(startUrl).origin,
  maxPages,
  robots,
  seededCount,
});

The step parameter is a Cloudflare Workflow primitive that ensures durability: if the Worker is evicted, execution resumes at the same step.

---

Chunked Crawling: O(1) Memory Design

The core innovation in how OpenSEO crawls websites is its chunk-and-persist model. Instead of loading the entire site into memory, runCrawlPhase loops over discrete chunks until completion.

How a Single Chunk Works (runCrawlChunk)

1. Claim URLs from the scratchpad Durable Object (scratchpad.claimChunk), up to CHUNK_TARGET_PAGES

2. Manage concurrency with an adaptive crawl window (windowSize) that expands or contracts based on success rates

3. Launch fetches via crawlPage until the window fills, the 90-second soft deadline fires, or the batch queue hits MAX_QUEUED_PERSIST_BATCHES

4. Persist sub-batches of PERSIST_BATCH_SIZE pages via persistCrawledPages—serialized to prevent DB contention

5. Return counters (attempted, pending, endWindow) so the next chunk continues seamlessly

This design guarantees memory usage stays O(1) regardless of site size.

---

Fetching Individual Pages (crawlPage)

Every URL passes through crawlPage in site-audit-workflow-helpers.ts. The implementation reveals careful attention to edge cases:

const response = await fetch(url, {
  headers: {
    "User-Agent": "OpenSEO-Audit/1.0",
    Accept: "text/html,application/xhtml+xml",
  },
  redirect: "manual",        // Handle redirects explicitly
  signal: AbortSignal.timeout(15_000),  // 15s timeout
});

Key behaviors:

• Manual redirect handling — Each hop becomes its own page row with full metadata
• 1 MiB body cap — readTextUpTo(MAX_HTML_BYTES) prevents memory exhaustion from massive pages
• Fetch classification — classifyFetch() labels responses as ok, blocked (Cloudflare challenges), or error
• Dynamic parser loading — The page-analyzer is import()ed only when needed, keeping cold starts fast

For valid HTML pages, crawlPage returns a CrawledPageResult containing:

• Title, meta description, canonical URL
• Heading hierarchy counts (H1-H6)
• Internal/external link lists
• Image alt text coverage
• Structured data detection flags

---

Persistence and Frontier Management

After each sub-batch, persistCrawledPages performs several critical operations:

1. Deterministic ID Assignment

const pageId = deterministicAuditRowId(auditId, url);

This makes retries idempotent—re-processing the same URL yields the same database row.

2. Issue Generation

runPageReporters analyzes each page for SEO issues (duplicate titles, missing alt text, etc.) and generates structured issue objects.

3. Database Write

AuditRepository.insertCrawledBatch atomically inserts:

• Crawled page rows
• Generated SEO issue rows

4. Link Discovery

The system extracts:

• Internal links (up to MAX_STORED_LINKS_PER_PAGE = 500)
• Newly discovered URLs (capped at MAX_DISCOVERED_PER_BATCH = 20,000)

These feed back into the scratchpad frontier via scratchpad.recordBatch, keeping the crawl same-origin only and robots.txt aware.

---

Progress Reporting and Observability

After each persisted sub-batch, the workflow calls:

AuditProgressKV.pushCrawledUrls(auditId, urls.length);

Clients poll status via get_audit_status:

async function pollStatus(auditId: string) {
  while (true) {
    const { text } = await client.get_audit_status({
      projectId: "proj_456",
      auditId,
    });
    console.log(text); // "Audit abc123: crawling — 42/100 pages..."
    if (/completed|failed/.test(text)) break;
    await new Promise(r => setTimeout(r, 5_000));
  }
}

---

Error Handling and Resilience

OpenSEO anticipates Worker evictions and memory pressure:

Scenario	Response
Chunk timeout	Automatic retry with reduced RETRY_CRAWL_WINDOW
Memory overrun	Adaptive window shrinks on next iteration
Fetch failure	emptyPageResult records fetchClass: "error" with partial metadata
Workflow crash	Durable Objects preserve frontier state; execution resumes at last completed step

This ensures partial audit results are never lost—even catastrophic failures leave recoverable data in the database.

---

Complete Code Example: Custom Crawl Script

For developers building on OpenSEO's internals, the crawlPage helper is directly importable:

import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";

(async () => {
  const result = await crawlPage(
    "https://example.com/about",
    1,               // crawl depth from start URL
    false,           // not in sitemap
  );

  console.log(result.title);           // "About Us – Example Corp"
  console.log(result.links.length);    // internal link count
  console.log(result.images.filter(img => !img.alt).length); // missing alt tags
})();

---

Summary

• OpenSEO crawls websites using a chunked, Durable Object-backed workflow that maintains constant memory regardless of site size
• Three orchestration layers separate concerns: API tools (runSiteAuditTool), workflow phases (SiteAuditWorkflow), and fetch primitives (crawlPage)
• Adaptive rate limiting via the crawl window prevents Worker memory exhaustion
• Incremental persistence after every sub-batch ensures durability and enables real-time progress reporting
• Robust error handling with automatic retries and idempotent writes guarantees audit completion even under resource constraints

---

Frequently Asked Questions

What happens if a site has more pages than the maxPages budget?

OpenSEO's runCrawlPhase respects maxPages as a hard limit. Once attemptedPages reaches the budget, the loop terminates regardless of remaining frontier URLs. The scratchpad Durable Object preserves the unconsumed frontier, allowing future audits to resume where this one left off.

How does OpenSEO handle JavaScript-rendered content?

According to the source code, OpenSEO does not execute JavaScript during crawling. The crawlPage function fetches raw HTML and passes it to the static page-analyzer. For SPAs or heavy JavaScript sites, the crawler sees only the initial server-rendered HTML.

Why use Durable Objects instead of a database for the crawl frontier?

Durable Objects provide sub-millisecond coordination latency and strong consistency for the frontier state—critical for concurrent chunk execution. A relational database would introduce query overhead and contention during high-frequency claim/release operations. The scratchpad design (in AuditScratchpad.ts) keeps hot path operations in memory while still persisting progress to the database in batches.

Can I run OpenSEO's crawler outside of Cloudflare Workers?

The current architecture is tightly coupled to Cloudflare primitives: Workflows for orchestration, Durable Objects for state, and KV for progress reporting. The core fetch and parse logic in site-audit-workflow-helpers.ts and page-analyzer.ts could theoretically be extracted, but the chunking and durability guarantees would require reimplementation using equivalent primitives in another environment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}