# How OpenSEO Crawls Websites for Site Audits: A Deep Dive into the Cloudflare Workers Architecture

> Discover how OpenSEO crawls websites for audits. Learn about its Durable Object workflow, efficient page processing, and constant memory usage for any site size.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-06

---

**OpenSEO crawls websites using a multi-stage, Durable Object-backed workflow that processes pages in bounded chunks, keeping memory usage constant even for sites with thousands of pages.**

The `every-app/open-seo` repository implements a site audit crawler designed specifically for the constraints of Cloudflare Workers. Its architecture prioritizes **memory safety**, **incremental persistence**, and **fault tolerance** over raw speed. This article breaks down exactly how the crawl phase works, from the initial API call to the final database write.

---

## Overview of the Crawl Architecture

OpenSEO's crawler is not a monolithic script. It distributes work across **Durable Objects** for state management and uses **chunked execution** to respect the 128 MB memory limit of Cloudflare Workers.

| Component | Purpose | Source File |
|-----------|---------|-------------|
| **`runSiteAuditTool`** | Public API entry point that validates input and launches workflows | [`src/server/mcp/tools/site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts) |
| **`SiteAuditWorkflow`** | Orchestrates three phases: discovery → crawling → reporting | [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts) |
| **`runCrawlPhase`** | Loops over crawl chunks until the page budget is exhausted | [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) |
| **`runCrawlChunk`** | Fetches a batch of URLs concurrently, with adaptive rate limiting | [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) |
| **`crawlPage`** | Performs the actual HTTP fetch with custom headers, redirect handling, and body size caps | [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) |
| **`page-analyzer`** | Dynamically imported HTML parser for extracting SEO signals | [`src/server/lib/audit/page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/page-analyzer.ts) |
| **`AuditScratchpad`** | Durable Object holding the crawl frontier and visited set | [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts) |
| **`AuditRepository`** | Persists crawled pages and derived SEO issues to the database | [`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts) |

---

## Starting an Audit: The Entry Point

All audits begin with the **`run_site_audit`** MCP tool. The handler validates the start URL, optional `maxPages` budget, and Lighthouse flag, then delegates to `AuditService.startAudit`.

```ts
// Example: Starting a site audit via the MCP client
await client.run_site_audit({
  projectId: "proj_123",
  url: "https://example.com",
  maxPages: 200,
  runLighthouse: true,
});

```

This creates an `Audit` database row and triggers a Cloudflare Workflow instance with `workflowInstanceId`.

---

## Workflow Orchestration: Three Distinct Phases

The `SiteAuditWorkflow` runs as a series of **durable steps**—each step can suspend and resume without losing state. The crawling phase is invoked like this:

```ts
const crawl = await runCrawlPhase(step, {
  auditId,
  workflowInstanceId,
  origin: new URL(startUrl).origin,
  maxPages,
  robots,
  seededCount,
});

```

The `step` parameter is a Cloudflare Workflow primitive that ensures durability: if the Worker is evicted, execution resumes at the same step.

---

## Chunked Crawling: O(1) Memory Design

The core innovation in how OpenSEO crawls websites is its **chunk-and-persist** model. Instead of loading the entire site into memory, `runCrawlPhase` loops over discrete chunks until completion.

### How a Single Chunk Works (`runCrawlChunk`)

1. **Claim URLs** from the scratchpad Durable Object (`scratchpad.claimChunk`), up to `CHUNK_TARGET_PAGES`

2. **Manage concurrency** with an adaptive crawl window (`windowSize`) that expands or contracts based on success rates

3. **Launch fetches** via `crawlPage` until the window fills, the 90-second soft deadline fires, or the batch queue hits `MAX_QUEUED_PERSIST_BATCHES`

4. **Persist sub-batches** of `PERSIST_BATCH_SIZE` pages via `persistCrawledPages`—serialized to prevent DB contention

5. **Return counters** (`attempted`, `pending`, `endWindow`) so the next chunk continues seamlessly

This design guarantees **memory usage stays O(1)** regardless of site size.

---

## Fetching Individual Pages (`crawlPage`)

Every URL passes through `crawlPage` in [`site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/site-audit-workflow-helpers.ts). The implementation reveals careful attention to edge cases:

```ts
const response = await fetch(url, {
  headers: {
    "User-Agent": "OpenSEO-Audit/1.0",
    Accept: "text/html,application/xhtml+xml",
  },
  redirect: "manual",        // Handle redirects explicitly
  signal: AbortSignal.timeout(15_000),  // 15s timeout
});

```

Key behaviors:

- **Manual redirect handling** — Each hop becomes its own page row with full metadata
- **1 MiB body cap** — `readTextUpTo(MAX_HTML_BYTES)` prevents memory exhaustion from massive pages
- **Fetch classification** — `classifyFetch()` labels responses as `ok`, `blocked` (Cloudflare challenges), or `error`
- **Dynamic parser loading** — The `page-analyzer` is `import()`ed only when needed, keeping cold starts fast

For valid HTML pages, `crawlPage` returns a `CrawledPageResult` containing:

- Title, meta description, canonical URL
- Heading hierarchy counts (H1-H6)
- Internal/external link lists
- Image alt text coverage
- Structured data detection flags

---

## Persistence and Frontier Management

After each sub-batch, `persistCrawledPages` performs several critical operations:

### 1. Deterministic ID Assignment

```ts
const pageId = deterministicAuditRowId(auditId, url);

```

This makes retries **idempotent**—re-processing the same URL yields the same database row.

### 2. Issue Generation

`runPageReporters` analyzes each page for SEO issues (duplicate titles, missing alt text, etc.) and generates structured issue objects.

### 3. Database Write

`AuditRepository.insertCrawledBatch` atomically inserts:
- Crawled page rows
- Generated SEO issue rows

### 4. Link Discovery

The system extracts:
- **Internal links** (up to `MAX_STORED_LINKS_PER_PAGE = 500`)
- **Newly discovered URLs** (capped at `MAX_DISCOVERED_PER_BATCH = 20,000`)

These feed back into the scratchpad frontier via `scratchpad.recordBatch`, keeping the crawl **same-origin only** and **robots.txt aware**.

---

## Progress Reporting and Observability

After each persisted sub-batch, the workflow calls:

```ts
AuditProgressKV.pushCrawledUrls(auditId, urls.length);

```

Clients poll status via `get_audit_status`:

```ts
async function pollStatus(auditId: string) {
  while (true) {
    const { text } = await client.get_audit_status({
      projectId: "proj_456",
      auditId,
    });
    console.log(text); // "Audit abc123: crawling — 42/100 pages..."
    if (/completed|failed/.test(text)) break;
    await new Promise(r => setTimeout(r, 5_000));
  }
}

```

---

## Error Handling and Resilience

OpenSEO anticipates Worker evictions and memory pressure:

| Scenario | Response |
|----------|----------|
| Chunk timeout | Automatic retry with reduced `RETRY_CRAWL_WINDOW` |
| Memory overrun | Adaptive window shrinks on next iteration |
| Fetch failure | `emptyPageResult` records `fetchClass: "error"` with partial metadata |
| Workflow crash | Durable Objects preserve frontier state; execution resumes at last completed step |

This ensures **partial audit results are never lost**—even catastrophic failures leave recoverable data in the database.

---

## Complete Code Example: Custom Crawl Script

For developers building on OpenSEO's internals, the `crawlPage` helper is directly importable:

```ts
import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";

(async () => {
  const result = await crawlPage(
    "https://example.com/about",
    1,               // crawl depth from start URL
    false,           // not in sitemap
  );

  console.log(result.title);           // "About Us – Example Corp"
  console.log(result.links.length);    // internal link count
  console.log(result.images.filter(img => !img.alt).length); // missing alt tags
})();

```

---

## Summary

- **OpenSEO crawls websites** using a chunked, Durable Object-backed workflow that maintains constant memory regardless of site size
- **Three orchestration layers** separate concerns: API tools (`runSiteAuditTool`), workflow phases (`SiteAuditWorkflow`), and fetch primitives (`crawlPage`)
- **Adaptive rate limiting** via the crawl window prevents Worker memory exhaustion
- **Incremental persistence** after every sub-batch ensures durability and enables real-time progress reporting
- **Robust error handling** with automatic retries and idempotent writes guarantees audit completion even under resource constraints

---

## Frequently Asked Questions

### What happens if a site has more pages than the maxPages budget?

OpenSEO's `runCrawlPhase` respects `maxPages` as a hard limit. Once `attemptedPages` reaches the budget, the loop terminates regardless of remaining frontier URLs. The scratchpad Durable Object preserves the unconsumed frontier, allowing future audits to resume where this one left off.

### How does OpenSEO handle JavaScript-rendered content?

According to the source code, OpenSEO does not execute JavaScript during crawling. The `crawlPage` function fetches raw HTML and passes it to the static `page-analyzer`. For SPAs or heavy JavaScript sites, the crawler sees only the initial server-rendered HTML.

### Why use Durable Objects instead of a database for the crawl frontier?

Durable Objects provide **sub-millisecond coordination latency** and **strong consistency** for the frontier state—critical for concurrent chunk execution. A relational database would introduce query overhead and contention during high-frequency claim/release operations. The scratchpad design (in [`AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/AuditScratchpad.ts)) keeps hot path operations in memory while still persisting progress to the database in batches.

### Can I run OpenSEO's crawler outside of Cloudflare Workers?

The current architecture is tightly coupled to Cloudflare primitives: Workflows for orchestration, Durable Objects for state, and KV for progress reporting. The core fetch and parse logic in [`site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/site-audit-workflow-helpers.ts) and [`page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/page-analyzer.ts) could theoretically be extracted, but the chunking and durability guarantees would require reimplementation using equivalent primitives in another environment.