# How the OpenSEO Site Audit Engine Works: A Technical Deep-Dive

> Explore the OpenSEO site audit engine, a Cloudflare-powered pipeline that crawls sites, detects SEO issues, and provides programmatic access to server functions and MCP tools.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-06

---

**The OpenSEO site audit engine is a Cloudflare-powered server-side pipeline that crawls websites, detects SEO issues, and exposes typed server-functions and MCP tools for programmatic access.**

OpenSEO's **site audit engine** is the core infrastructure that powers comprehensive SEO analysis at scale. It combines Cloudflare Durable Objects, multi-phase crawling workflows, and a tiered capacity system to deliver automated site audits. This guide explains the complete architecture, from audit initiation through final report retrieval, with references to the actual source implementation in the `every-app/open-seo` repository.

## Starting an Audit: The `startAudit` Flow

When a client initiates an audit—either through the API function `startAudit` or the MCP tool `run_site_audit`—the engine executes a five-step orchestration defined in **[`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)**.

### Step 1: Resolve Plan Limits

The engine first determines the organization's tier using `AuditService.resolveAuditLimitTier`. Valid tiers are `free`, `paid`, or `self_hosted`, each with distinct crawling capacities. This lookup gates all subsequent operations.

### Step 2: Validate Capacity

Two functions enforce hard limits:
- `clampAuditMaxPages` — caps the request at the tier's per-audit page maximum
- `getEstimatedAuditCapacity` — calculates weighted capacity units (pages + optional Lighthouse runs)

If estimated capacity exceeds the organization's `maxCapacityUnits`, the request fails immediately with `AppError(AUDIT_CAPACITY_REACHED)`.

### Step 3: Create Audit Record

`AuditRepository.createAudit` persists a new row with:
- UUID (serves as workflow ID)
- Normalized start URL
- Calculated capacity allocation
- Initial status: `"pending"`

### Step 4: Spawn Durable Workflow

The critical execution handoff occurs via `env.SITE_AUDIT_WORKFLOW.create`, launching the `site_audit_workflow` with the audit UUID as its identifier. This Cloudflare Durable Workflow runs independently of the HTTP request lifecycle.

### Step 5: Failure Handling

A `catch` block ensures atomic rollback: if workflow creation fails, the audit row is deleted and the error bubbles to the caller.

The MCP wrapper in **[`src/server/mcp/tools/site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts)** adds Lighthouse configuration logic (`auto` vs `none`) and analytics instrumentation via PostHog before delegating to the core service.

## The Crawling Workflow: `site_audit_workflow`

Once spawned, the workflow executes the actual crawl using helper functions from **[`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts)**. The pipeline has five distinct phases:

### 1. URL Normalization

`resolveStartUrlRedirects` follows all redirects from the submitted URL to its final destination. This prevents the crawler from treating redirect chains as separate origins and ensures scope integrity.

### 2. Robots-Aware Crawling

A same-origin crawler fetches pages sequentially while respecting:
- [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) directives
- `nofollow` link attributes
- Meta robots tags

For each crawled page, the engine extracts:
- HTTP status code
- `<title>` content
- Meta description
- Word count
- Internal/outbound link counts
- Canonical URL

### 3. Lighthouse Sampling (Optional)

When `lighthouseStrategy` is `"auto"`, the workflow selects a representative page sample and runs Google Lighthouse to capture:
- Core Web Vitals (LCP, FID/INP, CLS)
- Performance scores
- Accessibility metrics

Lighthouse runs consume additional capacity units as defined in **[`audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/audit-capacity.ts)**.

### 4. Issue Detection

Each page undergoes rule-based analysis for SEO problems:
- **Technical**: broken links, server errors, redirect chains
- **On-page**: duplicate titles, missing meta descriptions, thin content
- **Architecture**: orphan pages, canonical misconfigurations, crawl depth excess

Findings write to `audit_issues` tables with severity classification (`critical`, `warning`, `info`).

### 5. Temporary State Management

Interim crawl data persists to a dedicated Durable Object accessed via `env.AUDIT_ENGINE`. This scratchpad serves as high-throughput temporary storage without database contention. Upon completion or abortion, `env.AUDIT_ENGINE.destroyScratchpad` releases these resources.

## Querying Status and Results

Clients interact with running and completed audits through three primary server-functions, all exposed via `@tanstack/react-start` API routes.

### `getAuditStatus`: Live Progress Monitoring

```

Input:  { auditId: string, projectId: string }
Output: { currentPhase, pagesCrawled, pagesTotal, lighthouseProgress?, errorCode? }

```

The implementation in `AuditService.getStatus` includes **self-healing logic**: if a workflow disappears (crash, timeout) while the audit row still shows `"running"`, the function reconciles the state to `"failed"` with appropriate diagnostics.

### `getAuditResults`: Full Report Retrieval

Returns the complete audit payload:
- Base audit metadata
- All crawled pages with extracted fields
- Lighthouse run data (if enabled)
- Full issue inventory with locations and severities

The effective Lighthouse strategy is parsed from the stored JSON `config` field.

### `getAuditHistory`: Project-Level Listing

Lists recent audits with summary statistics, enabling UI timeline views and trend analysis.

### MCP Tool Equivalents

Three thin wrappers in **[`site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/site-audit-tools.ts)** expose the same data with formatted output:
- `get_audit_status` — human-readable progress strings
- `get_audit_issues` — severity-filtered, prioritized lists
- `get_audit_pages` — paginated page metadata with fetch classification (`ok`, `error`, `redirect`)

## Capacity and Quota Enforcement

The engine maintains two hard limits per organization, defined in **[`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts)**:

| Limit | Description | Violation Behavior |
|-------|-------------|------------------|
| `maxRunningAudits` | Concurrent active workflows | `AppError(AUDIT_ALREADY_RUNNING)` |
| `maxCapacityUnits` | Weighted sum of pages + Lighthouse | `AppError(AUDIT_CAPACITY_REACHED)` |

Capacity units are calculated as: `pages + (lighthouseRuns × LIGHTHOUSE_WEIGHT)`. The MCP layer translates thrown errors into conversational refusal messages for agent consumers.

When an audit completes, fails, or is deleted, its reserved capacity is released, enabling new audits within the same billing period.

## Audit Deletion and Cleanup

The `deleteAudit` flow in [`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts) performs ordered teardown:

1. Lookup workflow ID from audit record
2. If status is `"running"`, terminate the Durable Workflow
3. Delete database rows (cascade to pages, issues, Lighthouse data)
4. Invoke `env.AUDIT_ENGINE.destroyScratchpad` for DO cleanup

This prevents resource leaks and immediately frees quota for replacement audits.

## Code Examples

### Starting an Audit via API

```typescript
import { startAudit } from "@/serverFunctions/audit";

const audit = await startAudit({
  startUrl: "https://example.com",
  maxPages: 200,
  lighthouseStrategy: "auto",
});
// Returns: { auditId: "uuid", status: "pending" }

```

### Starting via MCP Tool

```typescript
import { runSiteAuditTool } from "@/server/mcp/tools/site-audit-tools";

await runSiteAuditTool.handler(
  {
    projectId: "proj_123",
    url: "https://example.com",
    maxPages: 200,
    runLighthouse: true,
  },
  context
);

```

### Polling Status with Self-Healing Detection

```typescript
import { getAuditStatus } from "@/serverFunctions/audit";

const { status } = await getAuditStatus({
  auditId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  projectId: "proj_123",
});

console.log(`${status.currentPhase}: ${status.pagesCrawled}/${status.pagesTotal}`);
// "crawling: 47/200" or "failed: workflow reconciliation detected orphaned state"

```

### Retrieving Prioritized Issues

```typescript
import { getAuditIssuesTool } from "@/server/mcp/tools/site-audit-tools";

const { summary, issues } = await getAuditIssuesTool.handler(
  {
    projectId: "proj_123",
    auditId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    severity: "critical",
  },
  context
);

summary.forEach((s) => {
  console.log(`${s.severity}: ${s.title} (${s.count} occurrences)`);
});

```

### Listing Crawled Pages

```typescript
const { pages, total } = await getAuditPagesTool.handler(
  {
    projectId: "proj_123",
    auditId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    fetchClass: "ok",
    limit: 50,
  },
  context
);

pages.forEach((p) => {
  console.log(`${p.url} — ${p.title} (${p.wordCount} words)`);
});

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) | Core orchestration: start, status, results, history, delete |
| [`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts) | Tier definitions and capacity calculations |
| [`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts) | Database layer for audits, pages, issues, Lighthouse data |
| [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) | Durable Workflow implementation for crawl orchestration |
| [`src/server/mcp/tools/site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts) | MCP wrappers for agent and UI consumption |
| [`src/types/schemas/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/types/schemas/audit.ts) | Zod validation schemas (`startAuditSchema`, etc.) |
| [`src/audit-worker.ts`](https://github.com/every-app/open-seo/blob/main/src/audit-worker.ts) | RPC endpoint for DO scratchpad operations |
| [`specs/0009-site-audit-crawl-architecture.md`](https://github.com/every-app/open-seo/blob/main/specs/0009-site-audit-crawl-architecture.md) | Architectural design documentation |

## Summary

- The **OpenSEO site audit engine** runs as a Cloudflare Durable Workflow decoupled from HTTP request handlers, enabling long-running crawls without timeout constraints.
- **AuditService.ts** handles all client-facing operations with tier-aware capacity enforcement and automatic workflow state reconciliation.
- **Temporary crawl state** persists to a dedicated Durable Object (`env.AUDIT_ENGINE`), not the primary database, for performance isolation.
- **Issue detection** runs during the crawl phase against extracted page metadata, with findings stored in relational tables for efficient querying.
- **MCP tool wrappers** provide identical functionality through both HTTP API and agent-native interfaces, with formatted output suitable for LLM consumption.

## Frequently Asked Questions

### What is a Durable Workflow in OpenSEO and why is it used for site audits?

A Durable Workflow is Cloudflare's serverless execution environment that maintains state across multiple invocations. OpenSEO uses it for site audits because crawling hundreds or thousands of pages exceeds standard HTTP request timeouts. The workflow persists its execution position, survives infrastructure restarts, and can run for extended durations while the client polls for completion.

### How does OpenSEO prevent a single user from monopolizing audit resources?

The engine implements two guardrails: `maxRunningAudits` limits concurrent workflows per organization, and `maxCapacityUnits` enforces a weighted quota on total crawling activity. Both limits are tier-aware (`free`, `paid`, `self_hosted`) and checked atomically before any workflow creation. Exceeded limits return structured errors that the MCP layer translates into user-friendly messages.

### Can I run a site audit without Lighthouse performance analysis?

Yes. The `lighthouseStrategy` parameter accepts `"none"` to disable performance scanning. This reduces capacity consumption and crawl duration. When set to `"auto"`, the workflow samples representative pages for Lighthouse analysis. The strategy is stored in the audit config and exposed in result queries.

### What happens if a site audit workflow crashes mid-crawl?

The `getAuditStatus` implementation includes reconciliation logic: if a row shows `"running"` but the corresponding Durable Workflow no longer exists, the service updates the status to `"failed"` with an appropriate error code. This prevents permanently stuck audits and releases reserved capacity for new requests.