How the OpenSEO Site Audit Engine Works: A Technical Deep-Dive

The OpenSEO site audit engine is a Cloudflare-powered server-side pipeline that crawls websites, detects SEO issues, and exposes typed server-functions and MCP tools for programmatic access.

OpenSEO's site audit engine is the core infrastructure that powers comprehensive SEO analysis at scale. It combines Cloudflare Durable Objects, multi-phase crawling workflows, and a tiered capacity system to deliver automated site audits. This guide explains the complete architecture, from audit initiation through final report retrieval, with references to the actual source implementation in the every-app/open-seo repository.

Starting an Audit: The startAudit Flow

When a client initiates an audit—either through the API function startAudit or the MCP tool run_site_audit—the engine executes a five-step orchestration defined in src/server/features/audit/services/AuditService.ts.

Step 1: Resolve Plan Limits

The engine first determines the organization's tier using AuditService.resolveAuditLimitTier. Valid tiers are free, paid, or self_hosted, each with distinct crawling capacities. This lookup gates all subsequent operations.

Step 2: Validate Capacity

Two functions enforce hard limits:

• clampAuditMaxPages — caps the request at the tier's per-audit page maximum
• getEstimatedAuditCapacity — calculates weighted capacity units (pages + optional Lighthouse runs)

If estimated capacity exceeds the organization's maxCapacityUnits, the request fails immediately with AppError(AUDIT_CAPACITY_REACHED).

Step 3: Create Audit Record

AuditRepository.createAudit persists a new row with:

• UUID (serves as workflow ID)
• Normalized start URL
• Calculated capacity allocation
• Initial status: "pending"

Step 4: Spawn Durable Workflow

The critical execution handoff occurs via env.SITE_AUDIT_WORKFLOW.create, launching the site_audit_workflow with the audit UUID as its identifier. This Cloudflare Durable Workflow runs independently of the HTTP request lifecycle.

Step 5: Failure Handling

A catch block ensures atomic rollback: if workflow creation fails, the audit row is deleted and the error bubbles to the caller.

The MCP wrapper in src/server/mcp/tools/site-audit-tools.ts adds Lighthouse configuration logic (auto vs none) and analytics instrumentation via PostHog before delegating to the core service.

The Crawling Workflow: site_audit_workflow

Once spawned, the workflow executes the actual crawl using helper functions from src/server/workflows/site-audit-workflow-helpers.ts. The pipeline has five distinct phases:

1. URL Normalization

resolveStartUrlRedirects follows all redirects from the submitted URL to its final destination. This prevents the crawler from treating redirect chains as separate origins and ensures scope integrity.

2. Robots-Aware Crawling

A same-origin crawler fetches pages sequentially while respecting:

• robots.txt directives
• nofollow link attributes
• Meta robots tags

For each crawled page, the engine extracts:

• HTTP status code
• <title> content
• Meta description
• Word count
• Internal/outbound link counts
• Canonical URL

3. Lighthouse Sampling (Optional)

When lighthouseStrategy is "auto", the workflow selects a representative page sample and runs Google Lighthouse to capture:

• Core Web Vitals (LCP, FID/INP, CLS)
• Performance scores
• Accessibility metrics

Lighthouse runs consume additional capacity units as defined in audit-capacity.ts.

4. Issue Detection

Each page undergoes rule-based analysis for SEO problems:

• Technical: broken links, server errors, redirect chains
• On-page: duplicate titles, missing meta descriptions, thin content
• Architecture: orphan pages, canonical misconfigurations, crawl depth excess

Findings write to audit_issues tables with severity classification (critical, warning, info).

5. Temporary State Management

Interim crawl data persists to a dedicated Durable Object accessed via env.AUDIT_ENGINE. This scratchpad serves as high-throughput temporary storage without database contention. Upon completion or abortion, env.AUDIT_ENGINE.destroyScratchpad releases these resources.

Querying Status and Results

Clients interact with running and completed audits through three primary server-functions, all exposed via @tanstack/react-start API routes.

getAuditStatus: Live Progress Monitoring

Input:  { auditId: string, projectId: string }
Output: { currentPhase, pagesCrawled, pagesTotal, lighthouseProgress?, errorCode? }

The implementation in AuditService.getStatus includes self-healing logic: if a workflow disappears (crash, timeout) while the audit row still shows "running", the function reconciles the state to "failed" with appropriate diagnostics.

getAuditResults: Full Report Retrieval

Returns the complete audit payload:

• Base audit metadata
• All crawled pages with extracted fields
• Lighthouse run data (if enabled)
• Full issue inventory with locations and severities

The effective Lighthouse strategy is parsed from the stored JSON config field.

getAuditHistory: Project-Level Listing

Lists recent audits with summary statistics, enabling UI timeline views and trend analysis.

MCP Tool Equivalents

Three thin wrappers in site-audit-tools.ts expose the same data with formatted output:

• get_audit_status — human-readable progress strings
• get_audit_issues — severity-filtered, prioritized lists
• get_audit_pages — paginated page metadata with fetch classification (ok, error, redirect)

Capacity and Quota Enforcement

The engine maintains two hard limits per organization, defined in src/server/features/audit/services/audit-capacity.ts:

Limit	Description	Violation Behavior
maxRunningAudits	Concurrent active workflows	AppError(AUDIT_ALREADY_RUNNING)
maxCapacityUnits	Weighted sum of pages + Lighthouse	AppError(AUDIT_CAPACITY_REACHED)

Capacity units are calculated as: pages + (lighthouseRuns × LIGHTHOUSE_WEIGHT). The MCP layer translates thrown errors into conversational refusal messages for agent consumers.

When an audit completes, fails, or is deleted, its reserved capacity is released, enabling new audits within the same billing period.

Audit Deletion and Cleanup

The deleteAudit flow in AuditService.ts performs ordered teardown:

1. Lookup workflow ID from audit record
2. If status is "running", terminate the Durable Workflow
3. Delete database rows (cascade to pages, issues, Lighthouse data)
4. Invoke env.AUDIT_ENGINE.destroyScratchpad for DO cleanup

This prevents resource leaks and immediately frees quota for replacement audits.

Code Examples

Starting an Audit via API

import { startAudit } from "@/serverFunctions/audit";

const audit = await startAudit({
  startUrl: "https://example.com",
  maxPages: 200,
  lighthouseStrategy: "auto",
});
// Returns: { auditId: "uuid", status: "pending" }

Starting via MCP Tool

import { runSiteAuditTool } from "@/server/mcp/tools/site-audit-tools";

await runSiteAuditTool.handler(
  {
    projectId: "proj_123",
    url: "https://example.com",
    maxPages: 200,
    runLighthouse: true,
  },
  context
);

Polling Status with Self-Healing Detection

import { getAuditStatus } from "@/serverFunctions/audit";

const { status } = await getAuditStatus({
  auditId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  projectId: "proj_123",
});

console.log(`${status.currentPhase}: ${status.pagesCrawled}/${status.pagesTotal}`);
// "crawling: 47/200" or "failed: workflow reconciliation detected orphaned state"

Retrieving Prioritized Issues

import { getAuditIssuesTool } from "@/server/mcp/tools/site-audit-tools";

const { summary, issues } = await getAuditIssuesTool.handler(
  {
    projectId: "proj_123",
    auditId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    severity: "critical",
  },
  context
);

summary.forEach((s) => {
  console.log(`${s.severity}: ${s.title} (${s.count} occurrences)`);
});

Listing Crawled Pages

const { pages, total } = await getAuditPagesTool.handler(
  {
    projectId: "proj_123",
    auditId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    fetchClass: "ok",
    limit: 50,
  },
  context
);

pages.forEach((p) => {
  console.log(`${p.url} — ${p.title} (${p.wordCount} words)`);
});

Key Source Files

File	Purpose
src/server/features/audit/services/AuditService.ts	Core orchestration: start, status, results, history, delete
src/server/features/audit/services/audit-capacity.ts	Tier definitions and capacity calculations
src/server/features/audit/repositories/AuditRepository.ts	Database layer for audits, pages, issues, Lighthouse data
src/server/workflows/site-audit-workflow-helpers.ts	Durable Workflow implementation for crawl orchestration
src/server/mcp/tools/site-audit-tools.ts	MCP wrappers for agent and UI consumption
src/types/schemas/audit.ts	Zod validation schemas (startAuditSchema, etc.)
src/audit-worker.ts	RPC endpoint for DO scratchpad operations
specs/0009-site-audit-crawl-architecture.md	Architectural design documentation

Summary

• The OpenSEO site audit engine runs as a Cloudflare Durable Workflow decoupled from HTTP request handlers, enabling long-running crawls without timeout constraints.
• AuditService.ts handles all client-facing operations with tier-aware capacity enforcement and automatic workflow state reconciliation.
• Temporary crawl state persists to a dedicated Durable Object (env.AUDIT_ENGINE), not the primary database, for performance isolation.
• Issue detection runs during the crawl phase against extracted page metadata, with findings stored in relational tables for efficient querying.
• MCP tool wrappers provide identical functionality through both HTTP API and agent-native interfaces, with formatted output suitable for LLM consumption.

Frequently Asked Questions

What is a Durable Workflow in OpenSEO and why is it used for site audits?

A Durable Workflow is Cloudflare's serverless execution environment that maintains state across multiple invocations. OpenSEO uses it for site audits because crawling hundreds or thousands of pages exceeds standard HTTP request timeouts. The workflow persists its execution position, survives infrastructure restarts, and can run for extended durations while the client polls for completion.

How does OpenSEO prevent a single user from monopolizing audit resources?

The engine implements two guardrails: maxRunningAudits limits concurrent workflows per organization, and maxCapacityUnits enforces a weighted quota on total crawling activity. Both limits are tier-aware (free, paid, self_hosted) and checked atomically before any workflow creation. Exceeded limits return structured errors that the MCP layer translates into user-friendly messages.

Can I run a site audit without Lighthouse performance analysis?

Yes. The lighthouseStrategy parameter accepts "none" to disable performance scanning. This reduces capacity consumption and crawl duration. When set to "auto", the workflow samples representative pages for Lighthouse analysis. The strategy is stored in the audit config and exposed in result queries.

What happens if a site audit workflow crashes mid-crawl?

The getAuditStatus implementation includes reconciliation logic: if a row shows "running" but the corresponding Durable Workflow no longer exists, the service updates the status to "failed" with an appropriate error code. This prevents permanently stuck audits and releases reserved capacity for new requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}