Site Audit Implementation and Limits in OpenSEO: A Deep Dive into Cloudflare Workers Architecture

Site audits in OpenSEO are implemented as durable Cloudflare Workers workflows that crawl websites with configurable page budgets, run Lighthouse performance checks, and enforce tier-specific limits ranging from 50 pages on free plans to 10,000 pages on paid or self-hosted deployments.

OpenSEO's audit system combines a BFS-style crawler, SQLite persistence via D1, and real-time progress tracking through KV storage. This article examines the complete implementation—from service-layer validation through workflow orchestration to the specific limits that govern usage.

Starting a Site Audit: Service Layer and Validation

Audit initiation flows through AuditService.startAudit in src/server/features/audit/services/AuditService.ts. The method performs six sequential operations before launching any crawler.

First, resolveAuditLimitTier determines whether the request originates from a hosted environment with a paid plan (checked via customerHasManagedAccess / customerHasPaidPlan) or from a self-hosted deployment—the latter defaulting to paid-tier privileges.

Next, clampAuditMaxPages bounds the user-supplied maxPages between MIN_AUDIT_PAGES (10) and PAID_MAX_AUDIT_PAGES (10,000), with a default of DEFAULT_AUDIT_PAGES (50). This clamping occurs in src/server/features/audit/services/audit-capacity.ts.

Capacity enforcement follows: getEstimatedAuditCapacity calculates required units (pages + Lighthouse checks), then validates against maxCapacityUnits and maxRunningAudits for the resolved tier. Exceeding these triggers AUDIT_CAPACITY_REACHED or AUDIT_ALREADY_RUNNING errors.

Finally, AuditRepository.createAudit inserts the audit record, and env.SITE_AUDIT_WORKFLOW.create launches the durable workflow with auditId, projectId, startUrl, and parsed AuditConfig.

The Crawl Phase: BFS Implementation with Memory Constraints

The crawl engine resides in src/server/workflows/siteAuditWorkflowCrawl.ts, implementing runCrawlPhase with explicit constraints for Cloudflare Workers' ~1 MiB per-step limit.

Frontier Management

Two in-memory queues manage discovery: linkQueue for page links and sitemapQueue for sitemap URLs. The durable step returns only a minimal set of new frontier URLs to stay under step-size limits. URL eligibility is determined by shouldQueueCrawlLink, which verifies same-origin, isCrawlableUrl, robots.txt permissions, and deduplication against visited and queued sets.

Batch Processing

Concurrency is capped at CRAWL_CONCURRENCY (25) parallel fetches via crawlPage. Each page receives a deterministic ID through deterministicAuditRowId, ensuring idempotent retries. After fetching, runPageReporters(page) executes issue checkers for broken links, performance problems, and other diagnostics.

Persistence Pattern

AuditRepository.insertCrawledBatch commits pages and detected issues to D1, while AuditProgressKV.pushCrawledUrls in src/server/lib/audit/progress-kv.ts updates the live progress feed. The phase terminates when maxPages is reached or the frontier empties, returning a summary with URLs, status codes, titles, and capped internal link lists.

Lighthouse and Issue Detection Phases

Following successful crawl completion, the workflow orchestrated by src/server/workflows/SiteAuditWorkflow.ts proceeds to Lighthouse analysis. For strategy === "auto", this samples up to 20 pages; full and none strategies adjust accordingly. Results are persisted via AuditRepository.insertLighthouseResults using the same deterministic ID scheme for consistency.

Issue detection runs during crawling via the reporter system, categorizing problems by severity and type for final aggregation.

Complete Tier Limits and Capacity Model

All limits are centrally defined in src/shared/audit-limits.ts and enforced through src/server/features/audit/services/audit-capacity.ts.

Constant	Value	Description
MIN_AUDIT_PAGES	10	Absolute minimum crawl size
DEFAULT_AUDIT_PAGES	50	Default when unspecified
FREE_MAX_AUDIT_PAGES	50	Free-tier per-audit ceiling
PAID_MAX_AUDIT_PAGES	10,000	Paid/self-hosted per-audit ceiling
AUDIT_LIMITS.free.maxCapacityUnits	2,000	Free concurrent capacity budget
AUDIT_LIMITS.free.maxRunningAudits	1	Free concurrent audit limit
AUDIT_LIMITS.paid.maxCapacityUnits	100,000	Paid concurrent capacity budget
AUDIT_LIMITS.paid.maxRunningAudits	Infinity	Paid concurrent audit limit

The capacity unit formula weights pages and Lighthouse checks to prevent resource abuse, with paid tiers receiving effectively unlimited concurrent audits.

Client API and Usage Examples

Starting an Audit

import { startAudit } from "@/serverFunctions/audit";

const { auditId } = await startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  maxPages: 200,
  lighthouseStrategy: "auto", // "auto" | "full" | "none"
});

Polling Status

import { getStatus } from "@/serverFunctions/audit";

const status = await getStatus({ auditId, projectId: "proj_123" });
console.log(status.pagesCrawled, status.pagesTotal, status.currentPhase);

Retrieving Results

import { getResults } from "@/serverFunctions/audit";

const result = await getResults({ auditId, projectId: "proj_123" });
console.log(result.pages.length, result.lighthouse.length, result.issues.length);

Canceling an Audit

import { remove } from "@/serverFunctions/audit";

await remove({ auditId, projectId: "proj_123" });

Data Flow Architecture

The audit system spans multiple storage systems:

• D1 (SQLite): Persistent storage for audit metadata, crawled pages, Lighthouse results, and issues via AuditRepository.ts
• KV: Real-time progress streaming through progress-kv.ts
• Workflow state: Durable execution state managed by Cloudflare Workers runtime

This separation allows the UI to poll lightweight KV progress during execution while fetching comprehensive results from D1 upon completion.

Summary

• Entry point: AuditService.startAudit in src/server/features/audit/services/AuditService.ts validates tiers, clamps pages, and reserves capacity before workflow creation
• Crawl engine: runCrawlPhase in src/server/workflows/siteAuditWorkflowCrawl.ts implements BFS crawling with 25-page concurrency and ~1 MiB step-size limits
• Limits source: src/shared/audit-limits.ts defines constants; audit-capacity.ts implements tier resolution and enforcement
• Free tier: 50 pages maximum, 1 concurrent audit, 2,000 capacity units
• Paid/self-hosted: 10,000 pages maximum, unlimited concurrent audits, 100,000 capacity units
• Completion: AuditRepository.completeAudit finalizes status; AuditService.remove handles cancellation and cleanup

Frequently Asked Questions

How does OpenSEO prevent free-tier users from exceeding crawl limits?

The resolveAuditLimitTier function in audit-capacity.ts detects hosted versus self-hosted deployments and checks customerHasPaidPlan status. For free-tier requests, FREE_MAX_AUDIT_PAGES (50) overrides any larger maxPages value, and AUDIT_LIMITS.free.maxRunningAudits (1) prevents multiple concurrent audits. Exceeding either triggers specific error codes: AUDIT_PAGE_LIMIT_EXCEEDED or AUDIT_ALREADY_RUNNING.

Why does the crawl phase use in-memory queues instead of durable storage?

Cloudflare Workers impose a ~1 MiB limit on individual workflow step outputs. To stay within this constraint, runCrawlPhase maintains linkQueue and sitemapQueue in memory and only returns minimal frontier state between steps. Deterministic IDs via deterministicAuditRowId ensure that retries remain idempotent despite this ephemeral queue approach.

What happens when a site audit is cancelled mid-crawl?

AuditService.remove in AuditService.ts terminates the running workflow through the Workers runtime API and deletes the audit row from D1. Partial crawl data is not retained; the audit must be restarted from the beginning. This design prioritizes clean resource reclamation over resumability.

Can self-hosted deployments customize audit limits beyond the paid tier defaults?

Self-hosted deployments automatically receive paid-tier limits through resolveAuditLimitTier, which checks customerHasManagedAccess to distinguish hosted from self-hosted environments. While the source code hardcodes PAID_MAX_AUDIT_PAGES at 10,000, self-hosted operators can modify src/shared/audit-limits.ts constants and AUDIT_LIMITS.paid values before deployment to establish custom ceilings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}