# Site Audit Implementation and Limits in OpenSEO: A Deep Dive into Cloudflare Workers Architecture

> Discover how OpenSEO implements site audits using Cloudflare Workers. Learn about configurable page budgets and tier-specific limits for efficient website crawling and performance checks.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-08-09

---

**Site audits in OpenSEO are implemented as durable Cloudflare Workers workflows that crawl websites with configurable page budgets, run Lighthouse performance checks, and enforce tier-specific limits ranging from 50 pages on free plans to 10,000 pages on paid or self-hosted deployments.**

OpenSEO's audit system combines a BFS-style crawler, SQLite persistence via D1, and real-time progress tracking through KV storage. This article examines the complete implementation—from service-layer validation through workflow orchestration to the specific limits that govern usage.

## Starting a Site Audit: Service Layer and Validation

Audit initiation flows through `AuditService.startAudit` in **[`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)**. The method performs six sequential operations before launching any crawler.

First, `resolveAuditLimitTier` determines whether the request originates from a hosted environment with a paid plan (checked via `customerHasManagedAccess` / `customerHasPaidPlan`) or from a self-hosted deployment—the latter defaulting to paid-tier privileges. 

Next, `clampAuditMaxPages` bounds the user-supplied `maxPages` between `MIN_AUDIT_PAGES` (10) and `PAID_MAX_AUDIT_PAGES` (10,000), with a default of `DEFAULT_AUDIT_PAGES` (50). This clamping occurs in **[`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts)**.

Capacity enforcement follows: `getEstimatedAuditCapacity` calculates required units (pages + Lighthouse checks), then validates against `maxCapacityUnits` and `maxRunningAudits` for the resolved tier. Exceeding these triggers `AUDIT_CAPACITY_REACHED` or `AUDIT_ALREADY_RUNNING` errors.

Finally, `AuditRepository.createAudit` inserts the audit record, and `env.SITE_AUDIT_WORKFLOW.create` launches the durable workflow with `auditId`, `projectId`, `startUrl`, and parsed `AuditConfig`.

## The Crawl Phase: BFS Implementation with Memory Constraints

The crawl engine resides in **[`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts)**, implementing `runCrawlPhase` with explicit constraints for Cloudflare Workers' ~1 MiB per-step limit.

### Frontier Management

Two in-memory queues manage discovery: `linkQueue` for page links and `sitemapQueue` for sitemap URLs. The durable step returns only a minimal set of new frontier URLs to stay under step-size limits. URL eligibility is determined by `shouldQueueCrawlLink`, which verifies same-origin, `isCrawlableUrl`, robots.txt permissions, and deduplication against `visited` and `queued` sets.

### Batch Processing

Concurrency is capped at `CRAWL_CONCURRENCY` (25) parallel fetches via `crawlPage`. Each page receives a deterministic ID through `deterministicAuditRowId`, ensuring idempotent retries. After fetching, `runPageReporters(page)` executes issue checkers for broken links, performance problems, and other diagnostics.

### Persistence Pattern

`AuditRepository.insertCrawledBatch` commits pages and detected issues to D1, while `AuditProgressKV.pushCrawledUrls` in **[`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)** updates the live progress feed. The phase terminates when `maxPages` is reached or the frontier empties, returning a summary with URLs, status codes, titles, and capped internal link lists.

## Lighthouse and Issue Detection Phases

Following successful crawl completion, the workflow orchestrated by **[`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts)** proceeds to Lighthouse analysis. For `strategy === "auto"`, this samples up to 20 pages; `full` and `none` strategies adjust accordingly. Results are persisted via `AuditRepository.insertLighthouseResults` using the same deterministic ID scheme for consistency.

Issue detection runs during crawling via the reporter system, categorizing problems by severity and type for final aggregation.

## Complete Tier Limits and Capacity Model

All limits are centrally defined in **[`src/shared/audit-limits.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-limits.ts)** and enforced through **[`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts)**.

| Constant | Value | Description |
|----------|-------|-------------|
| `MIN_AUDIT_PAGES` | 10 | Absolute minimum crawl size |
| `DEFAULT_AUDIT_PAGES` | 50 | Default when unspecified |
| `FREE_MAX_AUDIT_PAGES` | 50 | Free-tier per-audit ceiling |
| `PAID_MAX_AUDIT_PAGES` | 10,000 | Paid/self-hosted per-audit ceiling |
| `AUDIT_LIMITS.free.maxCapacityUnits` | 2,000 | Free concurrent capacity budget |
| `AUDIT_LIMITS.free.maxRunningAudits` | 1 | Free concurrent audit limit |
| `AUDIT_LIMITS.paid.maxCapacityUnits` | 100,000 | Paid concurrent capacity budget |
| `AUDIT_LIMITS.paid.maxRunningAudits` | `Infinity` | Paid concurrent audit limit |

The capacity unit formula weights pages and Lighthouse checks to prevent resource abuse, with paid tiers receiving effectively unlimited concurrent audits.

## Client API and Usage Examples

### Starting an Audit

```typescript
import { startAudit } from "@/serverFunctions/audit";

const { auditId } = await startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  maxPages: 200,
  lighthouseStrategy: "auto", // "auto" | "full" | "none"
});

```

### Polling Status

```typescript
import { getStatus } from "@/serverFunctions/audit";

const status = await getStatus({ auditId, projectId: "proj_123" });
console.log(status.pagesCrawled, status.pagesTotal, status.currentPhase);

```

### Retrieving Results

```typescript
import { getResults } from "@/serverFunctions/audit";

const result = await getResults({ auditId, projectId: "proj_123" });
console.log(result.pages.length, result.lighthouse.length, result.issues.length);

```

### Canceling an Audit

```typescript
import { remove } from "@/serverFunctions/audit";

await remove({ auditId, projectId: "proj_123" });

```

## Data Flow Architecture

The audit system spans multiple storage systems:

- **D1 (SQLite)**: Persistent storage for audit metadata, crawled pages, Lighthouse results, and issues via [`AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/AuditRepository.ts)
- **KV**: Real-time progress streaming through [`progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/progress-kv.ts)
- **Workflow state**: Durable execution state managed by Cloudflare Workers runtime

This separation allows the UI to poll lightweight KV progress during execution while fetching comprehensive results from D1 upon completion.

## Summary

- **Entry point**: `AuditService.startAudit` in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) validates tiers, clamps pages, and reserves capacity before workflow creation
- **Crawl engine**: `runCrawlPhase` in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) implements BFS crawling with 25-page concurrency and ~1 MiB step-size limits
- **Limits source**: [`src/shared/audit-limits.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-limits.ts) defines constants; [`audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/audit-capacity.ts) implements tier resolution and enforcement
- **Free tier**: 50 pages maximum, 1 concurrent audit, 2,000 capacity units
- **Paid/self-hosted**: 10,000 pages maximum, unlimited concurrent audits, 100,000 capacity units
- **Completion**: `AuditRepository.completeAudit` finalizes status; `AuditService.remove` handles cancellation and cleanup

## Frequently Asked Questions

### How does OpenSEO prevent free-tier users from exceeding crawl limits?

The `resolveAuditLimitTier` function in [`audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/audit-capacity.ts) detects hosted versus self-hosted deployments and checks `customerHasPaidPlan` status. For free-tier requests, `FREE_MAX_AUDIT_PAGES` (50) overrides any larger `maxPages` value, and `AUDIT_LIMITS.free.maxRunningAudits` (1) prevents multiple concurrent audits. Exceeding either triggers specific error codes: `AUDIT_PAGE_LIMIT_EXCEEDED` or `AUDIT_ALREADY_RUNNING`.

### Why does the crawl phase use in-memory queues instead of durable storage?

Cloudflare Workers impose a ~1 MiB limit on individual workflow step outputs. To stay within this constraint, `runCrawlPhase` maintains `linkQueue` and `sitemapQueue` in memory and only returns minimal frontier state between steps. Deterministic IDs via `deterministicAuditRowId` ensure that retries remain idempotent despite this ephemeral queue approach.

### What happens when a site audit is cancelled mid-crawl?

`AuditService.remove` in [`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts) terminates the running workflow through the Workers runtime API and deletes the audit row from D1. Partial crawl data is not retained; the audit must be restarted from the beginning. This design prioritizes clean resource reclamation over resumability.

### Can self-hosted deployments customize audit limits beyond the paid tier defaults?

Self-hosted deployments automatically receive paid-tier limits through `resolveAuditLimitTier`, which checks `customerHasManagedAccess` to distinguish hosted from self-hosted environments. While the source code hardcodes `PAID_MAX_AUDIT_PAGES` at 10,000, self-hosted operators can modify [`src/shared/audit-limits.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-limits.ts) constants and `AUDIT_LIMITS.paid` values before deployment to establish custom ceilings.