Performance Implications of Running Complex Audits in Open-SEO: A Technical Deep Dive

Running complex audits in Open-SEO increases resource consumption proportionally with page count, adds 30–90 seconds of latency per Lighthouse batch, and scales SQL query complexity with the size of the persisted dataset, though built-in caps and checkpointing prevent runaway resource usage.

Open-SEO is an open-source site audit engine that orchestrates multi-phase workflows across Cloudflare Workers. Understanding the performance implications of running complex audits—those configured with high page limits, enabled Lighthouse checks, or extensive issue detection—helps developers optimize throughput, predict execution time, and avoid hitting Worker CPU quotas.

How Open-SEO Structures Audit Workflows

The audit pipeline is divided into distinct phases, each with specific computational costs. Knowing how these phases interact clarifies where bottlenecks emerge when scaling audit complexity.

Discovery Phase

The runDiscoveryPhase function in src/server/workflows/siteAuditWorkflowPhases.ts initiates the workflow by pulling URLs from the start page’s sitemap, robots.txt, and other entry points. Performance costs here scale with the number of URLs discovered relative to the maxPages limit and the parsing overhead of robots.txt rules.

Crawl Phase

Implemented in src/server/workflows/siteAuditWorkflowCrawl.ts, the crawl phase visits each discovered URL, stores the raw response, and updates the frontier. This phase generates significant HTTP request latency for every page fetched, sustained database writes via AuditRepository.insertPage, and maintains frontier state in a Durable Object (DO)-based scratchpad. Each URL adds linear overhead in network I/O and disk persistence.

Lighthouse Phase (Optional)

The runLighthousePhase function calls external DataForSEO endpoints for a sampled subset of pages. When lighthouseStrategy is set to "auto", the system samples up to 10 pages and runs both mobile and desktop checks, resulting in 20 external API round-trips per audit. Each batch introduces 30–90 seconds of latency and persists results via AuditRepository.insertLighthouseResults, adding measurable overhead to the total execution window.

Multipage Checks Phase

Located in src/server/lib/audit/issues/multipage.ts, this phase executes SQL-based issue detectors—including broken-link and orphan-page analysis—against the audit_pages table. Query complexity increases with the number of persisted pages, potentially forcing the SQL planner into slower join strategies when analyzing link graphs across thousands of rows.

Finalization Phase

The finalizeAudit function aggregates detected issues, records audit completion via AuditRepository.completeAudit, and cleans up temporary scratchpad state. Very large audits may generate substantial issue datasets, increasing the cost of final aggregation and potential CSV export operations.

Critical Performance Cost Drivers

Several configuration parameters directly determine the resource intensity of an audit.

Page Budget and maxPages Configuration

The AuditConfig.maxPages parameter defaults to 50 and is enforced by clampAuditMaxPages in src/server/features/audit/services/audit-capacity.ts. Increasing this value to thousands of pages linearly scales HTTP fetches, database row creation, and frontier memory storage. A 10,000-page audit performs approximately 10,000 sequential HTTP requests plus corresponding database insertions, easily consuming multiple minutes of Worker CPU time and saturating I/O quotas.

Lighthouse Sampling Strategy

The lighthouseStrategy configuration determines external API load. The "auto" setting samples 10 pages and runs dual-device checks, while "none" removes this overhead entirely. Disabling Lighthouse eliminates the 30–90 second per-batch latency and avoids consumption of external API credits.

SQL-Based Issue Detection

Multipage checks execute pure SQL over the pages table. While the codebase mitigates risk by limiting the crawl frontier to SEED_RPC_BATCH = 2000 and checkpointing intermediate results, audits approaching the upper maxPages limit can still push the database query planner into expensive join operations, particularly for link-graph analyses.

Checkpointing and Memory Pressure

The workflow uses Cloudflare Workers’ pgStep checkpointing to survive failures across long-running audits. Each checkpoint adds minor serialization overhead but safeguards progress. The audit scratchpad—a DO stored in Cloudflare KV—maintains the frontier, link graph, and temporary metadata. Its size grows with discovered URL counts, and very large audits may approach DO storage limits, degrading read/write performance.

Built-In Safeguards Against Resource Exhaustion

Open-SEO implements hard limits to prevent complex audits from exhausting platform resources.

Per-Tier Audit Limits (AUDIT_LIMITS)

The src/shared/audit-limits.ts file defines hard caps for free, paid, and self-hosted tiers. These limits constrain the maximum maxPages value regardless of user input, ensuring audits remain within predictable resource bounds for the hosting tier’s infrastructure.

Capacity Clamping (clampAuditMaxPages)

Before execution, AuditService in src/server/features/audit/services/AuditService.ts passes the requested maxPages through clampAuditMaxPages in src/server/features/audit/services/audit-capacity.ts. This function enforces tier-specific boundaries, preventing accidental configuration of resource-prohibitive audits that could trigger Worker timeouts or memory limits.

Code Examples: Configuring Audit Complexity

Below are practical implementations showing how to adjust audit complexity and the corresponding service-layer handling.

Launch a lightweight audit with default constraints:

// 1️⃣ Simple audit: 50 pages max, no Lighthouse
await fetch("/api/audit", {
  method: "POST",
  body: JSON.stringify({
    projectId: "proj_123",
    startUrl: "https://example.com",
  }),
});

Launch a resource-intensive audit with expanded scope:

// 2️⃣ Complex audit: 200 pages with Lighthouse sampling
await fetch("/api/audit", {
  method: "POST",
  body: JSON.stringify({
    projectId: "proj_123",
    startUrl: "https://example.com",
    maxPages: 200,                 // Increases crawl workload linearly
    lighthouseStrategy: "auto",   // Adds up to 20 external Lighthouse checks
  }),
});

Server-side enforcement of capacity limits:

// 3️⃣ Inside src/serverFunctions/audit.ts, AuditService respects limits
import { AuditService } from "@/server/features/audit/services/AuditService";

await AuditService.start({
  projectId,
  startUrl,
  maxPages: data.maxPages,               // Automatically clamped by clampAuditMaxPages()
  lighthouseStrategy: data.lighthouseStrategy ?? "auto",
});

Summary

• Page count linearly scales resource usage: Higher maxPages values in src/server/workflows/siteAuditWorkflowCrawl.ts directly increase HTTP latency, database writes, and scratchpad memory consumption.
• Lighthouse adds significant external latency: Enabling "auto" sampling triggers up to 20 DataForSEO API calls per audit, adding 30–90 seconds of blocking latency.
• SQL checks worsen with dataset size: Multipage issue detection in src/server/lib/audit/issues/multipage.ts executes complex joins that slow as the audit_pages table grows.
• Hard caps prevent runaway execution: The AUDIT_LIMITS constant in src/shared/audit-limits.ts and the clampAuditMaxPages function enforce maximum boundaries appropriate to the hosting tier.

Frequently Asked Questions

How does increasing maxPages affect Open-SEO audit performance?

Increasing maxPages linearly scales the number of HTTP requests, database insertions via AuditRepository.insertPage, and frontier state stored in the DO scratchpad. According to src/server/workflows/siteAuditWorkflowCrawl.ts, each additional page requires a network fetch and persistence operation, meaning a 10,000-page audit performs roughly 10,000 times the work of a single-page audit, easily exhausting Worker CPU quotas without proper checkpointing.

What is the latency cost of enabling Lighthouse in complex audits?

Enabling the "auto" lighthouseStrategy triggers runLighthousePhase to sample up to 10 pages and run both mobile and desktop checks against DataForSEO’s API. This generates 20 external round-trips that typically add 30–90 seconds of latency per audit batch. Disabling Lighthouse by setting the strategy to "none" removes this overhead entirely, as implemented in src/server/workflows/siteAuditWorkflowPhases.ts.

How does Open-SEO prevent audits from consuming excessive resources?

The platform enforces tier-specific limits through AUDIT_LIMITS in src/shared/audit-limits.ts and clamps user input via clampAuditMaxPages in src/server/features/audit/services/audit-capacity.ts. These safeguards run before the crawl phase begins, ensuring requested page counts never exceed infrastructure capacity regardless of API input.

Where are multipage SQL checks executed in the codebase?

Multipage issue detection occurs in src/server/lib/audit/issues/multipage.ts within the runMultipageChecks function. This module runs SQL queries against the audit_pages table to detect issues like broken links and orphan pages. Query performance degrades with table size because link-graph analyses require expensive joins across the entire persisted dataset.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}