How the Open-SEO Audit System Generates Reports: A 10-Stage Technical Pipeline

The Open-SEO audit system generates comprehensive SEO reports by orchestrating a Cloudflare Workers workflow that discovers URLs, crawls site architecture, executes Lighthouse performance audits, and aggregates technical issues into a structured JSON payload.

The open-source every-app/open-seo repository implements a sophisticated audit pipeline that transforms a single start URL into a detailed technical SEO analysis. Understanding how the open-seo audit system generates reports requires examining its Cloudflare Workers-based workflow architecture, Durable Object scratchpads, and PostgreSQL persistence layer.

The 10-Stage Audit Generation Pipeline

The report generation process runs as a tightly-coupled workflow inside a Cloudflare Workers environment. Each stage feeds sequentially into the next, culminating in a fully-typed JSON report.

Stage 1: Workflow Initiation

The process begins when a client calls the startAudit server function in [src/serverFunctions/audit.ts](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts). This function validates the request payload, checks the user's plan tier against defined limits, and generates a unique auditId that tracks the entire lifecycle of the audit.

Stage 2: Workflow Launch

The AuditService.startAudit method in [src/server/features/audit/services/AuditService.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) inserts a new row into the audit table and instantiates the SITE_AUDIT_WORKFLOW Cloudflare Workers Workflow. This workflow object orchestrates all subsequent phases, handling state persistence and step retries automatically.

Stage 3: Discovery Phase

During the discovery phase, the runDiscoveryPhase function in [src/server/workflows/siteAuditWorkflowPhases.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) extracts URLs from the start page's sitemap, robots.txt file, and any manually seeded URLs. These discovered URLs are written to a scratchpad—a Cloudflare Durable Object (DO) that serves as the crawl frontier, maintaining the queue of URLs awaiting processing.

Stage 4: Crawl Phase

The runCrawlPhase function, imported from [src/server/workflows/siteAuditWorkflowCrawl.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), pulls batches of URLs from the scratchpad DO and fetches each page. It stores page-specific metadata—including HTTP status codes, response sizes, and crawl timestamps—in the audit_pages table. Real-time progress tracking persists in the AuditProgressKV store defined in [src/server/lib/audit/progress-kv.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts), enabling live UI updates during the crawl.

Stage 5: Lighthouse Analysis

If the user enables Lighthouse auditing, the selectLighthousePages function selects a representative sample of crawled pages. The runLighthousePhase function, both located in [src/server/workflows/siteAuditWorkflowPhases.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), executes two Lighthouse runs per URL—one for mobile and one for desktop—via the DataForSEO API. Raw performance results are stored in the audit_lighthouse table before aggregation.

Stage 6: Multipage Checks

After the crawl completes, the runMultipageChecks function imported from [src/server/lib/audit/issues/multipage.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/issues/multipage.ts) executes cross-page analysis. This detects site-wide SEO issues such as duplicate content across URLs and globally missing H1 tags that single-page analysis cannot identify.

Stage 7: Link Integrity Validation

The runScratchpadLinkChecks function in [src/server/workflows/siteAuditWorkflowPhases.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) performs two final SQL-based queries against the scratchpad DO: identifying broken internal links (URLs that return 4xx/5xx status codes) and detecting orphan pages (URLs with no internal links pointing to them).

Stage 8: Issue Persistence

All detected issues—from broken links to duplicate content—are bulk-inserted into the audit_issues table via AuditRepository.insertIssues in [src/server/features/audit/repositories/AuditRepository.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts). This batch operation ensures atomic storage of potentially thousands of SEO issues.

Stage 9: Audit Finalization

The finalizeAudit function in [src/server/workflows/siteAuditWorkflowPhases.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) marks the audit row as completed, persists final crawl statistics, emits a PostHog analytics event, and destroys the scratchpad Durable Object to free resources.

Stage 10: Report Retrieval

Front-end components retrieve the completed report by calling getAuditResults (or getAuditStatus for progress checks) in [src/serverFunctions/audit.ts](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts). This delegates to AuditService.getResults, which aggregates data from the audit, audit_pages, audit_lighthouse, and audit_issues tables into a fully-typed JSON payload returned via TanStack server-function endpoints.

Report Data Composition

The final report generated by the open-seo audit system composes four distinct data layers:

• Audit Metadata – The auditId, start URL, processing status, timestamps, and plan limit constraints stored in the audit table.
• Crawled Pages – Complete URL inventory with HTTP status codes, response sizes, and crawl durations from audit_pages.
• Lighthouse Data – Performance, accessibility, best-practice, and SEO scores for each sampled page across mobile and desktop devices from audit_lighthouse.
• Detected Issues – Structured warnings including broken links, orphan pages, duplicate content, and missing H1 tags from audit_issues.

All data persists in PostgreSQL or SQLite tables defined in [src/db/pg/audit.schema.ts](https://github.com/every-app/open-seo/blob/main/src/db/pg/audit.schema.ts) (or [src/db/schema.ts](https://github.com/every-app/open-seo/blob/main/src/db/schema.ts) for SQLite implementations).

Client-Side Integration

Interacting with the audit pipeline requires three primary server-function calls:

Starting an Audit

// 1️⃣ Start a new audit (client‑side)
const { auditId } = await startAudit({
  startUrl: "https://example.com",
  maxPages: 100,
  lighthouseStrategy: "auto",
});
console.log("Audit launched:", auditId);

Monitoring Progress

// 2️⃣ Poll for status while the workflow runs
const status = await getAuditStatus({ auditId });
console.log(`Audit ${auditId} is ${status.currentPhase} (${status.status})`);

Retrieving Results

// 3️⃣ Retrieve the completed report
const report = await getAuditResults({ auditId });
console.log("Pages crawled:", report.pages.length);
console.log("Lighthouse samples:", report.lighthouse.length);
console.log("Detected issues:", report.issues.length);

These functions map directly to the server-function definitions in src/serverFunctions/audit.ts and delegate to AuditService methods (getStatus, getResults).

Summary

• The open-seo audit system generates reports through a 10-stage Cloudflare Workers workflow orchestrated by AuditService.
• URL discovery utilizes sitemaps and robots.txt, storing targets in a Durable Object scratchpad acting as the crawl frontier.
• Crawl progress tracks in AuditProgressKV while page metadata persists in the audit_pages table.
• Lighthouse audits execute via DataForSEO API for both mobile and desktop when enabled.
• Issue detection combines multipage analysis (duplicate content, missing H1) with scratchpad SQL checks (broken links, orphan pages).
• Final reports aggregate data from audit, audit_pages, audit_lighthouse, and audit_issues tables via getAuditResults.

Frequently Asked Questions

How long does the open-seo audit workflow take to complete?

The total execution time depends on the maxPages limit and whether Lighthouse auditing is enabled. A standard crawl of 100 pages typically completes in 2-3 minutes, while Lighthouse analysis adds approximately 30-60 seconds per sampled page due to DataForSEO API latency. The workflow persists progress in AuditProgressKV, allowing real-time status monitoring via getAuditStatus.

What storage systems does the audit workflow use?

The workflow leverages three distinct storage mechanisms: Cloudflare Durable Objects (the scratchpad DO) for temporary crawl frontier management, Cloudflare KV (AuditProgressKV) for real-time progress updates, and PostgreSQL or SQLite (configured via src/db/pg/audit.schema.ts) for persistent storage of audit metadata, pages, Lighthouse results, and detected issues.

Can I customize which SEO checks run during the audit?

While the core workflow in siteAuditWorkflowPhases.ts executes a fixed pipeline, you can control Lighthouse execution via the lighthouseStrategy parameter in startAudit. Advanced customization requires modifying the runMultipageChecks import in src/server/lib/audit/issues/multipage.ts or adding custom issue detectors to the pipeline before the persist issues stage.

How does the system handle very large sites with thousands of pages?

The system respects plan tier limits enforced during startAudit validation. The crawl phase processes URLs in configurable batches pulled from the scratchpad DO, preventing memory exhaustion. For large sites, the maxPages parameter caps the crawl scope, and the scratchpad-based architecture allows the workflow to resume from interruption without re-crawling completed URLs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}