# How the Open-SEO Audit System Generates Reports: A 10-Stage Technical Pipeline

> Discover how the Open-SEO audit system generates detailed reports via a 10-stage technical pipeline. Learn about URL discovery, site crawling, Lighthouse audits, and JSON aggregation.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-21

---

**The Open-SEO audit system generates comprehensive SEO reports by orchestrating a Cloudflare Workers workflow that discovers URLs, crawls site architecture, executes Lighthouse performance audits, and aggregates technical issues into a structured JSON payload.**

The open-source **every-app/open-seo** repository implements a sophisticated audit pipeline that transforms a single start URL into a detailed technical SEO analysis. Understanding how the **open-seo audit system generates reports** requires examining its Cloudflare Workers-based workflow architecture, Durable Object scratchpads, and PostgreSQL persistence layer.

## The 10-Stage Audit Generation Pipeline

The report generation process runs as a tightly-coupled workflow inside a Cloudflare Workers environment. Each stage feeds sequentially into the next, culminating in a fully-typed JSON report.

### Stage 1: Workflow Initiation

The process begins when a client calls the `startAudit` server function in [[`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts)](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts). This function validates the request payload, checks the user's plan tier against defined limits, and generates a unique `auditId` that tracks the entire lifecycle of the audit.

### Stage 2: Workflow Launch

The `AuditService.startAudit` method in [[`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) inserts a new row into the `audit` table and instantiates the `SITE_AUDIT_WORKFLOW` Cloudflare Workers Workflow. This workflow object orchestrates all subsequent phases, handling state persistence and step retries automatically.

### Stage 3: Discovery Phase

During the discovery phase, the `runDiscoveryPhase` function in [[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) extracts URLs from the start page's sitemap, robots.txt file, and any manually seeded URLs. These discovered URLs are written to a **scratchpad**—a Cloudflare Durable Object (DO) that serves as the crawl frontier, maintaining the queue of URLs awaiting processing.

### Stage 4: Crawl Phase

The `runCrawlPhase` function, imported from [[`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), pulls batches of URLs from the scratchpad DO and fetches each page. It stores page-specific metadata—including HTTP status codes, response sizes, and crawl timestamps—in the `audit_pages` table. Real-time progress tracking persists in the `AuditProgressKV` store defined in [[`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts), enabling live UI updates during the crawl.

### Stage 5: Lighthouse Analysis

If the user enables Lighthouse auditing, the `selectLighthousePages` function selects a representative sample of crawled pages. The `runLighthousePhase` function, both located in [[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), executes two Lighthouse runs per URL—one for mobile and one for desktop—via the **DataForSEO API**. Raw performance results are stored in the `audit_lighthouse` table before aggregation.

### Stage 6: Multipage Checks

After the crawl completes, the `runMultipageChecks` function imported from [[`src/server/lib/audit/issues/multipage.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/issues/multipage.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/issues/multipage.ts) executes cross-page analysis. This detects site-wide SEO issues such as duplicate content across URLs and globally missing H1 tags that single-page analysis cannot identify.

### Stage 7: Link Integrity Validation

The `runScratchpadLinkChecks` function in [[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) performs two final SQL-based queries against the scratchpad DO: identifying broken internal links (URLs that return 4xx/5xx status codes) and detecting orphan pages (URLs with no internal links pointing to them).

### Stage 8: Issue Persistence

All detected issues—from broken links to duplicate content—are bulk-inserted into the `audit_issues` table via `AuditRepository.insertIssues` in [[`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts). This batch operation ensures atomic storage of potentially thousands of SEO issues.

### Stage 9: Audit Finalization

The `finalizeAudit` function in [[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) marks the audit row as **completed**, persists final crawl statistics, emits a PostHog analytics event, and destroys the scratchpad Durable Object to free resources.

### Stage 10: Report Retrieval

Front-end components retrieve the completed report by calling `getAuditResults` (or `getAuditStatus` for progress checks) in [[`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts)](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts). This delegates to `AuditService.getResults`, which aggregates data from the `audit`, `audit_pages`, `audit_lighthouse`, and `audit_issues` tables into a fully-typed JSON payload returned via TanStack server-function endpoints.

## Report Data Composition

The final report generated by the open-seo audit system composes four distinct data layers:

- **Audit Metadata** – The `auditId`, start URL, processing status, timestamps, and plan limit constraints stored in the `audit` table.
- **Crawled Pages** – Complete URL inventory with HTTP status codes, response sizes, and crawl durations from `audit_pages`.
- **Lighthouse Data** – Performance, accessibility, best-practice, and SEO scores for each sampled page across mobile and desktop devices from `audit_lighthouse`.
- **Detected Issues** – Structured warnings including broken links, orphan pages, duplicate content, and missing H1 tags from `audit_issues`.

All data persists in PostgreSQL or SQLite tables defined in [[`src/db/pg/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/pg/audit.schema.ts)](https://github.com/every-app/open-seo/blob/main/src/db/pg/audit.schema.ts) (or [[`src/db/schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/schema.ts)](https://github.com/every-app/open-seo/blob/main/src/db/schema.ts) for SQLite implementations).

## Client-Side Integration

Interacting with the audit pipeline requires three primary server-function calls:

### Starting an Audit

```typescript
// 1️⃣ Start a new audit (client‑side)
const { auditId } = await startAudit({
  startUrl: "https://example.com",
  maxPages: 100,
  lighthouseStrategy: "auto",
});
console.log("Audit launched:", auditId);

```

### Monitoring Progress

```typescript
// 2️⃣ Poll for status while the workflow runs
const status = await getAuditStatus({ auditId });
console.log(`Audit ${auditId} is ${status.currentPhase} (${status.status})`);

```

### Retrieving Results

```typescript
// 3️⃣ Retrieve the completed report
const report = await getAuditResults({ auditId });
console.log("Pages crawled:", report.pages.length);
console.log("Lighthouse samples:", report.lighthouse.length);
console.log("Detected issues:", report.issues.length);

```

These functions map directly to the server-function definitions in [`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts) and delegate to `AuditService` methods (`getStatus`, `getResults`).

## Summary

- The **open-seo audit system** generates reports through a 10-stage Cloudflare Workers workflow orchestrated by `AuditService`.
- **URL discovery** utilizes sitemaps and robots.txt, storing targets in a Durable Object scratchpad acting as the crawl frontier.
- **Crawl progress** tracks in `AuditProgressKV` while page metadata persists in the `audit_pages` table.
- **Lighthouse audits** execute via DataForSEO API for both mobile and desktop when enabled.
- **Issue detection** combines multipage analysis (duplicate content, missing H1) with scratchpad SQL checks (broken links, orphan pages).
- **Final reports** aggregate data from `audit`, `audit_pages`, `audit_lighthouse`, and `audit_issues` tables via `getAuditResults`.

## Frequently Asked Questions

### How long does the open-seo audit workflow take to complete?

The total execution time depends on the `maxPages` limit and whether Lighthouse auditing is enabled. A standard crawl of 100 pages typically completes in 2-3 minutes, while Lighthouse analysis adds approximately 30-60 seconds per sampled page due to DataForSEO API latency. The workflow persists progress in `AuditProgressKV`, allowing real-time status monitoring via `getAuditStatus`.

### What storage systems does the audit workflow use?

The workflow leverages three distinct storage mechanisms: **Cloudflare Durable Objects** (the scratchpad DO) for temporary crawl frontier management, **Cloudflare KV** (`AuditProgressKV`) for real-time progress updates, and **PostgreSQL or SQLite** (configured via [`src/db/pg/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/pg/audit.schema.ts)) for persistent storage of audit metadata, pages, Lighthouse results, and detected issues.

### Can I customize which SEO checks run during the audit?

While the core workflow in [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) executes a fixed pipeline, you can control Lighthouse execution via the `lighthouseStrategy` parameter in `startAudit`. Advanced customization requires modifying the `runMultipageChecks` import in [`src/server/lib/audit/issues/multipage.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/issues/multipage.ts) or adding custom issue detectors to the pipeline before the `persist issues` stage.

### How does the system handle very large sites with thousands of pages?

The system respects plan tier limits enforced during `startAudit` validation. The crawl phase processes URLs in configurable batches pulled from the scratchpad DO, preventing memory exhaustion. For large sites, the `maxPages` parameter caps the crawl scope, and the scratchpad-based architecture allows the workflow to resume from interruption without re-crawling completed URLs.