How to Perform Site Audits with OpenSEO: Complete Technical Implementation

OpenSEO runs full-stack site audits through a resilient workflow orchestrated by AuditService and Cloudflare Durable Workers, crawling domains with configurable concurrency, running Lighthouse performance analysis, and generating structured issue reports while respecting tier-based limits.

Performing comprehensive site audits with OpenSEO begins with invoking the startAudit server function, which triggers a multi-phase durable workflow that discovers URLs, crawls pages with sliding-window concurrency, and optionally executes Lighthouse performance checks. This open-source SEO platform leverages Cloudflare Durable Workers to guarantee audits survive deployments and automatically retry failed steps. The entire process is managed through the AuditService class and persists structured data via the AuditRepository according to the every-app/open-seo source code.

How the OpenSEO Audit Architecture Works

The AuditService Orchestrator

At the heart of every site audit is the AuditService located in src/server/features/audit/services/AuditService.ts. This service handles request validation, capacity reservation, and workflow initialization. When a client requests an audit, AuditService.startAudit performs three critical validations before launching the durable workflow:

• Tier resolution: resolveAuditLimitTier (lines 32-48) checks whether the request originates from a hosted-mode organization (free or paid) or a self-hosted installation, enforcing AUDIT_LIMITS accordingly.
• URL normalization: The service validates and canonicalizes the start URL via src/server/lib/audit/url-policy.ts (lines 23-28), cleaning redirects and enforcing origin policies.
• Capacity reservation: getEstimatedAuditCapacity (lines 59-66) reserves resources based on the requested maxPages and Lighthouse strategy, aborting if limits are exceeded.

After validation, AuditRepository.createAudit (lines 80-88) persists the audit configuration to the database, generating a unique auditId for tracking.

Durable Workers and Workflow Resilience

OpenSEO executes audits inside Cloudflare Durable Workers via the SITE_AUDIT_WORKFLOW aux worker defined in src/audit-worker.ts. The workflow uses pgStep (durable steps) to chunk operations, ensuring each step stays within the 1 MiB output limit and survives compute restarts. A dedicated Durable Object called AuditScratchpad (implemented in src/server/features/audit/AuditScratchpad.ts) maintains the crawl frontier, storing discovered URLs durably across workflow steps.

The 10-Step Site Audit Workflow

The audit process follows a strict orchestration defined in src/server/workflows/siteAuditWorkflowPhases.ts:

1. Request Validation and Tier Resolution

The workflow begins when env.SITE_AUDIT_WORKFLOW.create (lines 9-24 in AuditService.ts) instantiates the durable worker with AuditPhasesParams. The service first resolves the organization's plan tier to determine crawl limits, then normalizes the start URL to its canonical origin.

2. URL Discovery and Seeding

The Discovery Phase (runDiscoveryPhase, line 69 in siteAuditWorkflowPhases.ts) fetches the target domain's robots.txt, extracts sitemap URLs, and seeds the initial crawl queue. These seeds are durably stored in the AuditScratchpad Durable Object, ensuring the crawl can resume after interruptions.

3. Distributed Crawling with AuditScratchpad

The Crawl Phase (runCrawlPhase) executes in chunks via runCrawlChunk (lines 26-44 in src/server/workflows/siteAuditWorkflowCrawl.ts). Each chunk:

• Claims a batch of URLs from the AuditScratchpad
• Fetches pages using sliding-window concurrency controlled by CRAWL_WINDOW
• Persists page metadata to the database via AuditRepository
• Enqueues newly discovered links back to the scratchpad frontier

This chunked approach ensures memory efficiency and provides natural checkpointing for long-running audits.

4. Lighthouse Performance Analysis

If the audit configuration specifies a Lighthouse strategy other than "none", the Lighthouse Phase (runLighthousePhase, lines 88-99) executes. The system:

• Selects representative pages via selectLighthousePages
• Calls DataForSEO's Lighthouse API through fetchLighthouseResult (defined in src/server/lib/audit/lighthouse.ts)
• Stores performance data via AuditRepository.insertLighthouseResults

5. Finalization and Result Retrieval

Upon completion (lines 96-106), the workflow updates the audit status, clears progress KV entries, and optionally destroys the AuditScratchpad Durable Object. Clients retrieve results by calling getAuditStatus or getAuditResults (lines 49-62 in src/serverFunctions/audit.ts), which assembles pages, Lighthouse metrics, and detected issues from the repository.

Implementing Site Audits in Your Application

Triggering Audits from the Client

The following TanStack React Start example demonstrates invoking the audit workflow from a browser client:

import { startAudit } from "@/serverFunctions/audit";
import { useMutation, useQueryClient } from "@tanstack/react-query";

function RunAuditButton({ projectId, startUrl }: { projectId: string; startUrl: string }) {
  const queryClient = useQueryClient();
  const mutation = useMutation({
    mutationFn: (maxPages: number) =>
      startAudit({ projectId, startUrl, maxPages, lighthouseStrategy: "auto" }),
    onSuccess: (data) => {
      // Store the auditId for later status polling
      queryClient.setQueryData(["audit", data.auditId], () => data);
    },
  });

  return (
    <button disabled={mutation.isLoading} onClick={() => mutation.mutate(100)}>
      {mutation.isLoading ? "Starting…" : "Start Site Audit"}
    </button>
  );
}

The startAudit function routes to AuditService.startAudit, which launches the durable workflow and returns immediately with an auditId for asynchronous tracking.

Polling for Audit Status

Monitor audit progress by polling the status endpoint every 5 seconds:

import { getAuditStatus } from "@/serverFunctions/audit";
import { useQuery } from "@tanstack/react-query";

function AuditStatus({ auditId }: { auditId: string }) {
  const { data, isLoading, error } = useQuery(
    ["auditStatus", auditId],
    () => getAuditStatus({ auditId }),
    { refetchInterval: 5000 } // poll every 5 s
  );

  if (isLoading) return <p>Loading status…</p>;
  if (error) return <p>Failed to load status.</p>;

  return (
    <div>
      <p>Status: {data.status}</p>
      <p>Pages crawled: {data.pagesCrawled}/{data.pagesTotal}</p>
      <p>Current phase: {data.currentPhase}</p>
    </div>
  );
}

Downloading Audit Reports

Retrieve complete audit results including all crawled pages, Lighthouse scores, and detected SEO issues:

import { getAuditResults } from "@/serverFunctions/audit";

async function downloadReport(auditId: string) {
  const result = await getAuditResults({ auditId });
  const blob = new Blob([JSON.stringify(result, null, 2)], { type: "application/json" });
  const url = URL.createObjectURL(blob);
  const a = document.createElement("a");
  a.href = url;
  a.download = `audit-${auditId}.json`;
  a.click();
}

Key Source Files and Responsibilities

Understanding the audit implementation requires familiarity with these core files in the every-app/open-seo repository:

• src/serverFunctions/audit.ts — Server-side API entry points exposing startAudit, getAuditStatus, and getAuditResults to clients.

• src/server/features/audit/services/AuditService.ts — Orchestrates the audit lifecycle, handling tier validation, capacity reservation (lines 59-66), and workflow instantiation (lines 9-24).

• src/server/workflows/siteAuditWorkflowPhases.ts — Defines the three-phase orchestration: Discovery (line 69), Crawl, and Lighthouse (lines 88-99), plus finalization logic (lines 96-106).

• src/server/workflows/siteAuditWorkflowCrawl.ts — Implements the chunked crawling algorithm with sliding-window concurrency (CRAWL_WINDOW) via runCrawlChunk (lines 26-44).

• src/server/features/audit/AuditScratchpad.ts — Durable Object maintaining the crawl frontier URL queue across workflow steps.

• src/server/lib/audit/url-policy.ts — Normalizes and validates start URLs, enforcing canonical origins (lines 23-28).

• src/server/lib/audit/lighthouse.ts — Integrates with DataForSEO to fetch performance metrics for selected pages.

• src/server/lib/audit/discovery.ts — Handles robots.txt parsing and sitemap extraction during the discovery phase.

• src/server/features/audit/repositories/AuditRepository.ts — Database abstraction for persisting audit configurations, page crawls, and Lighthouse results.

• src/audit-worker.ts — Cloudflare Worker entry point (configured via wrangler.audit.jsonc) that hosts the durable SITE_AUDIT_WORKFLOW.

Summary

• OpenSEO performs site audits through a distributed architecture combining AuditService with Cloudflare Durable Workers.
• The workflow follows 10 distinct phases: tier validation, URL normalization, capacity reservation, audit record creation, workflow launch, discovery, chunked crawling, Lighthouse analysis, finalization, and result retrieval.
• Durable Objects provide resilience: The AuditScratchpad maintains crawl state across compute restarts, while pgStep chunks ensure compliance with 1 MiB step limits.
• Implementation requires three client operations: calling startAudit to trigger, getAuditStatus to poll progress, and getAuditResults to retrieve the final report.
• Self-hosted and hosted modes enforce different limits through AUDIT_LIMITS validation in AuditService.resolveAuditLimitTier.

Frequently Asked Questions

How does OpenSEO handle large site audits that exceed memory limits?

OpenSEO implements chunked crawling using Durable Steps (pgStep) in src/server/workflows/siteAuditWorkflowCrawl.ts. Each crawl chunk processes a limited batch of URLs within a single step, ensuring output stays below the 1 MiB Cloudflare Workers limit. The AuditScratchpad Durable Object maintains the URL frontier durably, allowing the workflow to pause and resume across steps without losing progress.

What Lighthouse performance strategies are available when running audits?

The lighthouseStrategy parameter accepts "none", "auto", or specific selection criteria. When set to "auto" or a specific strategy, runLighthousePhase (lines 88-99 in siteAuditWorkflowPhases.ts) selects representative pages via selectLighthousePages and fetches metrics through DataForSEO's API. Results are stored via AuditRepository.insertLighthouseResults, linking performance data to specific crawled URLs.

Do self-hosted installations have different audit limits than hosted organizations?

Yes. According to AuditService.resolveAuditLimitTier (lines 32-48 in AuditService.ts), the system distinguishes between hosted-mode organizations (free or paid tiers) and self-hosted installations. The AUDIT_LIMITS configuration enforces different maximum page counts and concurrency limits based on this resolution, with self-hosted instances typically respecting environment-configured boundaries rather than subscription tiers.

Can audits resume if the Cloudflare Worker restarts during execution?

Absolutely. The audit workflow is designed for resilience across deployments. Because crawl state persists in the AuditScratchpad Durable Object (not in-memory) and each phase uses durable steps (pgStep), interrupted audits automatically resume from the last completed chunk. The workflow engine retry logic handles transient failures in runCrawlChunk and fetchLighthouseResult without client intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}