OpenSEO Audit System Best Practices: A Complete Technical Guide

The OpenSEO audit system is a tightly-coupled server-side workflow engine that coordinates URL discovery, Lighthouse analysis, and issue detection through TanStack React-Start server functions, requiring proper handling of tier limits, bot challenges, and progress polling for optimal implementation.

The every-app/open-seo repository provides a comprehensive infrastructure for programmatic SEO analysis through automated crawling and performance auditing. Understanding the architecture of this open-seo audit system ensures efficient API utilization while avoiding common integration pitfalls related to billing quotas and crawler policies.

Understanding the OpenSEO Audit Architecture

The system comprises four tightly integrated layers: server functions, service logic, workflow orchestration, and data persistence. In src/serverFunctions/audit.ts, TanStack React-Start server functions expose the public API surface, including startAudit, getAuditStatus, getAuditResults, getCrawlProgress, and deleteAudit. Each function validates input through Zod schemas defined in src/types/schemas/audit.ts before forwarding requests to the business logic layer.

The AuditService class in src/server/features/audit/services/AuditService.ts implements core orchestration: resolving billing limits, creating audit records, launching workflows, and persisting results through AuditRepository. Sequential execution phases—including discoverUrls, runLighthouse, and runMultipageChecks—are defined in src/server/workflows/siteAuditWorkflowPhases.ts and coordinated via the SiteAuditWorkflow engine.

The Audit Lifecycle

Initiating an Audit

Client applications trigger audits by calling the startAudit server function with parameters validated against startAuditSchema. Required parameters include startUrl and lighthouseStrategy (accepting desktop, mobile, or none), with optional maxPages constrained by organizational tier limits enforced via AuditService.resolveAuditLimitTier in src/shared/audit-limits.ts.

Crawl and Analysis Workflow

The workflow engine executes discrete phases that parse robots.txt, extract internal links using crawlability policies from src/server/lib/audit/url-policy.ts, and execute Lighthouse audits through AuditRepository.insertLighthouseResults. Issue detection runs multipage checks via runMultipageChecks and per-page reporters through runScratchpadLinkChecks, with findings stored via AuditRepository.insertIssues.

Real-Time Progress Monitoring

Throughout execution, AuditProgressKV (implemented in src/server/lib/audit/progress-kv.ts) maintains transient state for live updates. Clients poll getCrawlProgress to retrieve percentage completion and blocked page lists, enabling responsive UI feedback during lengthy crawl operations.

Retrieving Results

Upon completion, getAuditResults returns structured payloads containing page-level Lighthouse metrics, detected SEO issues cataloged in src/shared/audit-issues.ts, and summary statistics including pages crawled and error rates.

Cleanup and Resource Management

The deleteAudit function removes database records and clears KV storage entries. The server automatically handles KV cleanup upon workflow completion regardless of success status.

Common Pitfalls and Optimization Strategies

Exceeding Page Limits: Passing maxPages values exceeding tier-based quotas results in rejected requests. Query AuditService.resolveAuditLimitTier before initiating audits to determine permissible ranges for your organizational plan.

Bot Challenge Errors: Cloudflare and WAFs may block the OpenSEO-Audit user agent, generating 429 errors. Add this exact user agent string to target site allow-lists, as documented in src/shared/audit-issues.ts.

Missing Robots.txt: When unavailable, the crawler falls back to permissive policies that may over-crawl. Ensure target domains serve valid robots.txt files for optimal coverage.

Lighthouse Strategy Misconfiguration: Choosing none disables performance metrics entirely. Select desktop or mobile unless explicitly excluding performance data, as the UI defaults to desktop for comprehensive analysis.

Stale Progress States: Implement client-side retry mechanisms when polling getCrawlProgress, as workflow crashes may temporarily leave stale data in AuditProgressKV before server-side cleanup occurs.

Implementation Examples

Client-Side Audit Lifecycle

The following TypeScript example demonstrates the complete audit flow from initiation through result retrieval:

import { startAudit, getAuditStatus, getCrawlProgress, getAuditResults } from '@/serverFunctions/audit';

async function runCompleteAudit(url: string) {
  // Initiate audit with desktop Lighthouse strategy
  const { auditId } = await startAudit({ 
    startUrl: url,
    lighthouseStrategy: 'desktop',
    // Omit maxPages to use tier-based default
  });
  
  // Poll for completion
  while (true) {
    const status = await getAuditStatus({ auditId });
    if (status.state === 'completed') break;
    
    const progress = await getCrawlProgress({ auditId });
    console.log(`Progress: ${progress.percentage}% (${progress.pagesCrawled} pages)`);
    await new Promise(r => setTimeout(r, 2000));
  }
  
  // Retrieve final results
  const results = await getAuditResults({ auditId });
  return results;
}

Server-Side API Wrappers

For custom frontend integrations, wrap the TanStack server functions using createServerFn:

import { createServerFn } from '@tanstack/react-start';
import * as AuditAPI from '@/serverFunctions/audit';

export const auditAPI = {
  start: createServerFn(AuditAPI.startAudit),
  status: createServerFn(AuditAPI.getAuditStatus),
  results: createServerFn(AuditAPI.getAuditResults),
};

Key Configuration Files

• src/serverFunctions/audit.ts: Public API surface exposing start, status, results, progress, and delete operations.
• src/server/features/audit/services/AuditService.ts: Core orchestration logic including limit resolution and audit lifecycle management.
• src/server/workflows/siteAuditWorkflowPhases.ts: Sequential crawl and analysis phase definitions.
• src/types/schemas/audit.ts: Zod validation schemas for type-safe request handling.
• src/shared/audit-limits.ts: Tier-based page cap definitions and quota enforcement.
• src/shared/audit-issues.ts: Canonical SEO issue registry for UI rendering and CSV export.
• src/server/lib/audit/progress-kv.ts: KV storage implementation for real-time progress tracking.
• src/server/lib/audit/url-policy.ts: Crawlability determination functions respecting robots.txt and domain constraints.

Summary

• The open-seo audit system utilizes TanStack React-Start server functions in src/serverFunctions/audit.ts to expose a type-safe API for comprehensive SEO analysis.
• AuditService.ts enforces tier-based page limits through resolveAuditLimitTier and coordinates multi-phase workflows defined in siteAuditWorkflowPhases.ts.
• The workflow progresses through Discovery, Lighthouse Analysis, and Issue Detection phases, with real-time updates stored in AuditProgressKV.
• Clients must poll getCrawlProgress during execution and handle OpenSEO-Audit user agent allow-listing to prevent bot challenges.
• Always validate maxPages against organizational quotas and select appropriate lighthouseStrategy values to ensure comprehensive metrics collection.

Frequently Asked Questions

How do I handle page limit restrictions when starting an audit?

The system enforces tier-based page caps through AuditService.resolveAuditLimitTier in src/shared/audit-limits.ts. Rather than hardcoding values, query your organization's current tier to determine the maximum permissible maxPages parameter, or omit the parameter entirely to accept the default quota for your plan.

What causes "bot challenge" errors during crawls and how can I prevent them?

These errors occur when WAFs like Cloudflare block the OpenSEO-Audit user agent. To prevent blocking, add this exact user agent string to your target site's allow-list or firewall rules. The specific issue definitions are cataloged in src/shared/audit-issues.ts for reference.

Can I disable Lighthouse performance testing for faster crawls?

Yes, pass lighthouseStrategy: 'none' to startAudit to skip performance analysis. However, this eliminates Core Web Vitals and other metric data from results. For complete SEO audits, use desktop or mobile strategies, which the UI defaults to for comprehensive coverage.

How does the system handle real-time progress updates during long-running audits?

The workflow engine writes progress states to AuditProgressKV (implemented in src/server/lib/audit/progress-kv.ts) during each phase. Clients poll the getCrawlProgress server function to retrieve percentage completion and page counts. The server automatically clears KV entries upon workflow completion or failure, though clients should implement retry logic for resilience against transient stale states.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}