# OpenSEO Audit System Best Practices: A Complete Technical Guide

> Master OpenSEO audit system best practices with this technical guide. Learn to manage tier limits, bot challenges, and progress polling for optimal workflow engine implementation. Improve your SEO audits now.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: best-practices
- Published: 2026-08-21

---

**The OpenSEO audit system is a tightly-coupled server-side workflow engine that coordinates URL discovery, Lighthouse analysis, and issue detection through TanStack React-Start server functions, requiring proper handling of tier limits, bot challenges, and progress polling for optimal implementation.**

The `every-app/open-seo` repository provides a comprehensive infrastructure for programmatic SEO analysis through automated crawling and performance auditing. Understanding the architecture of this **open-seo audit system** ensures efficient API utilization while avoiding common integration pitfalls related to billing quotas and crawler policies.

## Understanding the OpenSEO Audit Architecture

The system comprises four tightly integrated layers: server functions, service logic, workflow orchestration, and data persistence. In [`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts), TanStack React-Start server functions expose the public API surface, including `startAudit`, `getAuditStatus`, `getAuditResults`, `getCrawlProgress`, and `deleteAudit`. Each function validates input through Zod schemas defined in [`src/types/schemas/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/types/schemas/audit.ts) before forwarding requests to the business logic layer.

The **AuditService** class in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) implements core orchestration: resolving billing limits, creating audit records, launching workflows, and persisting results through `AuditRepository`. Sequential execution phases—including `discoverUrls`, `runLighthouse`, and `runMultipageChecks`—are defined in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) and coordinated via the `SiteAuditWorkflow` engine.

## The Audit Lifecycle

### Initiating an Audit

Client applications trigger audits by calling the `startAudit` server function with parameters validated against `startAuditSchema`. Required parameters include `startUrl` and `lighthouseStrategy` (accepting `desktop`, `mobile`, or `none`), with optional `maxPages` constrained by organizational tier limits enforced via `AuditService.resolveAuditLimitTier` in [`src/shared/audit-limits.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-limits.ts).

### Crawl and Analysis Workflow

The workflow engine executes discrete phases that parse [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt), extract internal links using crawlability policies from [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts), and execute Lighthouse audits through `AuditRepository.insertLighthouseResults`. Issue detection runs multipage checks via `runMultipageChecks` and per-page reporters through `runScratchpadLinkChecks`, with findings stored via `AuditRepository.insertIssues`.

### Real-Time Progress Monitoring

Throughout execution, `AuditProgressKV` (implemented in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)) maintains transient state for live updates. Clients poll `getCrawlProgress` to retrieve percentage completion and blocked page lists, enabling responsive UI feedback during lengthy crawl operations.

### Retrieving Results

Upon completion, `getAuditResults` returns structured payloads containing page-level Lighthouse metrics, detected SEO issues cataloged in [`src/shared/audit-issues.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-issues.ts), and summary statistics including pages crawled and error rates.

### Cleanup and Resource Management

The `deleteAudit` function removes database records and clears KV storage entries. The server automatically handles KV cleanup upon workflow completion regardless of success status.

## Common Pitfalls and Optimization Strategies

**Exceeding Page Limits**: Passing `maxPages` values exceeding tier-based quotas results in rejected requests. Query `AuditService.resolveAuditLimitTier` before initiating audits to determine permissible ranges for your organizational plan.

**Bot Challenge Errors**: Cloudflare and WAFs may block the `OpenSEO-Audit` user agent, generating 429 errors. Add this exact user agent string to target site allow-lists, as documented in [`src/shared/audit-issues.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-issues.ts).

**Missing Robots.txt**: When unavailable, the crawler falls back to permissive policies that may over-crawl. Ensure target domains serve valid [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) files for optimal coverage.

**Lighthouse Strategy Misconfiguration**: Choosing `none` disables performance metrics entirely. Select `desktop` or `mobile` unless explicitly excluding performance data, as the UI defaults to `desktop` for comprehensive analysis.

**Stale Progress States**: Implement client-side retry mechanisms when polling `getCrawlProgress`, as workflow crashes may temporarily leave stale data in `AuditProgressKV` before server-side cleanup occurs.

## Implementation Examples

### Client-Side Audit Lifecycle

The following TypeScript example demonstrates the complete audit flow from initiation through result retrieval:

```typescript
import { startAudit, getAuditStatus, getCrawlProgress, getAuditResults } from '@/serverFunctions/audit';

async function runCompleteAudit(url: string) {
  // Initiate audit with desktop Lighthouse strategy
  const { auditId } = await startAudit({ 
    startUrl: url,
    lighthouseStrategy: 'desktop',
    // Omit maxPages to use tier-based default
  });
  
  // Poll for completion
  while (true) {
    const status = await getAuditStatus({ auditId });
    if (status.state === 'completed') break;
    
    const progress = await getCrawlProgress({ auditId });
    console.log(`Progress: ${progress.percentage}% (${progress.pagesCrawled} pages)`);
    await new Promise(r => setTimeout(r, 2000));
  }
  
  // Retrieve final results
  const results = await getAuditResults({ auditId });
  return results;
}

```

### Server-Side API Wrappers

For custom frontend integrations, wrap the TanStack server functions using `createServerFn`:

```typescript
import { createServerFn } from '@tanstack/react-start';
import * as AuditAPI from '@/serverFunctions/audit';

export const auditAPI = {
  start: createServerFn(AuditAPI.startAudit),
  status: createServerFn(AuditAPI.getAuditStatus),
  results: createServerFn(AuditAPI.getAuditResults),
};

```

## Key Configuration Files

- **[`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts)**: Public API surface exposing start, status, results, progress, and delete operations.
- **[`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)**: Core orchestration logic including limit resolution and audit lifecycle management.
- **[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)**: Sequential crawl and analysis phase definitions.
- **[`src/types/schemas/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/types/schemas/audit.ts)**: Zod validation schemas for type-safe request handling.
- **[`src/shared/audit-limits.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-limits.ts)**: Tier-based page cap definitions and quota enforcement.
- **[`src/shared/audit-issues.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-issues.ts)**: Canonical SEO issue registry for UI rendering and CSV export.
- **[`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)**: KV storage implementation for real-time progress tracking.
- **[`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts)**: Crawlability determination functions respecting robots.txt and domain constraints.

## Summary

- The **open-seo audit system** utilizes TanStack React-Start server functions in [`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts) to expose a type-safe API for comprehensive SEO analysis.
- **AuditService.ts** enforces tier-based page limits through `resolveAuditLimitTier` and coordinates multi-phase workflows defined in [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts).
- The workflow progresses through **Discovery**, **Lighthouse Analysis**, and **Issue Detection** phases, with real-time updates stored in `AuditProgressKV`.
- Clients must poll `getCrawlProgress` during execution and handle `OpenSEO-Audit` user agent allow-listing to prevent bot challenges.
- Always validate `maxPages` against organizational quotas and select appropriate `lighthouseStrategy` values to ensure comprehensive metrics collection.

## Frequently Asked Questions

### How do I handle page limit restrictions when starting an audit?

The system enforces tier-based page caps through `AuditService.resolveAuditLimitTier` in [`src/shared/audit-limits.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-limits.ts). Rather than hardcoding values, query your organization's current tier to determine the maximum permissible `maxPages` parameter, or omit the parameter entirely to accept the default quota for your plan.

### What causes "bot challenge" errors during crawls and how can I prevent them?

These errors occur when WAFs like Cloudflare block the `OpenSEO-Audit` user agent. To prevent blocking, add this exact user agent string to your target site's allow-list or firewall rules. The specific issue definitions are cataloged in [`src/shared/audit-issues.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/audit-issues.ts) for reference.

### Can I disable Lighthouse performance testing for faster crawls?

Yes, pass `lighthouseStrategy: 'none'` to `startAudit` to skip performance analysis. However, this eliminates Core Web Vitals and other metric data from results. For complete SEO audits, use `desktop` or `mobile` strategies, which the UI defaults to for comprehensive coverage.

### How does the system handle real-time progress updates during long-running audits?

The workflow engine writes progress states to `AuditProgressKV` (implemented in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)) during each phase. Clients poll the `getCrawlProgress` server function to retrieve percentage completion and page counts. The server automatically clears KV entries upon workflow completion or failure, though clients should implement retry logic for resilience against transient stale states.