# How the OpenSEO Audit Reconciler Detects and Resolves Stuck Site Audits

> Discover how the OpenSEO audit reconciler prevents stuck site audits from consuming resources. Learn how it detects and resolves zombie audits, saving error details and freeing capacity.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-30

---

**The OpenSEO audit reconciler is a safety mechanism that detects "zombie" audits stuck in a `running` state and marks them as failed, preserving error details and freeing capacity for new site audits.**

OpenSEO runs site-audit workflows on Cloudflare Workers, but when a workflow crashes due to OOM, CPU limits, or instance expiration, the `audits` table row can remain in a `running` state forever. The **audit reconciler** prevents these phantom jobs from consuming capacity by polling workflow status and persisting failures when necessary. According to the every-app/open-seo source code, this system operates through two coordinated entry points: a lazy reconciler triggered by client requests and a scheduled watchdog cron job.

## Why the Audit Reconciler Is Necessary

Cloudflare Workers provide ephemeral compute, but their lifecycle events do not automatically cascade to your database. If a site-audit workflow is killed by a deployment reset or exceeds memory limits, the `audits` row retains its `running` status. Without reconciliation, these stuck audits would permanently occupy capacity units and confuse users polling for status updates. The reconciler acts as a circuit breaker that validates workflow health against Cloudflare's API and flips stale rows to `failed` when the underlying instance is dead.

## Two Entry Points for Audit Reconciliation

The reconciliation pipeline is invoked through two distinct triggers designed for both immediate user feedback and background cleanup.

### Lazy Reconciliation via AuditService.getStatus

When a client polls an audit’s status through the API, `AuditService.getStatus` in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) (lines 38-44) proactively checks for zombies. If the audit status is `running`, the service immediately calls `reconcileRunningAudit` to verify the workflow instance is still alive. If the instance has terminated, the row is updated to `failed` before the response returns, ensuring users never see a stuck progress bar.

### Watchdog Cron Job

A scheduled cron running every 15 minutes executes `reconcileStaleAudits` from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) (lines 95-103). This watchdog sweeps all audits exceeding the stale threshold and batches them for examination. The cron is registered in [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts) (lines 16-24) and provides a backstop for audits that clients may never poll again.

## Core Reconciliation Logic in reconcileRunningAudit

The heart of the system is the `reconcileRunningAudit` function exported from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts). It implements a four-phase health check that distinguishes between temporary blips and actual workflow death.

```typescript
export async function reconcileRunningAudit(
  audit: RunningAudit,
): Promise<AuditErrorInfo | null> {
  if (!audit.workflowInstanceId) return null;

  let errorInfo: AuditErrorInfo | null = null;
  try {
    const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId);
    const status = await instance.status();

    if (status.status === "errored" || status.status === "terminated") {
      errorInfo = status.error
        ? classifyAuditError(
            typeof status.status === "string"
              ? status.error
              : (status.error.message ?? JSON.stringify(status.error)),
          )
        : { errorCode: "unknown", errorDetail: `Workflow instance ${status.status}` };
    }
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    if (!/not[ _]?found/i.test(message)) return null;
    if (!isOlderThan(audit.startedAt, INSTANCE_LOST_GRACE_MS)) return null;

    errorInfo = {
      errorCode: "instance_lost",
      errorDetail: "Workflow instance not found",
    };
  }

  if (!errorInfo) return null;

  await AuditRepository.failAudit(audit.id, audit.workflowInstanceId, {
    ...errorInfo,
    failedPhase: audit.currentPhase,
  });
  return errorInfo;
}

```

**Phase 1: Instance Lookup.** The function queries Cloudflare's Workflow API using `env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId)` to fetch the live instance object.

**Phase 2: Status Inspection.** If the workflow reports `errored` or `terminated`, the raw error is normalized through `classifyAuditError` (defined in [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts)) to generate structured `AuditErrorInfo`.

**Phase 3: Expiration Handling.** If the API throws a "not found" error (indicating the instance aged out of Cloudflare's retention), the reconciler applies a **10-minute grace period** (`INSTANCE_LOST_GRACE_MS`). Only audits older than this grace period receive the `instance_lost` error code.

**Phase 4: Atomic Persistence.** Finally, `AuditRepository.failAudit` updates the database row only if it remains `running`, guarding against race conditions where a slow workflow finishes legitimately during reconciliation.

## Handling Stale Audits in the Watchdog

The `reconcileStaleAudits` function relies on `getStaleRunningAudits` to identify candidates for reconciliation. This query handles timestamp dialect differences between Postgres and Cloudflare D1.

```typescript
async function getStaleRunningAudits(cutoff: Date, limit: number) {
  const iso = cutoff.toISOString();
  const startedBefore =
    getDatabaseProvider() === "postgres"
      ? iso
      : iso.replace("T", " ").slice(0, 19); // D1 stores "YYYY-MM-DD HH:MM:SS"

  return db.query.audits.findMany({
    where: and(
      eq(audits.status, "running"),
      lt(audits.startedAt, startedBefore),
    ),
    orderBy: audits.startedAt,
    limit,
  });
}

```

The watchdog uses **15 minutes** (`STALE_RUNNING_AFTER_MS`) as the cutoff threshold and processes candidates in batches of **100** (`WATCHDOG_BATCH_LIMIT`) to prevent cron timeouts. The timestamp normalization ensures the `lt()` (less than) comparison functions correctly whether the database expects ISO-8601 or SQL-standard datetime strings.

## Error Classification and Persistence

When workflows fail, raw error messages are processed by `classifyAuditError` to produce consistent error codes for downstream analytics. The reconciler then persists the failure via `AuditRepository.failAudit`, which updates the `audits` table with:

- `errorCode`: Either "instance_lost", "unknown", or a classified workflow error
- `errorDetail`: The sanitized error message or JSON string
- `failedPhase`: The audit phase (e.g., "crawling", "analysis") when the failure occurred

This structured approach allows the UI to display meaningful failure reasons rather than generic timeout messages.

## Telemetry and Observability

When the watchdog successfully reconciles an audit, it emits a PostHog event (`site_audit:complete`) with `reconciled_by: "watchdog"` and metadata including `error_code` and `pages_crawled` (as implemented in lines 13-21 of `reconcileStaleAudits`). This telemetry enables product teams to track reconciliation frequency and identify systemic workflow stability issues.

## Practical Code Examples

### Manually Invoking the Reconciler for Debugging

You can trigger reconciliation manually to verify specific audit states without waiting for the cron.

```typescript
import { reconcileRunningAudit } from "@/server/features/audit/services/auditReconciler";
import { db } from "@/db";
import { audits } from "@/db/schema";
import { eq } from "drizzle-orm";

async function debugReconcile(auditId: string) {
  const audit = await db.query.audits.findFirst({
    where: eq(audits.id, auditId),
  });

  if (!audit) throw new Error("Audit not found");
  
  const result = await reconcileRunningAudit({
    id: audit.id,
    workflowInstanceId: audit.workflowInstanceId,
    startedAt: audit.startedAt,
    currentPhase: audit.currentPhase,
  });

  console.log("Reconciliation result:", result);
}

```

### Self-Healing Status Checks in the UI

The status endpoint automatically self-heals when users query stuck audits.

```typescript
// Inside AuditService.getStatus
if (audit.status === "running") {
  const reconciled = await reconcileRunningAudit(audit);
  if (reconciled) {
    // Reload to return fresh failed status
    audit = (await AuditRepository.getAuditForProject(auditId, projectId)) ?? audit;
  }
}

```

### Cron Registration

The watchdog is already wired in [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts) and runs on the Cloudflare Workers cron schedule.

```typescript
// src/server.ts
export default {
  async scheduled(controller: ScheduledController, env: Env, ctx: ExecutionContext) {
    await withPgClient(async () => {
      await reconcileStaleAudits();
    });
  },
};

```

## Summary

- **The audit reconciler prevents capacity leaks** by detecting Cloudflare Workflow instances that have crashed or expired while the database still lists them as `running`.
- **Two triggers handle reconciliation**: a lazy check during `AuditService.getStatus` calls and a watchdog cron that runs every 15 minutes.
- **`reconcileRunningAudit`** verifies instance health through Cloudflare's API, handles "not found" errors after a 10-minute grace period (`INSTANCE_LOST_GRACE_MS`), and atomically updates the database via `AuditRepository.failAudit`.
- **The watchdog batches work** (100 audits max) and normalizes timestamps for both Postgres and D1 compatibility.
- **Structured error classification** via `classifyAuditError` ensures failed audits retain actionable diagnostic information for users.

## Frequently Asked Questions

### What causes a site audit to become "stuck" in OpenSEO?

A site audit becomes stuck when its Cloudflare Workflow instance terminates unexpectedly—due to out-of-memory errors, CPU limits, deployment resets, or instance retention expiration—while the corresponding row in the `audits` table remains marked as `running`. The workflow dies, but the database status is never updated to reflect the failure.

### How long does OpenSEO wait before marking a lost workflow as failed?

OpenSEO applies a **10-minute grace period** (`INSTANCE_LOST_GRACE_MS`) when a workflow instance returns a "not found" error. If the audit started less than 10 minutes ago, the reconciler assumes the instance might still be initializing and leaves the status unchanged. After 10 minutes, it marks the audit as failed with the error code `instance_lost`.

### Can I manually trigger the audit reconciler for debugging?

Yes, you can import `reconcileRunningAudit` from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) and pass it a `RunningAudit` object containing the audit ID, workflow instance ID, start time, and current phase. This is useful for debugging specific production audits without waiting for the 15-minute cron cycle or API polling to trigger the lazy reconciler.

### How does the reconciler handle different database providers?

The `getStaleRunningAudits` function dynamically formats the cutoff timestamp based on the active database provider. For Postgres, it uses ISO-8601 format (`2024-01-01T12:00:00.000Z`), while for Cloudflare D1, it reformats the string to SQL-standard datetime (`YYYY-MM-DD HH:MM:SS`). This ensures the "running before X" query executes correctly regardless of whether the deployment uses Postgres or D1.