How the OpenSEO Audit Reconciler Detects and Resolves Stuck Site Audits

The OpenSEO audit reconciler is a safety mechanism that detects "zombie" audits stuck in a running state and marks them as failed, preserving error details and freeing capacity for new site audits.

OpenSEO runs site-audit workflows on Cloudflare Workers, but when a workflow crashes due to OOM, CPU limits, or instance expiration, the audits table row can remain in a running state forever. The audit reconciler prevents these phantom jobs from consuming capacity by polling workflow status and persisting failures when necessary. According to the every-app/open-seo source code, this system operates through two coordinated entry points: a lazy reconciler triggered by client requests and a scheduled watchdog cron job.

Why the Audit Reconciler Is Necessary

Cloudflare Workers provide ephemeral compute, but their lifecycle events do not automatically cascade to your database. If a site-audit workflow is killed by a deployment reset or exceeds memory limits, the audits row retains its running status. Without reconciliation, these stuck audits would permanently occupy capacity units and confuse users polling for status updates. The reconciler acts as a circuit breaker that validates workflow health against Cloudflare's API and flips stale rows to failed when the underlying instance is dead.

Two Entry Points for Audit Reconciliation

The reconciliation pipeline is invoked through two distinct triggers designed for both immediate user feedback and background cleanup.

Lazy Reconciliation via AuditService.getStatus

When a client polls an audit’s status through the API, AuditService.getStatus in src/server/features/audit/services/AuditService.ts (lines 38-44) proactively checks for zombies. If the audit status is running, the service immediately calls reconcileRunningAudit to verify the workflow instance is still alive. If the instance has terminated, the row is updated to failed before the response returns, ensuring users never see a stuck progress bar.

Watchdog Cron Job

A scheduled cron running every 15 minutes executes reconcileStaleAudits from src/server/features/audit/services/auditReconciler.ts (lines 95-103). This watchdog sweeps all audits exceeding the stale threshold and batches them for examination. The cron is registered in src/server.ts (lines 16-24) and provides a backstop for audits that clients may never poll again.

Core Reconciliation Logic in reconcileRunningAudit

The heart of the system is the reconcileRunningAudit function exported from src/server/features/audit/services/auditReconciler.ts. It implements a four-phase health check that distinguishes between temporary blips and actual workflow death.

export async function reconcileRunningAudit(
  audit: RunningAudit,
): Promise<AuditErrorInfo | null> {
  if (!audit.workflowInstanceId) return null;

  let errorInfo: AuditErrorInfo | null = null;
  try {
    const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId);
    const status = await instance.status();

    if (status.status === "errored" || status.status === "terminated") {
      errorInfo = status.error
        ? classifyAuditError(
            typeof status.status === "string"
              ? status.error
              : (status.error.message ?? JSON.stringify(status.error)),
          )
        : { errorCode: "unknown", errorDetail: `Workflow instance ${status.status}` };
    }
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    if (!/not[ _]?found/i.test(message)) return null;
    if (!isOlderThan(audit.startedAt, INSTANCE_LOST_GRACE_MS)) return null;

    errorInfo = {
      errorCode: "instance_lost",
      errorDetail: "Workflow instance not found",
    };
  }

  if (!errorInfo) return null;

  await AuditRepository.failAudit(audit.id, audit.workflowInstanceId, {
    ...errorInfo,
    failedPhase: audit.currentPhase,
  });
  return errorInfo;
}

Phase 1: Instance Lookup. The function queries Cloudflare's Workflow API using env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId) to fetch the live instance object.

Phase 2: Status Inspection. If the workflow reports errored or terminated, the raw error is normalized through classifyAuditError (defined in src/server/lib/audit/audit-errors.ts) to generate structured AuditErrorInfo.

Phase 3: Expiration Handling. If the API throws a "not found" error (indicating the instance aged out of Cloudflare's retention), the reconciler applies a 10-minute grace period (INSTANCE_LOST_GRACE_MS). Only audits older than this grace period receive the instance_lost error code.

Phase 4: Atomic Persistence. Finally, AuditRepository.failAudit updates the database row only if it remains running, guarding against race conditions where a slow workflow finishes legitimately during reconciliation.

Handling Stale Audits in the Watchdog

The reconcileStaleAudits function relies on getStaleRunningAudits to identify candidates for reconciliation. This query handles timestamp dialect differences between Postgres and Cloudflare D1.

async function getStaleRunningAudits(cutoff: Date, limit: number) {
  const iso = cutoff.toISOString();
  const startedBefore =
    getDatabaseProvider() === "postgres"
      ? iso
      : iso.replace("T", " ").slice(0, 19); // D1 stores "YYYY-MM-DD HH:MM:SS"

  return db.query.audits.findMany({
    where: and(
      eq(audits.status, "running"),
      lt(audits.startedAt, startedBefore),
    ),
    orderBy: audits.startedAt,
    limit,
  });
}

The watchdog uses 15 minutes (STALE_RUNNING_AFTER_MS) as the cutoff threshold and processes candidates in batches of 100 (WATCHDOG_BATCH_LIMIT) to prevent cron timeouts. The timestamp normalization ensures the lt() (less than) comparison functions correctly whether the database expects ISO-8601 or SQL-standard datetime strings.

Error Classification and Persistence

When workflows fail, raw error messages are processed by classifyAuditError to produce consistent error codes for downstream analytics. The reconciler then persists the failure via AuditRepository.failAudit, which updates the audits table with:

• errorCode: Either "instance_lost", "unknown", or a classified workflow error
• errorDetail: The sanitized error message or JSON string
• failedPhase: The audit phase (e.g., "crawling", "analysis") when the failure occurred

This structured approach allows the UI to display meaningful failure reasons rather than generic timeout messages.

Telemetry and Observability

When the watchdog successfully reconciles an audit, it emits a PostHog event (site_audit:complete) with reconciled_by: "watchdog" and metadata including error_code and pages_crawled (as implemented in lines 13-21 of reconcileStaleAudits). This telemetry enables product teams to track reconciliation frequency and identify systemic workflow stability issues.

Practical Code Examples

Manually Invoking the Reconciler for Debugging

You can trigger reconciliation manually to verify specific audit states without waiting for the cron.

import { reconcileRunningAudit } from "@/server/features/audit/services/auditReconciler";
import { db } from "@/db";
import { audits } from "@/db/schema";
import { eq } from "drizzle-orm";

async function debugReconcile(auditId: string) {
  const audit = await db.query.audits.findFirst({
    where: eq(audits.id, auditId),
  });

  if (!audit) throw new Error("Audit not found");
  
  const result = await reconcileRunningAudit({
    id: audit.id,
    workflowInstanceId: audit.workflowInstanceId,
    startedAt: audit.startedAt,
    currentPhase: audit.currentPhase,
  });

  console.log("Reconciliation result:", result);
}

Self-Healing Status Checks in the UI

The status endpoint automatically self-heals when users query stuck audits.

// Inside AuditService.getStatus
if (audit.status === "running") {
  const reconciled = await reconcileRunningAudit(audit);
  if (reconciled) {
    // Reload to return fresh failed status
    audit = (await AuditRepository.getAuditForProject(auditId, projectId)) ?? audit;
  }
}

Cron Registration

The watchdog is already wired in src/server.ts and runs on the Cloudflare Workers cron schedule.

// src/server.ts
export default {
  async scheduled(controller: ScheduledController, env: Env, ctx: ExecutionContext) {
    await withPgClient(async () => {
      await reconcileStaleAudits();
    });
  },
};

Summary

• The audit reconciler prevents capacity leaks by detecting Cloudflare Workflow instances that have crashed or expired while the database still lists them as running.
• Two triggers handle reconciliation: a lazy check during AuditService.getStatus calls and a watchdog cron that runs every 15 minutes.
• reconcileRunningAudit verifies instance health through Cloudflare's API, handles "not found" errors after a 10-minute grace period (INSTANCE_LOST_GRACE_MS), and atomically updates the database via AuditRepository.failAudit.
• The watchdog batches work (100 audits max) and normalizes timestamps for both Postgres and D1 compatibility.
• Structured error classification via classifyAuditError ensures failed audits retain actionable diagnostic information for users.

Frequently Asked Questions

What causes a site audit to become "stuck" in OpenSEO?

A site audit becomes stuck when its Cloudflare Workflow instance terminates unexpectedly—due to out-of-memory errors, CPU limits, deployment resets, or instance retention expiration—while the corresponding row in the audits table remains marked as running. The workflow dies, but the database status is never updated to reflect the failure.

How long does OpenSEO wait before marking a lost workflow as failed?

OpenSEO applies a 10-minute grace period (INSTANCE_LOST_GRACE_MS) when a workflow instance returns a "not found" error. If the audit started less than 10 minutes ago, the reconciler assumes the instance might still be initializing and leaves the status unchanged. After 10 minutes, it marks the audit as failed with the error code instance_lost.

Can I manually trigger the audit reconciler for debugging?

Yes, you can import reconcileRunningAudit from src/server/features/audit/services/auditReconciler.ts and pass it a RunningAudit object containing the audit ID, workflow instance ID, start time, and current phase. This is useful for debugging specific production audits without waiting for the 15-minute cron cycle or API polling to trigger the lazy reconciler.

How does the reconciler handle different database providers?

The getStaleRunningAudits function dynamically formats the cutoff timestamp based on the active database provider. For Postgres, it uses ISO-8601 format (2024-01-01T12:00:00.000Z), while for Cloudflare D1, it reformats the string to SQL-standard datetime (YYYY-MM-DD HH:MM:SS). This ensures the "running before X" query executes correctly regardless of whether the deployment uses Postgres or D1.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}