How the Audit Reconciler Fixes Stale Audits Stuck in 'Running' State

The audit reconciler automatically detects and fails audits left in a "running" state by querying their Cloudflare Workflow instance status and marking them failed when the workflow has terminated, errored, or vanished, using both on-demand checks and a scheduled watchdog.

The every-app/open-seo platform orchestrates SEO audits through durable Cloudflare Workflows, yet infrastructure hiccups, unexpected terminations, or lost workflow instances can leave audit records stranded mid-execution. The auditReconciler service guarantees that no audit remains falsely "running" after its underlying compute has disappeared, ensuring the audit table accurately reflects reality.

Two-Pronged Detection Strategy

The reconciler employs complementary lazy and watchdog paths to catch every edge case regardless of traffic patterns.

On-Demand Reconciliation (Lazy Path)

Whenever the UI or an API consumer polls for an audit’s current status, AuditService.getStatus invokes reconcileRunningAudit. This lazy check ensures that frequently accessed audits are reconciled immediately without waiting for a background sweep, providing instant feedback to users while offloading work from the cron job.

Scheduled Watchdog (Cron Path)

A background cron job executes reconcileStaleAudits at regular intervals. This watchdog path queries the database for audits that have been running longer than a configured threshold, ensuring that abandoned audits are eventually failed even if they are never polled again. According to the open-seo source code, this batch process limits each tick to WATCHDOG_BATCH_LIMIT = 100 rows to maintain consistent performance.

Step-by-Step Stale Audit Resolution

Inside src/server/features/audit/services/auditReconciler.ts, the reconciliation flow follows a strict sequence to classify and terminate stranded audits.

1. Fetch Stale Candidates

The watchdog computes a cutoff timestamp using STALE_RUNNING_AFTER_MS (15 minutes) and calls getStaleRunningAudits. This returns up to 100 rows where status = "running" and startedAt exceeds the cutoff, isolating only truly abandoned jobs from healthy long-running crawls.

2. Inspect Workflow Instance State

For each candidate, reconcileRunningAudit first verifies the audit has a workflowInstanceId. It then queries the workflow subsystem via env.SITE_AUDIT_WORKFLOW.get(id) and checks instance.status() to determine if the workflow is active, terminated, or errored.

3. Detect Terminated or Errored Workflows

If the workflow reports status === "errored" or "terminated", the reconciler extracts the error payload. It passes this through classifyAuditError—defined in src/server/lib/audit/audit-errors.ts—to map raw messages to structured error codes like instance_lost or workflow_terminated.

4. Handle Missing Workflow Instances

When the workflow lookup throws (indicating the instance never existed or was purged), the catcher analyzes the error message. If the audit is older than INSTANCE_LOST_GRACE_MS (10 minutes), the system treats this as a definitive instance_lost error. This grace period prevents premature failure of audits that are initializing but haven’t yet persisted their workflow ID.

5. Persist Failure State

Upon confirming a dead workflow, the reconciler calls AuditRepository.failAudit—located in src/server/features/audit/repositories/AuditRepository.ts—to atomically update the audit row. This writes the classified error code, descriptive details, and the phase that was executing when the failure occurred.

6. Emit Telemetry and Continue

The watchdog logs the transition and sends a site_audit:complete event to PostHog via captureServerEvent, explicitly flagging the audit as failed by reconciliation. Individual errors during the batch are caught and isolated so that one failing audit does not abort the entire sweep, allowing the remaining candidates to be processed.

Key Configuration Constants

The reconciler balances responsiveness against false positives through three tunable constants in the source code:

• STALE_RUNNING_AFTER_MS: Set to 15 minutes, this defines how long an audit must remain in running status before it is eligible for the watchdog sweep.
• INSTANCE_LOST_GRACE_MS: Set to 10 minutes, this buffer allows newly created audits time to register their workflow instance before being marked as lost.
• WATCHDOG_BATCH_LIMIT: Capped at 100 audits per cron tick, this prevents the watchdog from overwhelming the database or the Cloudflare Workflow API during recovery from large backlogs.

Implementation Examples

You can invoke reconciliation manually for immediate feedback or register the watchdog as a cron job.

// Manual on-demand reconciliation (used by the UI)
import { reconcileRunningAudit } from "@/server/features/audit/services/auditReconciler";

async function pollAuditStatus(audit) {
  const errorInfo = await reconcileRunningAudit(audit);
  if (errorInfo) {
    // UI can now display a failure with errorInfo.errorCode / errorDetail
    console.log(`Audit failed with code: ${errorInfo.errorCode}`);
  }
}

// Cron-based watchdog (registered in the server's cron config)
import { reconcileStaleAudits } from "@/server/features/audit/services/auditReconciler";

export const auditWatchdog = {
  schedule: "*/5 * * * *", // every 5 minutes
  handler: async () => {
    await reconcileStaleAudits();
  },
};

Summary

• The audit reconciler in every-app/open-seo prevents audits from hanging indefinitely by monitoring their underlying Cloudflare Workflow instances.
• It combines on-demand checks during status polls with a scheduled watchdog cron job to catch orphaned audits.
• Staleness is determined by the 15-minute STALE_RUNNING_AFTER_MS threshold, while a 10-minute INSTANCE_LOST_GRACE_MS prevents false positives for new audits.
• Dead workflows are classified via classifyAuditError, persisted through AuditRepository.failAudit, and reported to PostHog for observability.
• The batch processor limits itself to 100 audits per tick to maintain system stability under load.

Frequently Asked Questions

How does the reconciler know if a Cloudflare Workflow has truly failed?

The reconciler queries the workflow instance status via env.SITE_AUDIT_WORKFLOW.get(id).status(). If the status returns "errored" or "terminated", or if the lookup throws a "not found" error after the 10-minute grace period has elapsed, the audit is marked as failed. This dual check covers both explicit failures and vanished instances.

What prevents the watchdog from failing audits that are simply slow?

Two safeguards prevent premature failure. First, STALE_RUNNING_AFTER_MS (15 minutes) ensures only audits running longer than this threshold are considered for reconciliation. Second, WATCHDOG_BATCH_LIMIT restricts each sweep to 100 candidates, allowing the system to prioritize older, more likely stale audits while healthy long-running jobs continue uninterrupted.

Can I trigger reconciliation manually for a specific audit?

Yes. Import reconcileRunningAudit from src/server/features/audit/services/auditReconciler.ts and pass the audit object. This performs the same workflow status check and failure classification as the watchdog, but immediately and for a single record. The function returns errorInfo if the audit is dead, or null if it remains healthy.

Where is the error classification logic defined?

The mapping from raw workflow errors to structured error codes lives in src/server/lib/audit/audit-errors.ts within the classifyAuditError function. This centralizes error handling, ensuring that the UI and analytics receive consistent error codes like instance_lost or workflow_terminated rather than raw stack traces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}