# How the Audit Reconciler Fixes Stale Audits Stuck in 'Running' State

> Fix stale audits stuck in running state. The audit reconciler automatically detects and fails audits by checking Cloudflare Workflow status and marking them failed when the workflow is terminated, errored, or vanished.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-19

---

**The audit reconciler automatically detects and fails audits left in a "running" state by querying their Cloudflare Workflow instance status and marking them failed when the workflow has terminated, errored, or vanished, using both on-demand checks and a scheduled watchdog.**

The every-app/open-seo platform orchestrates SEO audits through durable Cloudflare Workflows, yet infrastructure hiccups, unexpected terminations, or lost workflow instances can leave audit records stranded mid-execution. The `auditReconciler` service guarantees that no audit remains falsely "running" after its underlying compute has disappeared, ensuring the audit table accurately reflects reality.

## Two-Pronged Detection Strategy

The reconciler employs complementary **lazy** and **watchdog** paths to catch every edge case regardless of traffic patterns.

### On-Demand Reconciliation (Lazy Path)

Whenever the UI or an API consumer polls for an audit’s current status, `AuditService.getStatus` invokes `reconcileRunningAudit`. This *lazy* check ensures that frequently accessed audits are reconciled immediately without waiting for a background sweep, providing instant feedback to users while offloading work from the cron job.

### Scheduled Watchdog (Cron Path)

A background cron job executes `reconcileStaleAudits` at regular intervals. This *watchdog* path queries the database for audits that have been running longer than a configured threshold, ensuring that abandoned audits are eventually failed even if they are never polled again. According to the open-seo source code, this batch process limits each tick to `WATCHDOG_BATCH_LIMIT = 100` rows to maintain consistent performance.

## Step-by-Step Stale Audit Resolution

Inside [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts), the reconciliation flow follows a strict sequence to classify and terminate stranded audits.

### 1. Fetch Stale Candidates

The watchdog computes a cutoff timestamp using `STALE_RUNNING_AFTER_MS` (15 minutes) and calls `getStaleRunningAudits`. This returns up to 100 rows where `status = "running"` and `startedAt` exceeds the cutoff, isolating only truly abandoned jobs from healthy long-running crawls.

### 2. Inspect Workflow Instance State

For each candidate, `reconcileRunningAudit` first verifies the audit has a `workflowInstanceId`. It then queries the workflow subsystem via `env.SITE_AUDIT_WORKFLOW.get(id)` and checks `instance.status()` to determine if the workflow is active, terminated, or errored.

### 3. Detect Terminated or Errored Workflows

If the workflow reports `status === "errored"` or `"terminated"`, the reconciler extracts the error payload. It passes this through `classifyAuditError`—defined in [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts)—to map raw messages to structured error codes like `instance_lost` or `workflow_terminated`.

### 4. Handle Missing Workflow Instances

When the workflow lookup throws (indicating the instance never existed or was purged), the catcher analyzes the error message. If the audit is older than `INSTANCE_LOST_GRACE_MS` (10 minutes), the system treats this as a definitive `instance_lost` error. This grace period prevents premature failure of audits that are initializing but haven’t yet persisted their workflow ID.

### 5. Persist Failure State

Upon confirming a dead workflow, the reconciler calls `AuditRepository.failAudit`—located in [`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)—to atomically update the audit row. This writes the classified error code, descriptive details, and the phase that was executing when the failure occurred.

### 6. Emit Telemetry and Continue

The watchdog logs the transition and sends a `site_audit:complete` event to PostHog via `captureServerEvent`, explicitly flagging the audit as failed by reconciliation. Individual errors during the batch are caught and isolated so that one failing audit does not abort the entire sweep, allowing the remaining candidates to be processed.

## Key Configuration Constants

The reconciler balances responsiveness against false positives through three tunable constants in the source code:

- **`STALE_RUNNING_AFTER_MS`**: Set to 15 minutes, this defines how long an audit must remain in `running` status before it is eligible for the watchdog sweep.
- **`INSTANCE_LOST_GRACE_MS`**: Set to 10 minutes, this buffer allows newly created audits time to register their workflow instance before being marked as lost.
- **`WATCHDOG_BATCH_LIMIT`**: Capped at 100 audits per cron tick, this prevents the watchdog from overwhelming the database or the Cloudflare Workflow API during recovery from large backlogs.

## Implementation Examples

You can invoke reconciliation manually for immediate feedback or register the watchdog as a cron job.

```ts
// Manual on-demand reconciliation (used by the UI)
import { reconcileRunningAudit } from "@/server/features/audit/services/auditReconciler";

async function pollAuditStatus(audit) {
  const errorInfo = await reconcileRunningAudit(audit);
  if (errorInfo) {
    // UI can now display a failure with errorInfo.errorCode / errorDetail
    console.log(`Audit failed with code: ${errorInfo.errorCode}`);
  }
}

```

```ts
// Cron-based watchdog (registered in the server's cron config)
import { reconcileStaleAudits } from "@/server/features/audit/services/auditReconciler";

export const auditWatchdog = {
  schedule: "*/5 * * * *", // every 5 minutes
  handler: async () => {
    await reconcileStaleAudits();
  },
};

```

## Summary

- The **audit reconciler** in every-app/open-seo prevents audits from hanging indefinitely by monitoring their underlying Cloudflare Workflow instances.
- It combines **on-demand checks** during status polls with a **scheduled watchdog** cron job to catch orphaned audits.
- Staleness is determined by the **15-minute** `STALE_RUNNING_AFTER_MS` threshold, while a **10-minute** `INSTANCE_LOST_GRACE_MS` prevents false positives for new audits.
- Dead workflows are classified via `classifyAuditError`, persisted through `AuditRepository.failAudit`, and reported to PostHog for observability.
- The batch processor limits itself to **100 audits per tick** to maintain system stability under load.

## Frequently Asked Questions

### How does the reconciler know if a Cloudflare Workflow has truly failed?

The reconciler queries the workflow instance status via `env.SITE_AUDIT_WORKFLOW.get(id).status()`. If the status returns `"errored"` or `"terminated"`, or if the lookup throws a "not found" error after the 10-minute grace period has elapsed, the audit is marked as failed. This dual check covers both explicit failures and vanished instances.

### What prevents the watchdog from failing audits that are simply slow?

Two safeguards prevent premature failure. First, `STALE_RUNNING_AFTER_MS` (15 minutes) ensures only audits running longer than this threshold are considered for reconciliation. Second, `WATCHDOG_BATCH_LIMIT` restricts each sweep to 100 candidates, allowing the system to prioritize older, more likely stale audits while healthy long-running jobs continue uninterrupted.

### Can I trigger reconciliation manually for a specific audit?

Yes. Import `reconcileRunningAudit` from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) and pass the audit object. This performs the same workflow status check and failure classification as the watchdog, but immediately and for a single record. The function returns `errorInfo` if the audit is dead, or `null` if it remains healthy.

### Where is the error classification logic defined?

The mapping from raw workflow errors to structured error codes lives in [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts) within the `classifyAuditError` function. This centralizes error handling, ensuring that the UI and analytics receive consistent error codes like `instance_lost` or `workflow_terminated` rather than raw stack traces.