# How the Audit Reconciler Pattern Handles Stuck "Running" Audits in OpenSEO

> Discover the audit reconciler pattern in OpenSEO. Learn how this self-healing watchdog resolves stuck 'running' audits by verifying external APIs and preventing resource issues.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-09-04

---

**The audit reconciler pattern in OpenSEO is a self-healing watchdog that periodically scans for audits stuck in the "running" state, verifies their actual workflow status via external APIs, and forcibly transitions orphaned jobs to terminal states to prevent resource exhaustion and UI inconsistency.**

OpenSEO executes SEO audits as asynchronous background workflows—typically Cloudflare Workers or server-side jobs—that report lifecycle states back to a central database. When a workflow crashes, is terminated, or fails to report its completion, the corresponding audit record can remain stranded in the **"running"** state indefinitely. To maintain system health and accuracy, the codebase implements a robust reconciliation mechanism that bridges the gap between persisted state and reality.

## What Is the Audit Reconciler Pattern?

The **audit reconciler pattern** is a reliability design that treats the database as a stale cache of truth, requiring periodic synchronization with the actual workflow execution environment. Rather than assuming a "running" status in the database implies active processing, the reconciler interrogates the underlying workflow infrastructure to verify liveness.

This pattern solves the **orphaned audit problem**: when a Cloudflare Worker dies without updating the database, or when network partitions prevent status callbacks, the system must eventually detect and correct the discrepancy. The reconciler acts as a safety net that guarantees all audits reach a terminal state (`completed` or `failed`) within a bounded time window.

## Watchdog Trigger: Detecting Stale Audits

The reconciliation process initiates from the main server entry point. In [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts), a periodic watchdog schedules the reconciler to run at fixed intervals—typically every few seconds or minutes depending on deployment configuration.

The watchdog specifically targets audits whose `status` field equals `"running"` but whose associated workflow records are missing or have not reported a heartbeat within a defined safety threshold (commonly 30 minutes). This query acts as the first line of defense against state drift.

```typescript
// src/server.ts – watchdog initialization
// Watchdog fires: reconcile audits stuck in "running" whose workflow died

```

## Core Reconciliation Logic in auditReconciler.ts

The primary implementation resides in [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts). The service executes a multi-step validation pipeline:

1. **Query Candidates**: It queries `AuditRepository` for all audits with `status = "running"` where the `updated_at` timestamp exceeds the safety window.

2. **Verify Workflows**: For each candidate, it fetches the real workflow status via the Cloudflare API or internal workflow manager.

3. **State Synchronization**: 
   - If the workflow is **missing**, **failed**, or **completed**, the reconciler updates the audit record to match reality.
   - If the workflow is alive but appear stalled (e.g., exceeded maximum execution time), the reconciler may force-fail the audit to free capacity.

4. **Transactional Updates**: All modifications occur through `AuditRepository.updateAuditStatus`, ensuring atomic updates that prevent race conditions during concurrent reconciliation runs.

```typescript
// Inside auditReconciler.ts
await AuditRepository.updateStatus(auditId, {
  status: 'failed',
  failureReason: 'Workflow disappeared',
});

```

## Stale-Audit Reconciliation and Error Handling

For audits where the workflow infrastructure returns an explicit error or the execution context is irrecoverable, the codebase leverages logic defined in [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts). This **stale-audit reconciler** reads error messages from dead workflows and forces the audit into a terminal state.

This secondary reconciliation path ensures that "running" audits attached to definitively crashed processes do not linger in the database. By extracting failure reasons from the workflow provider and persisting them via the repository layer, the system maintains an accurate audit trail for debugging while clearing operational bottlenecks.

## Integration with Capacity Management

The reconciler operates in concert with [`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts), which enforces concurrency limits on simultaneous audits. Stuck "running" audits artificially consume capacity quota, potentially throttling new legitimate requests.

By transitioning stale audits to `failed` or `completed` states, the reconciler signals the capacity guard to release those slots. This feedback loop ensures the [`audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/audit-capacity.ts) governor can admit new audits without hitting false quota limits imposed by phantom "running" jobs.

## Implementation Example: Manual and Scheduled Reconciliation

Developers can invoke the reconciler manually for operational scripts or testing scenarios:

```typescript
import { runAuditReconciler } from '@/server/features/audit/services/auditReconciler';

async function reconcileNow() {
  await runAuditReconciler(); // scans and updates stuck audits
}
reconcileNow();

```

In production, the reconciler typically runs on a cron schedule. The following pattern executes a reconciliation pass every five minutes:

```typescript
import { schedule } from 'node-cron';
import { runAuditReconciler } from '@/server/features/audit/services/auditReconciler';

schedule('*/5 * * * *', async () => {
  try {
    await runAuditReconciler(); // every 5 minutes
  } catch (e) {
    console.error('Audit reconciler failed:', e);
  }
});

```

## Summary

- The **audit reconciler pattern** acts as a watchdog that corrects database state when background workflows fail to report their completion.

- **Detection** occurs via a scheduled watchdog in [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts) that queries for "running" audits exceeding a safety window (e.g., 30 minutes).

- **Validation** happens in [`auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/auditReconciler.ts), which queries the Cloudflare API or workflow manager to verify actual execution status.

- **Correction** uses transactional updates via `AuditRepository.updateStatus` to move orphaned audits to terminal states (`completed` or `failed`).

- **Capacity protection** works alongside [`audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/audit-capacity.ts) to free concurrency slots consumed by stuck audits, ensuring new requests are not throttled.

- **Idempotency** guarantees that concurrent reconciliation attempts cannot produce inconsistent states or duplicate terminal events.

## Frequently Asked Questions

### How does the reconciler determine if a "running" audit is actually stuck?

The reconciler compares the audit's `updated_at` timestamp against a configured safety window—typically 30 minutes. If the record has remained in the "running" state longer than this threshold without a workflow heartbeat, the reconciler queries the external Cloudflare API or workflow manager to verify whether the underlying process still exists. If the workflow is missing, failed, or completed, the audit is marked accordingly.

### What happens if the Cloudflare API is unavailable during reconciliation?

The reconciler wraps external API calls in try-catch blocks. If the Cloudflare API is unreachable, the reconciler logs the error and skips that specific audit without modifying its state. This prevents false negatives where a healthy but temporarily unreachable workflow might be incorrectly marked as failed. The next scheduled reconciliation attempt will retry the verification.

### Can the reconciler be triggered manually outside the scheduled watchdog?

Yes. The `runAuditReconciler()` function exported from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) can be imported and invoked directly in scripts, administrative tools, or test suites. This is useful for manual recovery operations after infrastructure outages or when debugging specific audit failures without waiting for the next cron interval.

### How does the reconciler prevent race conditions when updating audit status?

All status transitions flow through `AuditRepository.updateAuditStatus`, which performs transactional updates at the database level. This ensures that even if multiple reconciler instances or the original workflow process attempt to modify the same audit record simultaneously, the database maintains consistency and the final state accurately reflects the true workflow outcome.