How the Audit Reconciler Pattern Handles Stuck "Running" Audits in OpenSEO

The audit reconciler pattern in OpenSEO is a self-healing watchdog that periodically scans for audits stuck in the "running" state, verifies their actual workflow status via external APIs, and forcibly transitions orphaned jobs to terminal states to prevent resource exhaustion and UI inconsistency.

OpenSEO executes SEO audits as asynchronous background workflows—typically Cloudflare Workers or server-side jobs—that report lifecycle states back to a central database. When a workflow crashes, is terminated, or fails to report its completion, the corresponding audit record can remain stranded in the "running" state indefinitely. To maintain system health and accuracy, the codebase implements a robust reconciliation mechanism that bridges the gap between persisted state and reality.

What Is the Audit Reconciler Pattern?

The audit reconciler pattern is a reliability design that treats the database as a stale cache of truth, requiring periodic synchronization with the actual workflow execution environment. Rather than assuming a "running" status in the database implies active processing, the reconciler interrogates the underlying workflow infrastructure to verify liveness.

This pattern solves the orphaned audit problem: when a Cloudflare Worker dies without updating the database, or when network partitions prevent status callbacks, the system must eventually detect and correct the discrepancy. The reconciler acts as a safety net that guarantees all audits reach a terminal state (completed or failed) within a bounded time window.

Watchdog Trigger: Detecting Stale Audits

The reconciliation process initiates from the main server entry point. In src/server.ts, a periodic watchdog schedules the reconciler to run at fixed intervals—typically every few seconds or minutes depending on deployment configuration.

The watchdog specifically targets audits whose status field equals "running" but whose associated workflow records are missing or have not reported a heartbeat within a defined safety threshold (commonly 30 minutes). This query acts as the first line of defense against state drift.

// src/server.ts – watchdog initialization
// Watchdog fires: reconcile audits stuck in "running" whose workflow died

Core Reconciliation Logic in auditReconciler.ts

The primary implementation resides in src/server/features/audit/services/auditReconciler.ts. The service executes a multi-step validation pipeline:

1. Query Candidates: It queries AuditRepository for all audits with status = "running" where the updated_at timestamp exceeds the safety window.

2. Verify Workflows: For each candidate, it fetches the real workflow status via the Cloudflare API or internal workflow manager.

3. State Synchronization:

   • If the workflow is missing, failed, or completed, the reconciler updates the audit record to match reality.
   • If the workflow is alive but appear stalled (e.g., exceeded maximum execution time), the reconciler may force-fail the audit to free capacity.

4. Transactional Updates: All modifications occur through AuditRepository.updateAuditStatus, ensuring atomic updates that prevent race conditions during concurrent reconciliation runs.

// Inside auditReconciler.ts
await AuditRepository.updateStatus(auditId, {
  status: 'failed',
  failureReason: 'Workflow disappeared',
});

Stale-Audit Reconciliation and Error Handling

For audits where the workflow infrastructure returns an explicit error or the execution context is irrecoverable, the codebase leverages logic defined in src/server/lib/audit/audit-errors.ts. This stale-audit reconciler reads error messages from dead workflows and forces the audit into a terminal state.

This secondary reconciliation path ensures that "running" audits attached to definitively crashed processes do not linger in the database. By extracting failure reasons from the workflow provider and persisting them via the repository layer, the system maintains an accurate audit trail for debugging while clearing operational bottlenecks.

Integration with Capacity Management

The reconciler operates in concert with src/server/features/audit/services/audit-capacity.ts, which enforces concurrency limits on simultaneous audits. Stuck "running" audits artificially consume capacity quota, potentially throttling new legitimate requests.

By transitioning stale audits to failed or completed states, the reconciler signals the capacity guard to release those slots. This feedback loop ensures the audit-capacity.ts governor can admit new audits without hitting false quota limits imposed by phantom "running" jobs.

Implementation Example: Manual and Scheduled Reconciliation

Developers can invoke the reconciler manually for operational scripts or testing scenarios:

import { runAuditReconciler } from '@/server/features/audit/services/auditReconciler';

async function reconcileNow() {
  await runAuditReconciler(); // scans and updates stuck audits
}
reconcileNow();

In production, the reconciler typically runs on a cron schedule. The following pattern executes a reconciliation pass every five minutes:

import { schedule } from 'node-cron';
import { runAuditReconciler } from '@/server/features/audit/services/auditReconciler';

schedule('*/5 * * * *', async () => {
  try {
    await runAuditReconciler(); // every 5 minutes
  } catch (e) {
    console.error('Audit reconciler failed:', e);
  }
});

Summary

• The audit reconciler pattern acts as a watchdog that corrects database state when background workflows fail to report their completion.

• Detection occurs via a scheduled watchdog in src/server.ts that queries for "running" audits exceeding a safety window (e.g., 30 minutes).

• Validation happens in auditReconciler.ts, which queries the Cloudflare API or workflow manager to verify actual execution status.

• Correction uses transactional updates via AuditRepository.updateStatus to move orphaned audits to terminal states (completed or failed).

• Capacity protection works alongside audit-capacity.ts to free concurrency slots consumed by stuck audits, ensuring new requests are not throttled.

• Idempotency guarantees that concurrent reconciliation attempts cannot produce inconsistent states or duplicate terminal events.

Frequently Asked Questions

How does the reconciler determine if a "running" audit is actually stuck?

The reconciler compares the audit's updated_at timestamp against a configured safety window—typically 30 minutes. If the record has remained in the "running" state longer than this threshold without a workflow heartbeat, the reconciler queries the external Cloudflare API or workflow manager to verify whether the underlying process still exists. If the workflow is missing, failed, or completed, the audit is marked accordingly.

What happens if the Cloudflare API is unavailable during reconciliation?

The reconciler wraps external API calls in try-catch blocks. If the Cloudflare API is unreachable, the reconciler logs the error and skips that specific audit without modifying its state. This prevents false negatives where a healthy but temporarily unreachable workflow might be incorrectly marked as failed. The next scheduled reconciliation attempt will retry the verification.

Can the reconciler be triggered manually outside the scheduled watchdog?

Yes. The runAuditReconciler() function exported from src/server/features/audit/services/auditReconciler.ts can be imported and invoked directly in scripts, administrative tools, or test suites. This is useful for manual recovery operations after infrastructure outages or when debugging specific audit failures without waiting for the next cron interval.

How does the reconciler prevent race conditions when updating audit status?

All status transitions flow through AuditRepository.updateAuditStatus, which performs transactional updates at the database level. This ensures that even if multiple reconciler instances or the original workflow process attempt to modify the same audit record simultaneously, the database maintains consistency and the final state accurately reflects the true workflow outcome.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}