How the Site Audit Watchdog Detects and Recovers from OOM or CPU Limit Failures

The site audit watchdog detects OOM or CPU limit failures by periodically reconciling "running" audit rows against their Cloudflare Workflow instance statuses, marking workflows as failed when they report errored or terminated states or when instances disappear after a configurable grace period.

In the every-app/open-seo repository, the watchdog operates as a scheduled cron job that prevents zombie audit states by monitoring workflow health and automatically transitioning failed audits to a terminal state with full diagnostic capture.

Detection Mechanism

The watchdog implements a two-phase detection strategy that combines timestamp-based stale detection with real-time workflow status inspection.

Identifying Stale Running Audits

The reconciliation process begins in src/server/features/audit/services/auditReconciler.ts where reconcileStaleAudits queries for audits that have remained in a "running" state longer than the configurable threshold STALE_RUNNING_AFTER_MS. The function selects audits where the start time predates the cutoff and iterates through each row by calling reconcileRunningAudit (lines 95-105).

// Conceptual query logic from auditReconciler.ts
const staleAudits = await db.audits.findMany({
  where: {
    status: 'running',
    startedAt: { lt: new Date(Date.now() - STALE_RUNNING_AFTER_MS) }
  }
});

for (const audit of staleAudits) {
  await reconcileRunningAudit(audit);
}

Querying Workflow Instance Status

Inside reconcileRunningAudit (lines 54-82), the watchdog fetches the associated workflow instance using the Cloudflare Workers binding env.SITE_AUDIT_WORKFLOW and invokes instance.status(). When the returned status is errored or terminated, the system identifies a platform-initiated kill. These status values indicate the workflow was terminated by the Cloudflare platform due to out-of-memory (OOM) conditions, CPU limit exhaustion, or deployment resets—scenarios where the workflow never reaches its normal "mark-failed" step.

Handling Lost Instances

For cases where the workflow instance cannot be retrieved—either because it never existed or was purged by Cloudflare Workflows retention policies—the watchdog implements a grace period check using INSTANCE_LOST_GRACE_MS (lines 73-81). If the audit age exceeds this window and the instance remains missing, the watchdog classifies this as an instance_lost condition, which commonly follows an OOM or CPU-limit kill that prevented proper cleanup.

Recovery Process

Once the watchdog identifies a failure condition, it executes a structured recovery protocol to ensure state consistency and observability.

Error Classification

The classifyAuditError function (utilized from src/server/lib/audit/audit-errors.ts) processes the raw workflow error payload to extract specific error codes and map them to structured AuditErrorInfo objects. This classification distinguishes between instance_lost scenarios and specific runtime errors, enabling precise failure attribution.

State Persistence and Observability

The watchdog completes recovery by calling AuditRepository.failAudit to persist the failure state, storing the error code, detailed message, and the phase in which the termination occurred (lines 88-124). Simultaneously, the system emits a PostHog server event via site_audit:complete with status: "failed" and reconciled_by: "watchdog", ensuring the UI reflects the terminal state and downstream analytics systems receive notification.

// Simplified recovery flow from auditReconciler.ts
async function reconcileRunningAudit(audit: Audit) {
  const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowId);
  
  if (!instance) {
    if (Date.now() - audit.startedAt.getTime() > INSTANCE_LOST_GRACE_MS) {
      await failAudit(audit, { code: 'instance_lost', reason: 'Workflow instance missing' });
    }
    return;
  }
  
  const status = await instance.status();
  if (status === 'errored' || status === 'terminated') {
    const errorInfo = classifyAuditError(instance.error);
    await AuditRepository.failAudit(audit.id, errorInfo);
    await capturePostHogEvent('site_audit:complete', {
      status: 'failed',
      reconciled_by: 'watchdog',
      audit_id: audit.id
    });
  }
}

Implementation Example

Deploy the watchdog as a Cloudflare Worker cron trigger to ensure continuous monitoring without manual intervention. Register the reconciliation function in src/server.ts and implement the scheduled handler as follows:

// src/server.ts or dedicated cron worker
import { reconcileStaleAudits } from "@/server/features/audit/services/auditReconciler";

export default {
  async scheduled(event: ScheduledEvent, env: Env, ctx: ExecutionContext) {
    try {
      await reconcileStaleAudits(env);
      console.log("Audit watchdog completed sweep");
    } catch (err) {
      // Prevent cron crashes from suppressing future runs
      console.error("Audit watchdog failed:", err);
    }
  },
};

Configure the cron interval (e.g., */15 * * * *) according to your STALE_RUNNING_AFTER_MS threshold to ensure timely detection without excessive API calls to the Cloudflare Workflows platform.

Summary

• reconcileStaleAudits queries for audits running longer than STALE_RUNNING_AFTER_MS and delegates status checks to reconcileRunningAudit.
• Status detection relies on instance.status() returning errored or terminated, which signals OOM, CPU limit, or deployment kills.
• Lost instance detection uses INSTANCE_LOST_GRACE_MS to identify workflows removed by platform retention following hard kills.
• Recovery involves classifyAuditError for categorization, AuditRepository.failAudit for persistence, and PostHog events for observability.
• Implementation requires registering the watchdog as a scheduled Cloudflare Worker cron job to ensure automated, recurring health checks.

Frequently Asked Questions

How does the watchdog distinguish between a running workflow and one that hit a CPU limit?

The watchdog queries the workflow instance status using env.SITE_AUDIT_WORKFLOW.get(workflowId).status() as implemented in auditReconciler.ts lines 54-82. A CPU limit or OOM kill results in a terminated or errored status, whereas healthy running workflows return a running state. If the instance object itself is missing and the audit exceeds INSTANCE_LOST_GRACE_MS, the watchdog infers a platform-initiated termination that prevented proper status recording.

What happens if the Cloudflare Workflow instance is deleted before the watchdog runs?

When instance.status() throws or returns null because the instance was purged by Cloudflare's retention policies, the watchdog checks the audit age against INSTANCE_LOST_GRACE_MS (lines 73-81). Audits older than this threshold are marked as failed with code instance_lost, ensuring they do not remain in a zombie "running" state indefinitely. Audits within the grace period are ignored to allow for propagation delays.

Can the watchdog recover an audit after detecting an OOM failure?

No, the watchdog does not attempt to restart failed audits. According to the source code in auditReconciler.ts lines 88-124, the recovery process strictly transitions the audit to a failed state via AuditRepository.failAudit and captures diagnostic information. Retry logic must be implemented separately by upstream systems monitoring the site_audit:complete PostHog events or database state changes.

Which configuration variables control the watchdog detection timing?

Two primary constants govern detection sensitivity: STALE_RUNNING_AFTER_MS defines how long an audit must be running before the watchdog considers it for reconciliation, and INSTANCE_LOST_GRACE_MS sets the waiting period before a missing workflow instance is classified as permanently lost. These values are typically defined in environment configuration or constants within auditReconciler.ts and should be tuned based on expected audit duration and Cloudflare Workflows retention policies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}