# How the Site Audit Watchdog Detects and Recovers from OOM or CPU Limit Failures

> Learn how the site audit watchdog detects and recovers from OOM or CPU limit failures by reconciling running audits with Cloudflare Workflow instance statuses, ensuring timely issue resolution.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-09-12

---

**The site audit watchdog detects OOM or CPU limit failures by periodically reconciling "running" audit rows against their Cloudflare Workflow instance statuses, marking workflows as failed when they report `errored` or `terminated` states or when instances disappear after a configurable grace period.**

In the `every-app/open-seo` repository, the watchdog operates as a scheduled cron job that prevents zombie audit states by monitoring workflow health and automatically transitioning failed audits to a terminal state with full diagnostic capture.

## Detection Mechanism

The watchdog implements a two-phase detection strategy that combines timestamp-based stale detection with real-time workflow status inspection.

### Identifying Stale Running Audits

The reconciliation process begins in [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) where `reconcileStaleAudits` queries for audits that have remained in a "running" state longer than the configurable threshold `STALE_RUNNING_AFTER_MS`. The function selects audits where the start time predates the cutoff and iterates through each row by calling `reconcileRunningAudit` (lines 95-105).

```typescript
// Conceptual query logic from auditReconciler.ts
const staleAudits = await db.audits.findMany({
  where: {
    status: 'running',
    startedAt: { lt: new Date(Date.now() - STALE_RUNNING_AFTER_MS) }
  }
});

for (const audit of staleAudits) {
  await reconcileRunningAudit(audit);
}

```

### Querying Workflow Instance Status

Inside `reconcileRunningAudit` (lines 54-82), the watchdog fetches the associated workflow instance using the Cloudflare Workers binding `env.SITE_AUDIT_WORKFLOW` and invokes `instance.status()`. When the returned status is **`errored`** or **`terminated`**, the system identifies a platform-initiated kill. These status values indicate the workflow was terminated by the Cloudflare platform due to **out-of-memory (OOM) conditions**, **CPU limit exhaustion**, or deployment resets—scenarios where the workflow never reaches its normal "mark-failed" step.

### Handling Lost Instances

For cases where the workflow instance cannot be retrieved—either because it never existed or was purged by Cloudflare Workflows retention policies—the watchdog implements a grace period check using `INSTANCE_LOST_GRACE_MS` (lines 73-81). If the audit age exceeds this window and the instance remains missing, the watchdog classifies this as an `instance_lost` condition, which commonly follows an OOM or CPU-limit kill that prevented proper cleanup.

## Recovery Process

Once the watchdog identifies a failure condition, it executes a structured recovery protocol to ensure state consistency and observability.

### Error Classification

The `classifyAuditError` function (utilized from [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts)) processes the raw workflow error payload to extract specific error codes and map them to structured `AuditErrorInfo` objects. This classification distinguishes between `instance_lost` scenarios and specific runtime errors, enabling precise failure attribution.

### State Persistence and Observability

The watchdog completes recovery by calling `AuditRepository.failAudit` to persist the failure state, storing the error code, detailed message, and the phase in which the termination occurred (lines 88-124). Simultaneously, the system emits a PostHog server event via `site_audit:complete` with `status: "failed"` and `reconciled_by: "watchdog"`, ensuring the UI reflects the terminal state and downstream analytics systems receive notification.

```typescript
// Simplified recovery flow from auditReconciler.ts
async function reconcileRunningAudit(audit: Audit) {
  const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowId);
  
  if (!instance) {
    if (Date.now() - audit.startedAt.getTime() > INSTANCE_LOST_GRACE_MS) {
      await failAudit(audit, { code: 'instance_lost', reason: 'Workflow instance missing' });
    }
    return;
  }
  
  const status = await instance.status();
  if (status === 'errored' || status === 'terminated') {
    const errorInfo = classifyAuditError(instance.error);
    await AuditRepository.failAudit(audit.id, errorInfo);
    await capturePostHogEvent('site_audit:complete', {
      status: 'failed',
      reconciled_by: 'watchdog',
      audit_id: audit.id
    });
  }
}

```

## Implementation Example

Deploy the watchdog as a Cloudflare Worker cron trigger to ensure continuous monitoring without manual intervention. Register the reconciliation function in [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts) and implement the scheduled handler as follows:

```typescript
// src/server.ts or dedicated cron worker
import { reconcileStaleAudits } from "@/server/features/audit/services/auditReconciler";

export default {
  async scheduled(event: ScheduledEvent, env: Env, ctx: ExecutionContext) {
    try {
      await reconcileStaleAudits(env);
      console.log("Audit watchdog completed sweep");
    } catch (err) {
      // Prevent cron crashes from suppressing future runs
      console.error("Audit watchdog failed:", err);
    }
  },
};

```

Configure the cron interval (e.g., `*/15 * * * *`) according to your `STALE_RUNNING_AFTER_MS` threshold to ensure timely detection without excessive API calls to the Cloudflare Workflows platform.

## Summary

- **`reconcileStaleAudits`** queries for audits running longer than `STALE_RUNNING_AFTER_MS` and delegates status checks to `reconcileRunningAudit`.
- **Status detection** relies on `instance.status()` returning `errored` or `terminated`, which signals OOM, CPU limit, or deployment kills.
- **Lost instance detection** uses `INSTANCE_LOST_GRACE_MS` to identify workflows removed by platform retention following hard kills.
- **Recovery** involves `classifyAuditError` for categorization, `AuditRepository.failAudit` for persistence, and PostHog events for observability.
- **Implementation** requires registering the watchdog as a scheduled Cloudflare Worker cron job to ensure automated, recurring health checks.

## Frequently Asked Questions

### How does the watchdog distinguish between a running workflow and one that hit a CPU limit?

The watchdog queries the workflow instance status using `env.SITE_AUDIT_WORKFLOW.get(workflowId).status()` as implemented in [`auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/auditReconciler.ts) lines 54-82. A **CPU limit** or **OOM kill** results in a `terminated` or `errored` status, whereas healthy running workflows return a `running` state. If the instance object itself is missing and the audit exceeds `INSTANCE_LOST_GRACE_MS`, the watchdog infers a platform-initiated termination that prevented proper status recording.

### What happens if the Cloudflare Workflow instance is deleted before the watchdog runs?

When `instance.status()` throws or returns null because the instance was purged by Cloudflare's retention policies, the watchdog checks the audit age against `INSTANCE_LOST_GRACE_MS` (lines 73-81). Audits older than this threshold are marked as failed with code `instance_lost`, ensuring they do not remain in a zombie "running" state indefinitely. Audits within the grace period are ignored to allow for propagation delays.

### Can the watchdog recover an audit after detecting an OOM failure?

No, the watchdog does not attempt to restart failed audits. According to the source code in [`auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/auditReconciler.ts) lines 88-124, the recovery process strictly transitions the audit to a **failed** state via `AuditRepository.failAudit` and captures diagnostic information. Retry logic must be implemented separately by upstream systems monitoring the `site_audit:complete` PostHog events or database state changes.

### Which configuration variables control the watchdog detection timing?

Two primary constants govern detection sensitivity: **`STALE_RUNNING_AFTER_MS`** defines how long an audit must be running before the watchdog considers it for reconciliation, and **`INSTANCE_LOST_GRACE_MS`** sets the waiting period before a missing workflow instance is classified as permanently lost. These values are typically defined in environment configuration or constants within [`auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/auditReconciler.ts) and should be tuned based on expected audit duration and Cloudflare Workflows retention policies.