# How OpenSEO’s Watchdog Cron Reconciles Stale Site Audits

> Learn how OpenSEO's watchdog cron reconciles stale site audits. Discover how it identifies and resolves outdated audit processes automatically to ensure accuracy.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-09-04

---

**OpenSEO reconciles stale site audits by scanning for running audits older than 15 minutes, checking their Cloudflare Workflow status, and marking them as failed if the workflow errored, terminated, or disappeared after a grace period.**

The **every-app/open-seo** repository implements a resilient background process to prevent zombie audit records from persisting indefinitely. The **watchdog cron** automatically reconciles stale site audits that remain stuck in a `running` state long after their Cloudflare Workflow instances have failed or vanished. This ensures database consistency and accurate analytics tracking even when underlying infrastructure errors occur.

## Identifying Stale Audits in the Database

In [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts), the `reconcileStaleAudits` function serves as the entry point for the scheduled cron job. It calculates a cutoff timestamp using the **staleness threshold** defined by `STALE_RUNNING_AFTER_MS` (set to 15 minutes), then queries for audits with a `running` status whose `startedAt` timestamp precedes this cutoff.

### Cross-Database Timestamp Normalization

The underlying query (`getStaleRunningAudits`) handles timestamp formatting differences between **PostgreSQL** (ISO format) and **Cloudflare D1** (space-separated format). Results are ordered by `startedAt` to ensure deterministic processing order, preventing race conditions during batch reconciliation.

## Reconciling Individual Audit Records

For each candidate audit, the system calls `reconcileRunningAudit` to inspect the associated Cloudflare Workflow instance. The function retrieves the instance using `env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId)` and evaluates its current execution status.

### Workflow Status Verification

If the workflow status returns `errored` or `terminated`, the system extracts error details and classifies them via `classifyAuditError` from [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts). The audit row is then updated to **failed** status through `AuditRepository.failAudit` in [`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts).

### Handling Missing Workflow Instances

When the workflow instance cannot be found—either because it was never created or was purged by Cloudflare’s retention policies—the system applies a **grace period** defined by `INSTANCE_LOST_GRACE_MS` (10 minutes). Audits older than this window are marked as failed with the `instance_lost` error code, while newer records are temporarily preserved in case the instance is still initializing.

## Batch Processing and Observability

The watchdog processes up to `WATCHDOG_BATCH_LIMIT` (100) audits per cron tick, preventing memory exhaustion during bulk reconciliation failures. After marking an audit as failed, the system emits a server-side event via `captureServerEvent` in [`src/server/lib/posthog.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/posthog.ts), enabling analytics platforms like PostHog to track infrastructure reliability and error patterns.

```typescript
// 1️⃣  Entry point – called by the scheduled cron job
export async function reconcileStaleAudits() {
  const cutoff = new Date(Date.now() - STALE_RUNNING_AFTER_MS);
  const stale = await getStaleRunningAudits(cutoff, WATCHDOG_BATCH_LIMIT);

  for (const audit of stale) {
    try {
      const errorInfo = await reconcileRunningAudit(audit);
      if (!errorInfo) continue;                       // audit is still alive
      console.log(
        `Audit watchdog: marked ${audit.id} failed (${errorInfo.errorCode})`,
      );
      // analytics …
    } catch (e) {
      console.error(`Audit watchdog: failed to reconcile ${audit.id}:`, e);
    }
  }
}

```

```typescript
// 2️⃣  Core reconciliation logic
export async function reconcileRunningAudit(audit: RunningAudit) {
  if (!audit.workflowInstanceId) return null;

  let errorInfo: AuditErrorInfo | null = null;
  try {
    const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId);
    const status = await instance.status();

    if (status.status === "errored" || status.status === "terminated") {
      errorInfo = status.error
        ? classifyAuditError(
            typeof status.error === "string"
              ? status.error
              : status.error.message ?? JSON.stringify(status.error),
          )
        : { errorCode: "unknown", errorDetail: `Workflow ${status.status}` };
    }
  } catch (err) {
    // Instance missing → treat as lost after grace period
    const msg = err instanceof Error ? err.message : String(err);
    if (!/not[ _]?found/i.test(msg)) return null;
    if (!isOlderThan(audit.startedAt, INSTANCE_LOST_GRACE_MS)) return null;
    errorInfo = { errorCode: "instance_lost", errorDetail: "Workflow instance not found" };
  }

  if (!errorInfo) return null;
  await AuditRepository.failAudit(audit.id, audit.workflowInstanceId, {
    ...errorInfo,
    failedPhase: audit.currentPhase,
  });
  return errorInfo;
}

```

## Summary

- **Stale detection** relies on the `STALE_RUNNING_AFTER_MS` constant (15 minutes) to identify audits stuck in `running` status.
- **Workflow verification** queries Cloudflare Workflow instances to distinguish between healthy, errored, and terminated executions.
- **Grace period handling** allows 10 minutes (`INSTANCE_LOST_GRACE_MS`) for missing instances to appear before marking audits as `instance_lost`.
- **Batch limits** cap processing at 100 audits per cron tick via `WATCHDOG_BATCH_LIMIT`, ensuring predictable resource usage.
- **Analytics integration** sends failure events to PostHog through `captureServerEvent` for observability.

## Frequently Asked Questions

### How does OpenSEO determine if a site audit is stale?

OpenSEO calculates a cutoff time by subtracting `STALE_RUNNING_AFTER_MS` (15 minutes) from the current timestamp. Any audit with a `running` status and a `startedAt` value older than this cutoff is considered stale and queued for reconciliation.

### What happens if the Cloudflare Workflow instance is missing?

If the workflow instance cannot be retrieved, the system checks whether the audit's `startedAt` timestamp exceeds the `INSTANCE_LOST_GRACE_MS` grace period (10 minutes). If it does, the audit is marked as failed with the `instance_lost` error code; otherwise, it remains in the running state pending the next cron tick.

### How many stale audits can the watchdog process at once?

The watchdog respects the `WATCHDOG_BATCH_LIMIT` constant, which is set to 100 audits per cron execution. This prevents resource exhaustion while ensuring the system gradually clears zombie records even under heavy failure loads.

### Where is the audit reconciliation logic implemented?

The core logic resides in [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts), which defines both `reconcileStaleAudits` for batch processing and `reconcileRunningAudit` for individual record handling. Supporting utilities include `classifyAuditError` in [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts) and the repository methods in [`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts).