How OpenSEO’s Watchdog Cron Reconciles Stale Site Audits

OpenSEO reconciles stale site audits by scanning for running audits older than 15 minutes, checking their Cloudflare Workflow status, and marking them as failed if the workflow errored, terminated, or disappeared after a grace period.

The every-app/open-seo repository implements a resilient background process to prevent zombie audit records from persisting indefinitely. The watchdog cron automatically reconciles stale site audits that remain stuck in a running state long after their Cloudflare Workflow instances have failed or vanished. This ensures database consistency and accurate analytics tracking even when underlying infrastructure errors occur.

Identifying Stale Audits in the Database

In src/server/features/audit/services/auditReconciler.ts, the reconcileStaleAudits function serves as the entry point for the scheduled cron job. It calculates a cutoff timestamp using the staleness threshold defined by STALE_RUNNING_AFTER_MS (set to 15 minutes), then queries for audits with a running status whose startedAt timestamp precedes this cutoff.

Cross-Database Timestamp Normalization

The underlying query (getStaleRunningAudits) handles timestamp formatting differences between PostgreSQL (ISO format) and Cloudflare D1 (space-separated format). Results are ordered by startedAt to ensure deterministic processing order, preventing race conditions during batch reconciliation.

Reconciling Individual Audit Records

For each candidate audit, the system calls reconcileRunningAudit to inspect the associated Cloudflare Workflow instance. The function retrieves the instance using env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId) and evaluates its current execution status.

Workflow Status Verification

If the workflow status returns errored or terminated, the system extracts error details and classifies them via classifyAuditError from src/server/lib/audit/audit-errors.ts. The audit row is then updated to failed status through AuditRepository.failAudit in src/server/features/audit/repositories/AuditRepository.ts.

Handling Missing Workflow Instances

When the workflow instance cannot be found—either because it was never created or was purged by Cloudflare’s retention policies—the system applies a grace period defined by INSTANCE_LOST_GRACE_MS (10 minutes). Audits older than this window are marked as failed with the instance_lost error code, while newer records are temporarily preserved in case the instance is still initializing.

Batch Processing and Observability

The watchdog processes up to WATCHDOG_BATCH_LIMIT (100) audits per cron tick, preventing memory exhaustion during bulk reconciliation failures. After marking an audit as failed, the system emits a server-side event via captureServerEvent in src/server/lib/posthog.ts, enabling analytics platforms like PostHog to track infrastructure reliability and error patterns.

// 1️⃣  Entry point – called by the scheduled cron job
export async function reconcileStaleAudits() {
  const cutoff = new Date(Date.now() - STALE_RUNNING_AFTER_MS);
  const stale = await getStaleRunningAudits(cutoff, WATCHDOG_BATCH_LIMIT);

  for (const audit of stale) {
    try {
      const errorInfo = await reconcileRunningAudit(audit);
      if (!errorInfo) continue;                       // audit is still alive
      console.log(
        `Audit watchdog: marked ${audit.id} failed (${errorInfo.errorCode})`,
      );
      // analytics …
    } catch (e) {
      console.error(`Audit watchdog: failed to reconcile ${audit.id}:`, e);
    }
  }
}

// 2️⃣  Core reconciliation logic
export async function reconcileRunningAudit(audit: RunningAudit) {
  if (!audit.workflowInstanceId) return null;

  let errorInfo: AuditErrorInfo | null = null;
  try {
    const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId);
    const status = await instance.status();

    if (status.status === "errored" || status.status === "terminated") {
      errorInfo = status.error
        ? classifyAuditError(
            typeof status.error === "string"
              ? status.error
              : status.error.message ?? JSON.stringify(status.error),
          )
        : { errorCode: "unknown", errorDetail: `Workflow ${status.status}` };
    }
  } catch (err) {
    // Instance missing → treat as lost after grace period
    const msg = err instanceof Error ? err.message : String(err);
    if (!/not[ _]?found/i.test(msg)) return null;
    if (!isOlderThan(audit.startedAt, INSTANCE_LOST_GRACE_MS)) return null;
    errorInfo = { errorCode: "instance_lost", errorDetail: "Workflow instance not found" };
  }

  if (!errorInfo) return null;
  await AuditRepository.failAudit(audit.id, audit.workflowInstanceId, {
    ...errorInfo,
    failedPhase: audit.currentPhase,
  });
  return errorInfo;
}

Summary

• Stale detection relies on the STALE_RUNNING_AFTER_MS constant (15 minutes) to identify audits stuck in running status.
• Workflow verification queries Cloudflare Workflow instances to distinguish between healthy, errored, and terminated executions.
• Grace period handling allows 10 minutes (INSTANCE_LOST_GRACE_MS) for missing instances to appear before marking audits as instance_lost.
• Batch limits cap processing at 100 audits per cron tick via WATCHDOG_BATCH_LIMIT, ensuring predictable resource usage.
• Analytics integration sends failure events to PostHog through captureServerEvent for observability.

Frequently Asked Questions

How does OpenSEO determine if a site audit is stale?

OpenSEO calculates a cutoff time by subtracting STALE_RUNNING_AFTER_MS (15 minutes) from the current timestamp. Any audit with a running status and a startedAt value older than this cutoff is considered stale and queued for reconciliation.

What happens if the Cloudflare Workflow instance is missing?

If the workflow instance cannot be retrieved, the system checks whether the audit's startedAt timestamp exceeds the INSTANCE_LOST_GRACE_MS grace period (10 minutes). If it does, the audit is marked as failed with the instance_lost error code; otherwise, it remains in the running state pending the next cron tick.

How many stale audits can the watchdog process at once?

The watchdog respects the WATCHDOG_BATCH_LIMIT constant, which is set to 100 audits per cron execution. This prevents resource exhaustion while ensuring the system gradually clears zombie records even under heavy failure loads.

Where is the audit reconciliation logic implemented?

The core logic resides in src/server/features/audit/services/auditReconciler.ts, which defines both reconcileStaleAudits for batch processing and reconcileRunningAudit for individual record handling. Supporting utilities include classifyAuditError in src/server/lib/audit/audit-errors.ts and the repository methods in src/server/features/audit/repositories/AuditRepository.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}