# How OpenSEO Handles Zombie Audits and Reconciliation in Site Audits

> Learn how OpenSEO prevents zombie audits and handles reconciliation by detecting unexpected Cloudflare Workflow terminations and marking failed audits, ensuring data integrity.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-09-02

---

**OpenSEO prevents orphaned audit records by combining on-demand status checks with a periodic watchdog that detects when Cloudflare Workflow instances terminate unexpectedly and immediately marks their associated audits as failed.**

The **every-app/open-seo** repository orchestrates site audits using Cloudflare Workflows, storing each run’s metadata in the `audits` table with a `workflowInstanceId` link. When workflows crash, exceed memory limits, or expire from Cloudflare’s retention window, they can leave behind **zombie audits**—rows stuck in a "running" state despite the underlying compute being gone. To ensure **zombie audit reconciliation** happens reliably without manual intervention, OpenSEO implements a defensive two-layer strategy that gracefully handles both active polling scenarios and abandoned background jobs.

## What Causes Zombie Audits in Cloudflare Workflows

A zombie audit occurs when the `workflowInstanceId` stored in the database points to a Workflow instance that no longer exists or has entered a terminal error state. Common triggers include Out-of-Memory (OOM) kills, deployment resets, or the natural expiration of workflow logs from Cloudflare’s retention policy. Without reconciliation, these orphaned rows consume audit capacity and display perpetually "running" statuses to users.

The system mitigates this by treating the database row and the Workflow instance as a distributed system that must be periodically reconciled, using grace periods to avoid false positives on newly created audits.

## On-Demand Self-Healing During Status Checks

Whenever a client polls for audit progress via `AuditService.getStatus` in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) (lines 44-52), the service performs a lazy health check before returning the current state.

If the audit status is `"running"`, the service immediately invokes `reconcileRunningAudit` from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts). This function queries the Cloudflare Workflow API for the current instance status. If the workflow reports `"errored"`, `"terminated"`, or returns a "not found" error, the audit is instantly marked as **failed** with a classified error code such as `instance_lost`.

This **lazy self-heal** pattern guarantees that any user viewing the audit dashboard will never see a permanently running zombie; the UI will always reflect the true terminal state within milliseconds of the page load.

## Periodic Watchdog for Stale Audits

To catch audits that are never polled—for example, when a user closes their browser tab—OpenSEO runs a cron job that executes `reconcileStaleAudits` every few minutes. Located in [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) (lines 95-130), this watchdog performs a batched sweep over the database.

The query selects audits where the `startedAt` timestamp exceeds the `STALE_RUNNING_AFTER_MS` threshold (15 minutes by default) and limits the batch size to `WATCHDOG_BATCH_LIMIT` (100 rows) to maintain performance. Each stale candidate is passed through the same `reconcileRunningAudit` logic used by the UI path, ensuring consistent error classification and state transition handling.

## Core Reconciliation Logic

The heart of the system resides in the `reconcileRunningAudit` function, which normalizes workflow failures into actionable error metadata:

```typescript
// src/server/features/audit/services/auditReconciler.ts
export async function reconcileRunningAudit(
  audit: RunningAudit,
): Promise<AuditErrorInfo | null> {
  if (!audit.workflowInstanceId) return null;

  let errorInfo: AuditErrorInfo | null = null;
  try {
    const instance = await env.SITE_AUDIT_WORKFLOW.get(audit.workflowInstanceId);
    const status = await instance.status();
    if (status.status === "errored" || status.status === "terminated") {
      errorInfo = status.error
        ? classifyAuditError(
            typeof status.error === "string"
              ? status.error
              : (status.error.message ?? JSON.stringify(status.error)),
          )
        : {
            errorCode: "unknown",
            errorDetail: `Workflow instance ${status.status}`,
          };
    }
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    if (!/not[ _]?found/i.test(message)) return null;
    if (!isOlderThan(audit.startedAt, INSTANCE_LOST_GRACE_MS)) return null;
    errorInfo = {
      errorCode: "instance_lost",
      errorDetail: "Workflow instance not found",
    };
  }

  if (!errorInfo) return null;

  await AuditRepository.failAudit(audit.id, audit.workflowInstanceId, {
    ...errorInfo,
    failedPhase: audit.currentPhase,
  });
  return errorInfo;
}

```

This function distinguishes between transient API errors and true zombie states. It only returns an error object when the workflow is definitively gone or broken, triggering `AuditRepository.failAudit` to persist the failure and capture the `failedPhase` for debugging.

## Grace Periods and Error Classification

To prevent race conditions where a newly created audit is incorrectly flagged as failed while its workflow is still being provisioned, the reconciler enforces two time-based guards:

- **`INSTANCE_LOST_GRACE_MS`** (10 minutes): Ensures audits younger than this threshold are not marked failed when the workflow instance is temporarily unreachable.
- **`STALE_RUNNING_AFTER_MS`** (15 minutes): Defines the minimum age for an audit to be considered a candidate for the watchdog sweep.

Error codes are normalized through `classifyAuditError` from [`src/server/lib/audit/audit-errors.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/audit-errors.ts), converting raw workflow exceptions into consistent identifiers like `instance_lost` or specific infrastructure error codes used in analytics and UI messaging.

## Watchdog Analytics and Monitoring

When the watchdog successfully reconciles a zombie audit, it emits a structured analytics event for observability:

```typescript
// Excerpt from reconcileStaleAudits in auditReconciler.ts
const errorInfo = await reconcileRunningAudit(audit);
if (!errorInfo) continue;

const project = await db.query.projects.findFirst({
  where: eq(projects.id, audit.projectId),
});
if (project) {
  await captureServerEvent({
    distinctId: audit.startedByUserId,
    event: "site_audit:complete",
    organizationId: project.organizationId,
    properties: {
      project_id: audit.projectId,
      status: "failed",
      reconciled_by: "watchdog",
      error_code: errorInfo.errorCode,
      pages_crawled: audit.pagesCrawled,
      pages_total: audit.pagesTotal,
    },
  });
}

```

This integration with PostHog (via `captureServerEvent`) allows operators to track reconciliation rates, identify systemic workflow stability issues, and ensure capacity is properly reclaimed. Individual reconciliation failures are caught and logged without stopping the batch sweep, ensuring the watchdog remains resilient.

## Manual Reconciliation for Administrative Tasks

For support teams or testing scenarios, the same reconciliation logic can be invoked outside the standard UI and cron paths:

```typescript
import { reconcileRunningAudit } from "@/server/features/audit/services/auditReconciler";

async function forceReconcile(auditId: string) {
  const audit = await db.query.audits.findFirst({
    where: eq(audits.id, auditId),
  });
  if (!audit) throw new Error("Audit not found");

  const errorInfo = await reconcileRunningAudit({
    id: audit.id,
    workflowInstanceId: audit.workflowInstanceId,
    startedAt: audit.startedAt,
    currentPhase: audit.currentPhase,
  });

  if (errorInfo) {
    console.log(`Audit ${auditId} was marked failed:`, errorInfo);
  } else {
    console.log(`Audit ${auditId} is still running`);
  }
}

```

This pattern leverages [`AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/AuditRepository.ts) for database updates and ensures administrators can manually clean up edge cases without direct database manipulation.

## Summary

- **Zombie audits** occur when Cloudflare Workflow instances terminate or expire while their database rows remain in a "running" state.
- **Lazy self-healing** checks workflow status on every `getStatus` poll, immediately failing orphaned audits before they reach the user interface.
- **Periodic watchdog** sweeps run every few minutes via `reconcileStaleAudits`, processing batches of stale audits to ensure eventual consistency.
- **Grace periods** (`INSTANCE_LOST_GRACE_MS` and `STALE_RUNNING_AFTER_MS`) prevent false positives on young audits.
- **Error classification** normalizes workflow errors into actionable codes like `instance_lost`, stored alongside the `failedPhase` for debugging.
- **Analytics integration** tracks reconciliation events to monitor infrastructure health and audit capacity utilization.

## Frequently Asked Questions

### What is a zombie audit in OpenSEO?

A zombie audit is a database row in the `audits` table that remains stuck in a "running" status even though its associated Cloudflare Workflow instance has crashed, been terminated, or expired from retention. These orphaned records consume audit capacity and provide misleading status information to users until they are reconciled and marked as failed.

### How long does OpenSEO wait before marking an audit as a zombie?

OpenSEO uses two configurable timeouts. The **periodic watchdog** considers an audit stale after `STALE_RUNNING_AFTER_MS` (15 minutes), while the **on-demand reconciler** waits `INSTANCE_LOST_GRACE_MS` (10 minutes) before concluding that a missing workflow instance is truly lost. These grace periods prevent premature failure marking during normal workflow startup delays.

### Can I manually trigger reconciliation for a specific audit ID?

Yes. You can import `reconcileRunningAudit` from [`src/server/features/audit/services/auditReconciler.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/auditReconciler.ts) and pass it an audit object containing the `workflowInstanceId`, `startedAt`, and `currentPhase`. This is useful for administrative tools or test suites that need to force a status check outside the standard UI polling or cron job schedules.

### What happens to the data when a zombie audit is reconciled?

When reconciliation detects a lost workflow, it calls `AuditRepository.failAudit` to update the row status to "failed", records the specific error code (such as `instance_lost`), and stores the phase that was active when the failure occurred. The system also emits a `site_audit:complete` analytics event with failure metadata, ensuring that partial crawl data and error context are preserved for debugging while freeing up the audit slot for new requests.