How Audit Progress Is Tracked in OpenSEO Using Cloudflare KV

OpenSEO tracks live crawl progress by writing a time-limited feed of crawled pages to Cloudflare KV, storing up to 300 recent entries with a 30-minute TTL that automatically expires stale data.

OpenSEO is an open-source SEO auditing tool that leverages Cloudflare's edge infrastructure to provide real-time visibility into site crawls. The application uses Cloudflare KV as a lightweight, ephemeral store to track audit progress without burdening the primary database or persisting temporary crawl data long-term.

KV Namespace Configuration

The Cloudflare KV namespace is bound to the worker environment through the wrangler.jsonc configuration file. The binding named KV exposes the namespace to all workers via env.KV, allowing seamless access to edge storage throughout the application.

According to the source code in wrangler.jsonc【https://github.com/every-app/open-seo/blob/main/wrangler.jsonc#L73-L78】, this binding enables the audit progress tracking functionality across the entire OpenSEO platform.

Key Structure and Data Schema

Each audit receives a dedicated KV key prefixed with audit-progress:, constructed by the key(auditId) helper function in src/server/lib/audit/progress-kv.ts【https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts#L14-L38】.

The stored value adheres to a strict Zod schema defined as crawledUrlEntrySchema:

{
  url: string,          // Crawled page URL
  statusCode: number,   // HTTP response code
  title: string,        // Page title
  crawledAt: number     // Unix timestamp (milliseconds)
}

This schema ensures type safety for all entries pushed to the KV store during the crawl process.

Writing Crawl Progress Updates

During a site audit, the system maintains a capped list of recent crawled pages using the pushCrawledUrls function. This implementation stores a maximum of 300 entries (MAX_ENTRIES = 300) and automatically expires data after 30 minutes (TTL_SECONDS = 30 * 60).

New entries are prepended to the array, ensuring the most recently crawled URLs appear first while older entries roll off the end【https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts#L41-L55】.

The crawl workflow in src/server/workflows/siteAuditWorkflowCrawl.ts invokes this update mechanism after every crawl batch through the persistCrawlProgress function【https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts#L319-L337】:

await AuditProgressKV.pushCrawledUrls(
  auditId,
  crawledBatch.map((page) => ({
    url: page.url,
    statusCode: page.statusCode,
    title: page.title,
    crawledAt: Date.now(),
  })),
);

This batch-oriented approach minimizes KV write operations while keeping the progress feed current.

Reading Progress for the Frontend

The frontend UI polls for updates through AuditService.getCrawlProgress, which delegates to AuditProgressKV.getCrawledUrls【https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts#L10-L18】.

This method returns the JSON array stored in KV (newest first), allowing the dashboard to display the live crawl feed without querying the primary database:

async function getCrawlProgress(auditId: string, projectId: string) {
  const audit = await AuditRepository.getAuditForProject(auditId, projectId);
  if (!audit) throw new AppError("NOT_FOUND");
  return AuditProgressKV.getCrawledUrls(auditId);
}

Cleanup and Lifecycle Management

When an audit completes or is aborted, OpenSEO explicitly removes the KV entry using AuditProgressKV.clear(auditId)【https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts#L70-L73】. This immediate cleanup, combined with the 30-minute TTL, ensures that temporary progress data never lingers in the edge cache longer than necessary.

The complete lifecycle management—automatic expiration plus explicit deletion—keeps storage costs minimal and prevents data leakage between audit sessions.

Complete Implementation Reference

The core KV interaction logic resides in src/server/lib/audit/progress-kv.ts:

import { env } from "cloudflare:workers";
import { z } from "zod";
import { jsonCodec } from "@/shared/json";

const KV_PREFIX = "audit-progress:";
const TTL_SECONDS = 30 * 60;
const MAX_ENTRIES = 300;

const crawledUrlEntrySchema = z.object({
  url: z.string(),
  statusCode: z.number(),
  title: z.string(),
  crawledAt: z.number(),
});
type CrawledUrlEntry = z.infer<typeof crawledUrlEntrySchema>;

function key(auditId: string) {
  return `${KV_PREFIX}${auditId}`;
}

export async function pushCrawledUrls(
  auditId: string,
  nextEntries: CrawledUrlEntry[],
): Promise<void> {
  if (!nextEntries.length) return;
  const k = key(auditId);
  const existing = await env.KV.get(k, "text");
  const base = existing ? JSON.parse(existing) : [];
  const merged = [...nextEntries, ...base].slice(0, MAX_ENTRIES);
  await env.KV.put(k, JSON.stringify(merged), { expirationTtl: TTL_SECONDS });
}

export async function getCrawledUrls(auditId: string) {
  const raw = await env.KV.get(key(auditId), "text");
  return raw ? JSON.parse(raw) : [];
}

export async function clear(auditId: string) {
  await env.KV.delete(key(auditId));
}

Summary

• Cloudflare KV provides the edge-cached storage layer for temporary audit progress data in OpenSEO
• Key naming uses the audit-progress: prefix with a 30-minute TTL to ensure automatic expiration
• Data structure stores up to 300 recent crawled pages with URL, status code, title, and timestamp
• Write operations occur via pushCrawledUrls in the crawl workflow, prepending new batches while maintaining the cap
• Read operations flow through AuditService to the frontend, providing real-time visibility without database load
• Cleanup happens both automatically (TTL) and explicitly (clear function) when audits finish

Frequently Asked Questions

How does OpenSEO prevent KV storage from filling up with stale audit data?

OpenSEO implements a dual-layer expiration strategy. Each KV entry carries a 30-minute TTL (expirationTtl: 1800) that Cloudflare enforces automatically. Additionally, the AuditProgressKV.clear(auditId) function explicitly deletes keys when audits complete or abort, ensuring immediate cleanup regardless of the remaining TTL.

Why does the progress feed store only 300 entries instead of all crawled pages?

The 300-entry cap (MAX_ENTRIES) creates a sliding window of recent activity that balances real-time visibility with storage efficiency. Since the primary purpose is showing users "currently crawling" status rather than maintaining a complete history, this limit prevents KV storage costs from scaling with site size while preserving the most relevant recent activity.

What happens if the crawl workflow fails between KV writes?

Because KV writes occur after each batch via persistCrawlProgress, a workflow failure results in the loss of only the current batch's progress entries—not the entire audit. The previous batches remain stored in KV until the 30-minute TTL expires, allowing the UI to display the last known progress state even if the crawler encounters an error.

Can the audit progress tracking handle high-frequency crawling?

Yes, the implementation uses batch-oriented updates rather than per-page writes. By collecting multiple crawled pages into a single put operation, the system minimizes KV API calls. The prepended-array structure ensures O(1) insertion complexity (via array slicing) regardless of how frequently batches arrive, making it suitable for high-throughput crawling scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}