How the Open SEO Audit Progress System Functions with Cloudflare KV Storage

The audit progress system streams live crawl updates to the UI by storing temporary JSON arrays in Cloudflare KV under keys like audit-progress:<auditId>, capping entries at 300 with a 30-minute TTL for efficient edge caching.

The audit progress system in every-app/open-seo provides real-time visibility into website crawls without overloading the primary database. By leveraging Cloudflare KV as a transient data store, the system writes incremental crawl results to the edge, allowing the frontend to poll for updates while the background workflow processes thousands of pages. This architecture decouples live progress tracking from persistent audit storage, ensuring low-latency updates during long-running crawl operations.

KV Storage Schema and Key Structure

Each running audit receives a dedicated KV key following the format audit-progress:<auditId>. The value stored is a JSON-encoded array of objects describing crawled pages, with each entry containing the URL, HTTP status code, page title, and timestamp.

In src/server/lib/audit/progress-kv.ts, the schema is enforced using Zod via crawledUrlEntrySchema (lines 18‑24). This ensures type safety when serializing and deserializing data between the crawl worker and the UI.

Writing Incremental Progress Updates

As the crawl worker processes batches of pages, it persists progress by calling AuditProgressKV.pushCrawledUrls(auditId, entries). This method, implemented in src/server/lib/audit/progress-kv.ts (lines 44‑58), performs three critical operations:

1. Reads the existing array from KV using env.KV.get
2. Prepends the new batch of entries and enforces a maximum length of 300 entries (MAX_ENTRIES)
3. Writes the merged array back with a 30-minute TTL (TTL_SECONDS)

The write operation is invoked from the crawl workflow in src/server/workflows/siteAuditWorkflowCrawl.ts (lines 28‑36) after each batch completion. This allows the UI to poll increasingly recent results without flooding the relational database with temporary state updates.

import { AuditProgressKV } from "@/server/lib/audit/progress-kv";

/* Push a batch of crawled pages from inside the crawl workflow */
await AuditProgressKV.pushCrawledUrls(auditId, [
  {
    url: "https://example.com",
    statusCode: 200,
    title: "Example Domain",
    crawledAt: Date.now(),
  },
]);

Reading Live Progress for the Frontend

The frontend retrieves progress through AuditService.getProgress, which delegates to AuditProgressKV.getCrawledUrls(auditId) in src/server/lib/audit/progress-kv.ts (lines 60‑66). This method fetches the KV value and parses it into the typed array, returning the list ordered newest-first so the UI displays the most recent crawl activity at the top.

/* Retrieve the live progress for the UI */
const liveEntries = await AuditProgressKV.getCrawledUrls(auditId);
// liveEntries is an array of CrawledUrlEntry, newest first

Cleanup and TTL Management

When an audit finishes successfully, the system explicitly deletes the KV key to avoid stale data and free the TTL slot. The AuditProgressKV.clear(auditId) method calls env.KV.delete as implemented in src/server/lib/audit/progress-kv.ts (lines 70‑74). This is triggered from the audit finalization step in src/server/workflows/siteAuditWorkflowPhases.ts.

The 30-minute TTL serves as a safety net: if a client never explicitly clears the progress (for example, due to a crashed workflow), KV automatically removes the data after expiration.

/* Clear the temporary progress after the audit finishes */
await AuditProgressKV.clear(auditId);

Design Rationale: Why Cloudflare KV?

The audit progress system uses Cloudflare KV instead of the primary database for three specific architectural advantages:

• Low latency – KV reads execute at the edge with minimal latency, providing the UI with a fast "live feed" of crawl activity
• Ephemeral nature – The data is only needed while the crawl runs; the short TTL automatically removes stale entries without manual intervention
• Scalability – Capping the list at 300 entries keeps payloads small (under approximately 100KB), staying well within KV limits while providing sufficient context for progress indicators

Summary

• The audit progress system uses Cloudflare KV as a temporary channel to stream live crawl updates to the UI
• Each audit writes to a dedicated key (audit-progress:<auditId>) in src/server/lib/audit/progress-kv.ts using pushCrawledUrls
• Entries are capped at 300 items with a 30-minute TTL to manage storage costs and ensure automatic cleanup
• The crawl workflow in siteAuditWorkflowCrawl.ts writes batches after each processing step, while siteAuditWorkflowPhases.ts clears the key upon completion
• Frontend polling reads newest-first arrays via getCrawledUrls, enabling real-time progress bars without database load

Frequently Asked Questions

What data format does the audit progress system store in KV?

The system stores a JSON-encoded array of objects validated by crawledUrlEntrySchema. Each object contains the crawled URL, HTTP status code, page title, and a timestamp (crawledAt). This schema is defined in src/server/lib/audit/progress-kv.ts and ensures type-safe serialization between the worker and frontend.

How does the system prevent KV storage from growing indefinitely?

The pushCrawledUrls method enforces a hard limit of 300 entries (MAX_ENTRIES) by truncating the array after prepending new results. Additionally, each write sets a 30-minute TTL (TTL_SECONDS), ensuring automatic expiration even if the cleanup routine fails to execute.

Why use Cloudflare KV instead of the main database for progress updates?

Cloudflare KV provides edge-cached, low-latency reads that enable real-time UI updates without overwhelming the relational database with high-frequency write operations. The ephemeral nature of crawl progress—needed only during the active crawl window—makes KV's TTL and key-value structure more appropriate than persistent database rows.

How does the UI retrieve the latest crawl entries first?

The getCrawledUrls method in src/server/lib/audit/progress-kv.ts returns the array in the order it is stored: newest-first. This is achieved because pushCrawledUrls prepends new batches to the beginning of the array before saving back to KV, ensuring the most recent crawled pages appear at index zero.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}