# How the Open SEO Audit Progress System Functions with Cloudflare KV Storage

> Discover how the Open SEO audit progress system uses Cloudflare KV storage to stream live crawl updates to your UI efficiently. Learn about key management and caching strategies.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-07-31

---

**The audit progress system streams live crawl updates to the UI by storing temporary JSON arrays in Cloudflare KV under keys like `audit-progress:<auditId>`, capping entries at 300 with a 30-minute TTL for efficient edge caching.**

The audit progress system in every-app/open-seo provides real-time visibility into website crawls without overloading the primary database. By leveraging Cloudflare KV as a transient data store, the system writes incremental crawl results to the edge, allowing the frontend to poll for updates while the background workflow processes thousands of pages. This architecture decouples live progress tracking from persistent audit storage, ensuring low-latency updates during long-running crawl operations.

## KV Storage Schema and Key Structure

Each running audit receives a dedicated KV key following the format `audit-progress:<auditId>`. The value stored is a JSON-encoded array of objects describing crawled pages, with each entry containing the URL, HTTP status code, page title, and timestamp.

In [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts), the schema is enforced using Zod via `crawledUrlEntrySchema` (lines 18‑24). This ensures type safety when serializing and deserializing data between the crawl worker and the UI.

## Writing Incremental Progress Updates

As the crawl worker processes batches of pages, it persists progress by calling `AuditProgressKV.pushCrawledUrls(auditId, entries)`. This method, implemented in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) (lines 44‑58), performs three critical operations:

1. **Reads** the existing array from KV using `env.KV.get`
2. **Prepends** the new batch of entries and enforces a maximum length of 300 entries (`MAX_ENTRIES`)
3. **Writes** the merged array back with a 30-minute TTL (`TTL_SECONDS`)

The write operation is invoked from the crawl workflow in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) (lines 28‑36) after each batch completion. This allows the UI to poll increasingly recent results without flooding the relational database with temporary state updates.

```typescript
import { AuditProgressKV } from "@/server/lib/audit/progress-kv";

/* Push a batch of crawled pages from inside the crawl workflow */
await AuditProgressKV.pushCrawledUrls(auditId, [
  {
    url: "https://example.com",
    statusCode: 200,
    title: "Example Domain",
    crawledAt: Date.now(),
  },
]);

```

## Reading Live Progress for the Frontend

The frontend retrieves progress through `AuditService.getProgress`, which delegates to `AuditProgressKV.getCrawledUrls(auditId)` in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) (lines 60‑66). This method fetches the KV value and parses it into the typed array, returning the list ordered newest-first so the UI displays the most recent crawl activity at the top.

```typescript
/* Retrieve the live progress for the UI */
const liveEntries = await AuditProgressKV.getCrawledUrls(auditId);
// liveEntries is an array of CrawledUrlEntry, newest first

```

## Cleanup and TTL Management

When an audit finishes successfully, the system explicitly deletes the KV key to avoid stale data and free the TTL slot. The `AuditProgressKV.clear(auditId)` method calls `env.KV.delete` as implemented in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) (lines 70‑74). This is triggered from the audit finalization step in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts).

The 30-minute TTL serves as a safety net: if a client never explicitly clears the progress (for example, due to a crashed workflow), KV automatically removes the data after expiration.

```typescript
/* Clear the temporary progress after the audit finishes */
await AuditProgressKV.clear(auditId);

```

## Design Rationale: Why Cloudflare KV?

The audit progress system uses Cloudflare KV instead of the primary database for three specific architectural advantages:

- **Low latency** – KV reads execute at the edge with minimal latency, providing the UI with a fast "live feed" of crawl activity
- **Ephemeral nature** – The data is only needed while the crawl runs; the short TTL automatically removes stale entries without manual intervention
- **Scalability** – Capping the list at 300 entries keeps payloads small (under approximately 100KB), staying well within KV limits while providing sufficient context for progress indicators

## Summary

- The audit progress system uses **Cloudflare KV** as a temporary channel to stream live crawl updates to the UI
- Each audit writes to a dedicated key (`audit-progress:<auditId>`) in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) using `pushCrawledUrls`
- Entries are capped at **300 items** with a **30-minute TTL** to manage storage costs and ensure automatic cleanup
- The crawl workflow in [`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts) writes batches after each processing step, while [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) clears the key upon completion
- Frontend polling reads newest-first arrays via `getCrawledUrls`, enabling real-time progress bars without database load

## Frequently Asked Questions

### What data format does the audit progress system store in KV?

The system stores a JSON-encoded array of objects validated by `crawledUrlEntrySchema`. Each object contains the crawled URL, HTTP status code, page title, and a timestamp (`crawledAt`). This schema is defined in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) and ensures type-safe serialization between the worker and frontend.

### How does the system prevent KV storage from growing indefinitely?

The `pushCrawledUrls` method enforces a hard limit of 300 entries (`MAX_ENTRIES`) by truncating the array after prepending new results. Additionally, each write sets a 30-minute TTL (`TTL_SECONDS`), ensuring automatic expiration even if the cleanup routine fails to execute.

### Why use Cloudflare KV instead of the main database for progress updates?

Cloudflare KV provides edge-cached, low-latency reads that enable real-time UI updates without overwhelming the relational database with high-frequency write operations. The ephemeral nature of crawl progress—needed only during the active crawl window—makes KV's TTL and key-value structure more appropriate than persistent database rows.

### How does the UI retrieve the latest crawl entries first?

The `getCrawledUrls` method in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) returns the array in the order it is stored: newest-first. This is achieved because `pushCrawledUrls` prepends new batches to the beginning of the array before saving back to KV, ensuring the most recent crawled pages appear at index zero.