# How OpenSEO Manages Crawl Frontiers for Site Audits in Cloudflare Durable Objects

> Discover how OpenSEO manages crawl frontiers for site audits using Cloudflare Durable Objects. Learn about the AuditScratchpad and its SQLite database for efficient URL tracking during large-scale crawls.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-08-20

---

**OpenSEO manages crawl frontiers for site audits using a Cloudflare Durable Object called `AuditScratchpad` that maintains a lightweight SQLite database to queue, lease, and track the state of every URL during large-scale crawls.**

Crawl frontiers are the backbone of any site audit engine—they determine which pages to fetch, in what order, and how to recover from failures. In OpenSEO, this complexity is encapsulated in a single Durable Object that persists state across workflow steps, enabling resumable, fault-tolerant crawls without bloating message payloads.

## What Is a Crawl Frontier in OpenSEO?

A **crawl frontier** is the set of URLs discovered but not yet processed. In [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts), OpenSEO implements this as three SQLite tables:

- **`frontier`** — tracks every URL's depth, source (`link` or `sitemap`), and current `state` (`pending`, `leased`, `crawled`)
- **`page_mirror`** — stores crawled page content and metadata
- **`links`** — records discovered internal links between pages

The frontier table schema at lines 90-96 ensures each URL is unique and queryable by state:

```typescript
CREATE TABLE IF NOT EXISTS frontier (
  url TEXT PRIMARY KEY,
  depth INTEGER,
  source TEXT,        -- 'link' or 'sitemap'
  state TEXT,         -- 'pending', 'leased', 'crawled'
  chunk_no INTEGER    -- which batch leased this URL
)

```

## Seeding the Frontier with Start URLs and Sitemaps

Every audit begins with **seeding**. The `seedStart()` and `seedSitemapUrls()` methods (lines 25-41) populate the frontier with initial URLs:

```typescript
// Seed from the manually provided start URL
await scratchpad.seedStart('https://example.com/');

// Seed from parsed sitemap.xml
await scratchpad.seedSitemapUrls([
  'https://example.com/page1',
  'https://example.com/page2'
]);

```

Sitemap URLs are **upserted** with `in_sitemap = 1`, allowing the frontier to distinguish between link-discovered and sitemap-only entries. This distinction drives crawl prioritization.

## Leasing URL Chunks with Priority Ordering

The `claimChunk()` method (lines 50-75) implements the core frontier traversal logic. Each workflow step requests a batch of URLs by chunk number:

1. **Idempotent retry check** — if this chunk already leased URLs, return them again
2. **Priority query** — prefer `source='link'` over `source='sitemap'` to prioritize editorially linked pages
3. **FIFO within class** — maintain crawl order for fairness
4. **State transition** — `pending` → `leased` with chunk number assignment

```typescript
const chunkNo = 1;
const urlsToCrawl = await scratchpad.claimChunk(chunkNo, 100);
// Returns: [{url: 'https://example.com/', depth: 0, source: 'link'}, ...]

```

This design ensures **exactly-once processing per chunk attempt**—critical for Cloudflare Workflows where steps may retry.

## Recording Batch Results and Discovering New URLs

After crawling, `recordBatch()` (lines 85-99) persists results and expands the frontier:

```typescript
await scratchpad.recordBatch({
  crawledUrls: urlsToCrawl.map(u => u.url),
  pages: [{url: 'https://example.com/', status: 200, title: '...'}],
  links: [{from: 'https://example.com/', to: 'https://example.com/about'}],
  discovered: ['https://example.com/about', 'https://example.com/contact']
});

```

The method performs four operations atomically:

- Updates frontier rows to `state='crawled'`
- Inserts or updates `page_mirror` rows
- Stores link edges (respecting storage budget caps)
- Enqueues newly discovered URLs as `pending` with `source='link'`

## Handling Failures Through Lease Releases

When a chunk exceeds its soft deadline, uncompleted URLs must return to the queue. The `releaseUrls()` method (lines 32-40) resets `leased` entries to `pending`:

```typescript
await scratchpad.releaseUrls(['https://example.com/slow-page']);

```

This prevents **orphaned leases** that would stall the crawl indefinitely.

## Monitoring Frontier Health with Statistics

OpenSEO exposes frontier metrics via `getStats()` (lines 42-48), enabling the workflow to detect completion or abort conditions:

| Metric | Meaning |
|--------|---------|
| `attempted` | URLs that reached `recordBatch()` |
| `pending` | URLs awaiting first lease |
| `seen` | Total unique URLs in frontier |

The workflow in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) polls these stats to decide when no work remains.

## Automatic Cleanup and Resource Limits

Every `AuditScratchpad` instance schedules a 7-day destruction alarm in its constructor (lines 17-23). This guarantees no persistent crawl state leaks after completion, even if workflows fail to signal cleanup.

Additionally, link storage operates under a **budget cap** to prevent SQLite size explosions on densely interlinked sites—a practical safeguard absent from simpler queue-based frontier implementations.

## Summary

- **Localized state**: SQLite in a Durable Object avoids large workflow payloads
- **Idempotent leasing**: Chunk-based retries prevent duplicate crawling
- **Smart prioritization**: Link-discovered URLs precede sitemap-only entries
- **Fault recovery**: Lease releases and statistics enable graceful degradation
- **Resource safety**: Storage budgets and automatic cleanup prevent runaway growth

## Frequently Asked Questions

### How does OpenSEO prevent duplicate crawling when workflow steps retry?

`claimChunk()` checks for existing leases by chunk number before selecting new URLs. If the same chunk is claimed twice, it returns the already-leased set, making the operation idempotent.

### Why does OpenSEO prioritize link-discovered URLs over sitemap URLs?

Links indicate editorial importance and crawlability—pages with more inbound links are typically more significant for SEO. Sitemap entries may include orphaned or low-value pages, so they receive secondary priority while still being crawled eventually.

### What happens if a crawl is interrupted mid-batch?

Any URLs still in `leased` state from incomplete chunks can be released via `releaseUrls()` and re-claimed by subsequent chunks. The frontier remains consistent, and `recordBatch()` can be safely re-invoked with updated results.

### Where does URL policy enforcement occur before frontier insertion?

The [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) module validates same-origin constraints and robots.txt rules before discovered URLs reach `recordBatch()`, ensuring only policy-compliant URLs enter the frontier queue.