Performance Considerations When Using every-app/open-seo: Optimizing for Cloudflare Workers Limits

Open SEO enforces strict internal caps on payload size, concurrency, and batch operations to stay within Cloudflare Workers' hard runtime limits, ensuring reliable audits without workflow failures.

When running SEO audits at scale, understanding the performance constraints baked into every-app/open-seo is critical for avoiding throttling and failed workflows. The codebase is architected specifically for the Cloudflare Workers environment, where step outputs must remain under 1 MiB and network concurrency is strictly rationed. This guide examines the internal throttling mechanisms, memory management strategies, and configuration options that govern audit performance.

Cloudflare Workers Runtime Constraints

The architecture of Open SEO is fundamentally shaped by Cloudflare Workers' execution model. The platform imposes a 1 MiB step output limit and restricted network I/O quotas that cannot be exceeded without triggering runtime exceptions.

Every workflow step in the audit pipeline must respect these boundaries. In src/server/workflows/siteAuditWorkflowPhases.ts, the code defines SITEMAP_SEED_BYTE_BUDGET = 768 KB to ensure the discovery phase never approaches the 1 MiB ceiling. This hard cap prevents the workflow from crashing when processing large sitemaps, automatically truncating the seed list before it can overflow the Workers payload limit.

Internal Batching and Throttling Mechanisms

To maintain performance within these constraints, the repository implements several fixed concurrency constants and batching strategies.

Sitemap Discovery Byte Budgeting

During the discovery phase, Open SEO strictly limits the initial URL seed list to prevent memory exhaustion. The constant SITEMAP_SEED_BYTE_BUDGET set to 768 KB in src/server/workflows/siteAuditWorkflowPhases.ts acts as a safety valve, ensuring that the initial crawl payload remains well under the 1 MiB Workers limit even when processing expansive XML sitemaps.

Crawl Concurrency Controls

Network parallelism is throttled through CRAWL_CONCURRENCY = 25, defined in src/server/workflows/siteAuditWorkflowCrawl.ts. This constant controls how many URLs are fetched simultaneously per batch.

Setting concurrency to 25 provides a deliberate trade-off between crawl speed and network reliability. Higher values risk saturating the Workers network I/O quota and triggering rate-limit errors (HTTP 429), while lower values unnecessarily extend audit duration for smaller sites.

Lighthouse Batch Processing

Performance audits are further constrained by LIGHTHOUSE_URL_BATCH_SIZE = 10 in src/server/workflows/siteAuditWorkflowPhases.ts. Since Lighthouse returns roughly two results per URL (mobile and desktop analyses), limiting batches to 10 URLs keeps the aggregated JSON response under the 1 MiB step output limit while minimizing the number of discrete Worker steps required to complete a full site audit.

Memory Management and Persistence Strategies

Open SEO avoids holding large datasets in memory by streaming progress to external storage and chunking database operations.

KV-Based Progress Tracking

Rather than accumulating crawled URLs in memory, the system persists progress to Cloudflare KV after each batch completes. In src/server/workflows/siteAuditWorkflowCrawl.ts, the AuditProgressKV.pushCrawledUrls method writes checkpoint data immediately following Promise.allSettled resolution.

This approach provides two performance benefits: it prevents memory bloat during large crawls and creates recoverable checkpoints if a workflow step is interrupted or retried.

Database Write Optimization

Bulk inserts are handled through the runBatch utility in src/db/runBatch.ts, which automatically splits SQL statements into chunks of ≤ 80 statements before execution. This chunking prevents overflow of D1's ~100 parameter limit per statement and maintains high write-throughput for both D1 and Postgres backends.

The helper ensures atomic transactions without manual batch management, as demonstrated by its internal logic that recursively splits oversized statement arrays until each chunk complies with driver constraints.

External API Efficiency

For paid data providers like DataForSEO, Open SEO implements credit-aware batching to minimize API costs and latency. In src/server/lib/dataforseo/keyword-metrics.ts, requests are capped at approximately 700 keywords per batch, reducing the number of external API calls while staying within provider-specific payload limits.

This batching strategy prevents runaway credit consumption on large keyword sets and reduces the cumulative network latency associated with sequential API requests.

Graceful Degradation Under Load

The crawling implementation uses Promise.allSettled rather than Promise.all when executing runCrawlBatch in src/server/workflows/siteAuditWorkflowCrawl.ts. This design choice ensures that a single failed fetch—whether due to timeout, DNS failure, or server error—does not abort the entire batch.

By isolating failures to individual URLs while allowing successful requests to persist, the system maintains throughput resilience even when scanning sites with intermittent availability issues.

Configuration Examples for Performance Tuning

While many limits are hardcoded for safety, several configuration options allow you to adjust audit scope and behavior.

Adjusting Audit Scope with maxPages

Control the total audit size by passing the maxPages parameter to runAuditPhases. This value directly influences the remaining argument passed to selectNextCrawlBatch, respecting the internal concurrency caps while limiting total execution time.

import { runAuditPhases } from "@/server/workflows/siteAuditWorkflowPhases";

const config = {
  maxPages: 2000,               // default is 1000; increase only if needed
  lighthouseStrategy: "full",   // "none" skips Lighthouse checks
};

await runAuditPhases(step, {
  auditId: "audit123",
  workflowInstanceId: "wf-abc",
  billingCustomer,
  projectId: "proj-42",
  startUrl: "https://example.com",
  config,
});

Modifying Crawl Concurrency (Advanced)

For deployments with higher Cloudflare quota limits, you can fork the repository and adjust the concurrency constant:

// In src/server/workflows/siteAuditWorkflowCrawl.ts
const CRAWL_CONCURRENCY = 40; // increase from 25 if you have higher quota

Monitor step logs for "429 Too Many Requests" errors when increasing this value, as higher concurrency can exhaust Workers' network subrequests allowance.

Using the runBatch Helper for Bulk Operations

When implementing custom database operations, use the provided batching utility to stay within driver limits:

import { runBatch } from "@/db/runBatch";

const statements = rows.map(row => db.prepare(`
  INSERT INTO audit_pages (audit_id, url, title, status_code)
  VALUES (?, ?, ?, ?)
`).bind(auditId, row.url, row.title, row.statusCode));

await runBatch(db, statements); // automatically splits into safe chunks

This helper guarantees compliance with D1's 100-parameter limit and ensures Postgres receives properly sized transaction batches.

Performance Timing with Workers API

For custom instrumentation, leverage the Workers Performance API available in the runtime:

const start = performance.now();
await someHeavyWork();
const durationMs = performance.now() - start;
console.log(`Work took ${durationMs.toFixed(2)} ms`);

The performance.now() method is declared in worker-configuration.d.ts and provides sub-millisecond precision without importing external dependencies.

Summary

• Hard limits rule the architecture: Open SEO caps sitemap seeds at 768 KB and Lighthouse batches at 10 URLs to stay within Cloudflare Workers' 1 MiB step output limit.
• Concurrency is deliberately throttled: The CRAWL_CONCURRENCY = 25 constant in siteAuditWorkflowCrawl.ts balances speed against Workers' network I/O quotas.
• Persistence prevents memory bloat: Crawl progress streams to KV storage after each batch, and database writes are chunked into ≤ 80-statement batches via runBatch.ts.
• External APIs are credit-protected: Data provider calls are batched at ~700 keywords per request to minimize latency and cost.
• Failure isolation maintains throughput: Promise.allSettled ensures individual URL failures do not stall entire crawl batches.

Frequently Asked Questions

What happens if the audit step output exceeds 1 MiB?

The Cloudflare Workers runtime will throw an exception and terminate the workflow step. Open SEO prevents this by enforcing SITEMAP_SEED_BYTE_BUDGET = 768 KB and LIGHTHOUSE_URL_BATCH_SIZE = 10 in src/server/workflows/siteAuditWorkflowPhases.ts, ensuring payloads remain safely under the limit.

Can I increase the crawl concurrency beyond 25?

Yes, but only by modifying the CRAWL_CONCURRENCY constant in src/server/workflows/siteAuditWorkflowCrawl.ts and redeploying your own fork. Increasing beyond 25 may trigger Cloudflare rate limits (HTTP 429) unless your Workers plan includes elevated network quotas.

How does Open SEO handle database inserts for thousands of pages?

The runBatch helper in src/db/runBatch.ts automatically splits large statement arrays into chunks of 80 or fewer statements. This prevents hitting D1's 100-parameter limit per statement and maintains atomic transaction integrity across both D1 and Postgres backends.

Is there a way to skip Lighthouse audits to improve speed?

Yes, set lighthouseStrategy: "none" in the audit configuration object passed to runAuditPhases. This bypasses the Lighthouse API calls entirely, eliminating the LIGHTHOUSE_URL_BATCH_SIZE = 10 throttling and significantly reducing total audit duration for large sites.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}