How OpenSEO Handles Rank Tracking: A Complete Technical Deep Dive

OpenSEO implements rank tracking as an asynchronous, credit-aware workflow using Cloudflare Workers with dual execution paths—live checks for manual triggers and queued polling for scheduled runs with automatic fallback.

OpenSEO is an open-source SEO platform built on Cloudflare's edge infrastructure. Its rank tracking system demonstrates how to build resilient, large-scale SERP monitoring that gracefully handles API rate limits, billing constraints, and partial failures. This article examines the actual implementation in the every-app/open-seo repository.

Architecture Overview

The rank tracking system spans three logical layers: workflow orchestration, execution paths, and persistence. All components run on Cloudflare Workers and use D1/PostgreSQL via Drizzle ORM for state management.

The core abstraction is a RankCheckRun—a single execution that processes all keywords for a given tracking configuration. Each run progresses through states: pending → running → completed or failed.

Workflow Orchestration

The entry point is RankCheckWorkflow in src/server/workflows/RankCheckWorkflow.ts. This class implements Cloudflare's Workflow API, providing durable execution with automatic retries and step-level observability.

Initialization and Validation

When triggered, the workflow performs three setup steps:

1. Claims the configuration to prevent concurrent runs against the same config
2. Validates billing credits via the Autumn service (src/server/billing/autumn.ts)
3. Enumerates keywords through prepareRankCheckKeywords

// src/server/workflows/RankCheckWorkflow.ts
export class RankCheckWorkflow extends WorkflowEntrypoint<Env, RankCheckPayload> {
  async run(event: WorkflowEvent<RankCheckPayload>, step: WorkflowStep) {
    const { runId, configId, billingCustomer, projectId, trigger } = event.payload;
    
    // Claim config and update status to 'running'
    await this.pgStep(step, 'claim-config', () => 
      RankTrackingRepository.claimConfig(configId, runId)
    );
    
    // Verify credits before proceeding
    const keywords = await prepareRankCheckKeywords(configId);
    await checkSufficientCredits(billingCustomer, keywords.length);
    
    // Route to appropriate execution path
    if (trigger === 'manual') {
      await runLiveCheck({ step, runId, keywords, ... });
    } else {
      await runQueuedCheck({ step, runId, keywords, ... });
    }
    
    await finalizeRankCheckRun(runId);
  }
}

The pgStep helper (src/server/workflows/pgStep.ts) wraps each operation in a Postgres-transaction-aware context with configurable retries and timeouts.

Live Check Path: Immediate SERP Retrieval

Manual rank checks use DataForSEO's live endpoint for synchronous results. This path prioritizes speed over cost.

Batch Processing

In src/server/workflows/rankCheckPaths.ts, runLiveCheck groups keywords into batches of KEYWORDS_PER_BATCH (defined in src/shared/rank-tracking.ts):

// src/server/workflows/rankCheckPaths.ts
export async function runLiveCheck(ctx: LiveCheckContext): Promise<void> {
  const { keywords, step, runId } = ctx;
  
  for (let i = 0; i < keywords.length; i += KEYWORDS_PER_BATCH) {
    const batch = keywords.slice(i, i + KEYWORDS_PER_BATCH);
    
    await pgStep(step, `live-batch-${i}`, BATCH_STEP_CONFIG, async () => {
      const results = await checkBatchLive(batch);
      await RankTrackingRepository.insertSnapshots(results);
      await RankTrackingRepository.incrementKeywordsChecked(runId, batch.length);
    });
  }
}

Single-Batch Execution

checkBatchLive uses Promise.allSettled to handle partial failures gracefully:

async function checkBatchLive(batch: RankCheckKeyword[]): Promise<RankSnapshot[]> {
  const settled = await Promise.allSettled(
    batch.map(k => client.serp.rankCheck({
      keyword: k.keyword,
      locationCode: k.locationCode,
      device: k.device,
      // ... additional params
    }))
  );
  
  return settled
    .filter((r): r is PromiseFulfilledResult<SerpResult> => r.status === 'fulfilled')
    .map(r => transformToSnapshot(r.value));
}

Each successful result becomes a rankSnapshots row with the detected position, URL, and SERP features present.

Queued Check Path: Cost-Optimized Scheduled Monitoring

Scheduled checks use DataForSEO's task queue API—cheaper but asynchronous. This path implements a polling loop with live fallback for reliability.

Task Submission

runQueuedCheck posts keywords in chunks of MAX_TASKS_PER_POST:

// src/server/workflows/rankCheckPaths.ts (excerpt)
export async function runQueuedCheck(ctx: QueuedCheckContext): Promise<void> {
  const pending: QueuedTask[] = [];
  const fallback: RankCheckKeyword[] = [];
  
  // Phase 1: Submit all keywords as queue tasks
  for (let i = 0; i < keywords.length; i += MAX_TASKS_PER_POST) {
    const chunk = keywords.slice(i, i + MAX_TASKS_PER_POST);
    const result = await client.serp.rankCheckTaskPost(chunk);
    
    if (result.success) {
      pending.push(...result.tasks);
    } else {
      fallback.push(...chunk); // Failed submission → direct live check
    }
  }

Polling with Configured Intervals

The workflow polls using QUEUED_POLL_INTERVALS—an array of sleep durations that back off appropriately:

// Inside runQueuedCheck
for (let round = 0; round < QUEUED_POLL_INTERVALS.length && pending.length > 0; round++) {
  await step.sleep(`wait-${round}`, QUEUED_POLL_INTERVALS[round]);
  
  const batch = pending.slice(0, TASK_GETS_PER_COLLECT);
  const outcome = await pgStep(step, `collect-${round}`, COLLECT_STEP_CONFIG, () =>
    collectQueuedRound(ctx, batch)
  );
  
  // Update state for next round
  pending = [...outcome.stillPending, ...pending.slice(TASK_GETS_PER_COLLECT)];
  fallback.push(...outcome.failed);
}

Classification and Fallback

collectQueuedRound calls DataForSEO's task_get endpoint and categorizes each task:

• Completed: Results inserted via RankTrackingRepository.insertSnapshots
• Still pending: Retained for next polling round
• Failed: Added to fallback list

After the final polling round, any remaining tasks are processed through checkBatchLive—ensuring no keyword is left unprocessed due to queue stagnation.

Data Persistence

The RankTrackingRepository (src/server/features/rank-tracking/repositories/RankTrackingRepository.ts) abstracts all database operations.

Atomic Snapshot Insertion

To handle potential duplicate runs, snapshots use onConflictDoNothing:

// src/server/features/rank-tracking/repositories/RankTrackingRepository.ts
async insertSnapshots(snapshots: RankSnapshot[]): Promise<void> {
  await executeInBatches(snapshots, (tx, snapshot) =>
    tx
      .insert(rankSnapshots)
      .values(snapshot)
      .onConflictDoNothing({
        target: [
          rankSnapshots.runId, 
          rankSnapshots.trackingKeywordId, 
          rankSnapshots.device
        ],
      })
  );
}

Schema Relationships

The database schema (src/db/schema.ts) defines three core tables:

• rankTrackingConfigs: User-defined tracking setups (keywords, locations, devices)
• rankCheckRuns: Individual execution records with status and progress
• rankSnapshots: Position results linked to runs and keywords

Finalization and Telemetry

When execution completes—or fails—the workflow calls dedicated cleanup functions:

Function	Purpose
finalizeRankCheckRun	Marks success, aggregates snapshot count, updates lastCheckedAt
markRankCheckRunFailed	Records error message, preserves partial progress

Both paths emit a PostHog event for analytics:

posthog.capture({
  event: 'rank_tracking:check_complete',
  properties: {
    runId,
    keywordsTotal,
    keywordsChecked,
    durationMs,
    trigger,
    hadErrors
  }
});

Triggering Rank Checks

Server functions provide the external interface. Here's how manual checks are initiated:

// src/serverFunctions/rank-tracking.ts
export async function startManualRankCheck(
  projectId: string,
  configId: string,
  billingCustomer: BillingCustomerContext,
): Promise<string> {
  const runId = crypto.randomUUID();
  
  await RankTrackingRepository.tryCreateRun({
    id: runId,
    configId,
    projectId,
    keywordsTotal: 0, // Updated after keyword enumeration
    status: 'pending'
  });

  // Dispatch workflow execution
  await new RankCheckWorkflow().run(
    { payload: { runId, configId, billingCustomer, projectId, trigger: 'manual' } },
    {} as WorkflowStep // Provided by Cloudflare runtime
  );
  
  return runId;
}

Scheduled checks use the same workflow with trigger: 'scheduled', invoked via Cloudflare Cron Triggers.

Summary

• OpenSEO rank tracking combines Cloudflare Workflows with DataForSEO APIs for durable, observable SERP monitoring
• Two execution paths optimize for speed (live) or cost (queued), with automatic fallback ensuring completeness
• Credit-aware design validates billing before execution and supports per-keyword cost estimation
• Resilient architecture uses Promise.allSettled, transaction-wrapped steps, and conflict-resistant inserts to handle partial failures

Frequently Asked Questions

What database does OpenSEO use for rank tracking?

OpenSEO uses PostgreSQL or Cloudflare D1 accessed through Drizzle ORM. The RankTrackingRepository class abstracts all queries, and the pgStep helper provides transaction management with retries. Schema definitions in src/db/schema.ts use Drizzle's type-safe table builders.

How does OpenSEO prevent duplicate rank check results?

Duplicate prevention happens at the database layer. The rankSnapshots table has a composite unique constraint on (runId, trackingKeywordId, device). The insertion uses onConflictDoNothing, so reprocessing the same keyword in a run produces no duplicate rows.

Can OpenSEO handle thousands of keywords in one run?

Yes. Both execution paths implement batching: live checks use KEYWORDS_PER_BATCH (typically 10-20), while queued checks submit MAX_TASKS_PER_POST tasks per API call (typically 100). The workflow persists progress after each batch, so runs can resume after interruptions.

What happens when DataForSEO's queue is slow?

The queued path includes a polling loop with configurable intervals (QUEUED_POLL_INTERVALS). After the final interval, any remaining unfinished tasks automatically fall back to live checks. This hybrid approach minimizes cost while guaranteeing result completeness.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}