How Open‑SEO's Rank Tracking Guard System Prevents API Over‑Fetching

Open‑SEO uses a multi‑layered guard system combining per‑tick budgets, wall‑clock deadlines, database‑level unique indexes, and stale‑run detection to cap SERP API calls and eliminate duplicate or runaway fetches.

The every-app/open-seo repository implements a sophisticated rank tracking guard system that protects external SERP API quotas from over‑consumption. Rather than relying on a single check, the system layers five independent safeguards across the scheduler, database schema, and workflow coordination. This article breaks down exactly how each mechanism works, where it lives in the codebase, and how they interact to guarantee API‑call limits are never exceeded.

Task‑Unit Budget Guard: Capping Per‑Tick Work

The scheduler's primary defense against burst traffic is a hard limit on keyword‑device combinations that can be initiated in a single tick.

In src/server/features/rank-tracking/services/scheduledRankChecks.ts, the constant SCHEDULED_TASK_UNIT_BUDGET = 1000 defines this ceiling. Before launching any run, the code calculates:

const taskUnits = kwCount * devicesCount(config.devices);

If unitsStarted + taskUnits would breach the budget, the loop terminates with stoppedByBudget = true. This ensures that even a backlog of thousands of due configurations cannot trigger more than 1,000 units worth of API calls in one scheduler execution.

Wall‑Clock Deadline: Preventing Worker Monopolization

Complementing the unit budget, a time‑based killswitch prevents any single tick from running indefinitely.

The same file declares TICK_DEADLINE_MS = 3 * 60_000 (three minutes). At each iteration, the scheduler compares Date.now() against deadline = tickStart + TICK_DEADLINE_MS. Once exceeded, the loop exits regardless of remaining budget. This protects against:

• Slow database queries stalling the scheduler
• Unexpected await‑blocks extending execution
• Overlapping tick executions caused by cron drift

Partial Unique Index: Database‑Level Exclusivity

The core coordination primitive is a PostgreSQL partial unique index that enforces single‑run semantics at the data layer.

In src/db/app.schema.ts, the index is defined as:

CREATE UNIQUE INDEX rank_check_runs_config_id_active
ON rank_check_runs (config_id)
WHERE status IN ('pending','running');

This guarantees that for any given config_id, only one row may exist with status = 'pending' or 'running'. Attempts to insert a duplicate fail with a constraint violation, which the application treats as an "already_running" signal. Unlike application‑level locking, this is race‑condition proof and survives process crashes.

Stale‑Run Detection: Reclaiming Orphaned Slots

When the unique index blocks insertion, the system must distinguish between valid in‑progress runs and zombie runs whose workflows have died.

The logic resides in src/server/features/rank-tracking/services/rankCheckRunGuards.ts. When blocked, beginRankCheckRun fetches the blocking run via getActiveRunForConfig, then evaluates staleness through getStaleRankCheckRunReason:

const staleReason = await getStaleRankCheckRunReason({
  run: blocker,
  runId: blocker.id,
  ageMs: Date.now() - new Date(blocker.startedAt).getTime(),
});

A run is stale if its workflow is missing, errored, terminated, or exceeds the startup grace window. When detected, failRunIfActive(blocker.id, staleReason, blocker) marks it failed—releasing the unique‑index slot—and the insertion retries.

Paid‑Plan and Zero‑Keyword Skip Logic

Before any API‑bound work begins, the scheduler applies business‑rule guards to avoid wasteful operations.

In scheduledRankChecks.ts, the function checkPaidPlan validates the organization's subscription. Free‑tier configs are claimed with lastSkipReason: "plan_required" and never reach the SERP API. Similarly, configs with zero keywords receive lastSkipReason: "no_keywords". These early exits populate skippedFree and skippedNoKeywords metrics for observability.

End‑to‑End Execution Flow

The five guards operate in strict sequence during each scheduler tick:

1. Select due configs — Query returns configs where next_check_at ≤ now.
2. Budget and deadline enforcement — Loop aborts when SCHEDULED_TASK_UNIT_BUDGET or TICK_DEADLINE_MS is exhausted.
3. Plan and keyword validation — Free or empty configs are claimed and skipped.
4. Slot reservation — RankTrackingRepository.claimDueConfig updates next_check_at and locks the row.
5. Workflow initiation — beginRankCheckRun attempts INSERT into rank_check_runs.
6. Conflict resolution — On unique‑index violation, stale‑run detection determines retry or skip.

Key Implementation Files

File	Responsibility
src/server/features/rank-tracking/services/scheduledRankChecks.ts	Budget, deadline, plan checks, and scheduler orchestration.
src/server/features/rank-tracking/services/rankCheckRunGuards.ts	Stale‑run detection, workflow status verification, and unique‑index conflict handling.
src/server/features/rank-tracking/repositories/RankTrackingRepository.ts	Database operations: claimDueConfig, tryCreateRun, getActiveRunForConfig.
src/db/app.schema.ts / src/db/pg/app.schema.ts	Partial unique index definition for run exclusivity.
src/server/features/rank-tracking/services/RankTrackingService.ts	Service‑layer consumption of repository guards.

Summary

• Task‑unit budget caps keyword‑device combinations per tick at 1,000.
• Wall‑clock deadline terminates ticks exceeding 3 minutes.
• Partial unique index enforces single active run per config at the database level.
• Stale‑run detection automatically recovers slots from crashed or hung workflows.
• Paid‑plan logic prevents API calls for free accounts or empty configurations.

Together, these mechanisms ensure the rank tracking system respects provider rate limits, minimizes redundant API consumption, and self‑heals from failure scenarios.

Frequently Asked Questions

What happens if two scheduler instances try to start the same rank check simultaneously?

The partial unique index on rank_check_runs (config_id) WHERE status IN ('pending','running') guarantees that only one INSERT succeeds. The second receives a constraint violation, which the guard interprets as "already_running" and skips. This database‑level enforcement is immune to race conditions between processes.

How does the system recover if a rank check workflow crashes without updating its status?

getStaleRankCheckRunReason in rankCheckRunGuards.ts queries the workflow status via getRankCheckWorkflowStatus. If the workflow is missing, terminated, or errored, the run is marked failed through failRunIfActive, releasing the unique‑index slot. A subsequent scheduler tick can then claim and restart the configuration.

Why use a partial unique index instead of application‑level locking?

Application locks (Redis, in‑memory, or advisory locks) fail across process restarts and network partitions. The PostgreSQL unique index persists with the data, survives crashes, and requires no additional infrastructure. It also provides exactly‑once semantics without the complexity of distributed lock coordination.

Can the budget and deadline constants be tuned for different deployment scenarios?

Yes. SCHEDULED_TASK_UNIT_BUDGET and TICK_DEADLINE_MS are defined as constants in scheduledRankChecks.ts. Adjusting these values changes the throughput ceiling and maximum tick duration. However, modifications should align with your SERP API provider's rate limits and the worker pool's capacity to avoid throttling or timeouts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}