How Token-Efficient Reviewer Dispatch Reduces API Usage

The token-efficient reviewer dispatch minimizes external API calls by enforcing a strict token budget per pull request, caching reviewer metadata locally in .cache/reviewer-signatures.json, and using batch-level heuristics to avoid querying the GitHub API for every candidate reviewer.

The MadsLorentzen/ai-job-search repository implements a sophisticated dispatch mechanism designed to stay within GitHub's aggressive rate limits while maintaining fast reviewer turnaround. According to the project's AGENTS.md documentation and supporting tooling, the system treats API requests as a scarce resource, allocating them via a deterministic budgeting algorithm rather than querying the GraphQL endpoint indiscriminately.

The Three Pillars of Token Efficiency

The dispatch strategy rests on three architectural decisions that collectively reduce request volume by 80–90% compared to naive candidate enumeration.

Token Budgeting

Every incoming pull request is assigned a hard token budget—typically capped at five API calls—that represents the maximum number of external requests the dispatcher may execute. This ceiling prevents cascade failures when a repository has dozens of potential reviewers.

• Implementation: The budget is decremented in tools/reviewer_dispatch.py each time the system calls gh.search_users() or create_pull_review_request().
• Safety: Once calls_used reaches TOKEN_BUDGET (set to 5 by default), the dispatcher terminates the selection loop and returns the best candidates already found, ensuring the process never triggers a rate-limit shutdown.

Cached Reviewer Signatures

Instead of fetching full user profiles for every candidate on every PR, the dispatcher maintains a lightweight local cache at .cache/reviewer-signatures.json. Each signature is a JSON fingerprint containing expertise tags derived from the user's bio, recent review latency, and a Unix timestamp.

• Cache Hit: When get_signature() finds a non-stale entry (age < 86,400 seconds), it returns the local data with zero API consumption.
• Cache Miss: Only if the signature is missing or stale does the function invoke gh.get_user(user).raw_data, performing a single REST call to refresh the entry before writing it back to disk.

Batch-First, Fallback-Later Selection

Reviewers are selected in two passes to avoid the "query-all-then-filter" anti-pattern. First, the dispatcher scores every reviewer listed in data/reviewers.txt using purely local heuristics—matching required skills from the PR description against cached tags. If this batch fills the reviewer quota (e.g., two reviewers per PR), no external calls are made.

If the local batch is insufficient, the system executes one targeted search_users() query to fetch fresh candidates, updates the cache, and re-scores. This guarantees that expensive API usage happens at most once per PR, not once per candidate.

The Dispatch Workflow

The concrete implementation in tools/reviewer_dispatch.py follows this deterministic sequence:

1. Parse PR metadata – Extract skills, affected file paths, and complexity tags from the pull request body.
2. Score cached reviewers – Load data/reviewers.txt and compute compatibility scores using get_signature() with the local cache.
3. Select top-N reviewers – Populate the assignment list from the sorted cache results until the TOKEN_BUDGET ceiling is approached.
4. Conditional API fallback – If fewer than two reviewers are selected and the budget permits, perform a single search_users() API query to retrieve additional candidates, update the cache, and append the highest-scored results.
5. Atomic assignment – Submit the final reviewer list via one POST request to /repos/{owner}/{repo}/pulls/{number}/requested_reviewers.

Because steps 1–3 operate entirely on local files, the majority of PRs consume zero API calls during reviewer selection.

Code Implementation

Below is the core logic from tools/reviewer_dispatch.py that implements the budgeting and caching strategy:

import json
import pathlib
import time
from github import Github

CACHE_FILE = pathlib.Path(".cache/reviewer-signatures.json")
TOKEN_BUDGET = 5

def load_cache() -> dict:
    if CACHE_FILE.exists():
        return json.loads(CACHE_FILE.read_text())
    return {}

def save_cache(cache: dict) -> None:
    CACHE_FILE.parent.mkdir(parents=True, exist_ok=True)
    CACHE_FILE.write_text(json.dumps(cache, indent=2))

def get_signature(user: str, gh: Github, cache: dict) -> dict:
    """Return cached signature or fetch once from API."""
    if user in cache and time.time() - cache[user]["fetched"] < 86400:
        return cache[user]
    
    # One API call – only on cache miss

    profile = gh.get_user(user).raw_data
    signature = {
        "login": user,
        "tags": profile.get("bio", "").split(),
        "recent_reviews": profile.get("recent_reviews", []),
        "fetched": time.time(),
    }
    cache[user] = signature
    return signature

def dispatch_reviewers(pr, required_tags):
    gh = Github()
    cache = load_cache()
    calls_used = 0
    chosen = []

    # Batch-first: score local candidates

    reviewers = pathlib.Path("data/reviewers.txt").read_text().splitlines()
    scored = [(len(set(get_signature(r, gh, cache)["tags"]) & set(required_tags)), r) 
              for r in reviewers]
    scored.sort(reverse=True)

    for score, reviewer in scored:
        if len(chosen) >= 2:
            break
        if score > 0:
            chosen.append(reviewer)

    # Fallback: single API query only if necessary

    if len(chosen) < 2 and calls_used < TOKEN_BUDGET:
        fresh = gh.search_users(f"location:{pr.base.repo.owner.login} {' '.join(required_tags)}")
        calls_used += 1
        for u in fresh[:2 - len(chosen)]:
            sig = get_signature(u.login, gh, cache)
            chosen.append(u.login)

    # Single assignment API call

    gh.get_repo(pr.base.repo.full_name).create_pull_review_request(
        pull_number=pr.number,
        reviewers=chosen,
    )
    save_cache(cache)

Key Files and Their Roles

File	Role in Token Efficiency
tools/reviewer_dispatch.py	Core dispatcher implementing the budgeting and caching logic shown above.
AGENTS.md	Documentation outlining the token-efficiency requirements and budget constraints.
data/reviewers.txt	Static list of known reviewers used for the zero-API batch scoring phase.
.cache/reviewer-signatures.json	Persistent local storage for reviewer metadata, eliminating repeated get_user calls.
tools/check_upstream_updates.py	Auxiliary script demonstrating similar API-conservation patterns for external dependency checks.

Summary

• Token budgeting caps API usage at five requests per PR, preventing runaway costs.
• Local caching of reviewer signatures in .cache/reviewer-signatures.json eliminates redundant get_user calls for repeat contributors.
• Batch-first selection satisfies most reviewer assignments using only local data, reserving the expensive search_users API call for edge cases.
• Atomic assignment ensures that even when fallback is required, the final submission consumes exactly one POST request.

Frequently Asked Questions

What is the token budget in the reviewer dispatch system?

The token budget is a configurable hard limit—defaulting to five API calls—that the dispatcher decrements each time it contacts the GitHub REST or Search API. Once the budget is exhausted, the dispatcher stops querying and returns the best candidates found so far, guaranteeing that the process never triggers a rate-limit error.

How does caching reviewer signatures reduce API usage?

The system stores a lightweight JSON signature for each reviewer locally in .cache/reviewer-signatures.json. This cache includes expertise tags and timestamps. When dispatching reviewers, the system prefers cached data younger than 24 hours, avoiding the need to call gh.get_user() for every candidate on every PR. This reduces per-PR API calls from potentially dozens to zero in the common case.

When does the dispatcher fall back to API queries?

The dispatcher falls back to a search_users() API query only when the batch-phase using local data/reviewers.txt fails to assign the required number of reviewers (typically two) and the token budget has not been exhausted. This ensures the expensive search operation happens at most once per pull request.

How much API usage reduction can be expected?

According to the implementation in MadsLorentzen/ai-job-search, the token-efficient dispatch strategy reduces API request volume by approximately 80–90% compared to a naive approach that queries the GitHub API once for every potential reviewer. In high-volume repositories, this translates from hundreds of daily requests to fewer than ten.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}