How the ArXiv Client Handles Rate Limiting and Retries: A Production-Grade Implementation

The ArXiv client enforces a configurable 3-second delay between all API requests and implements linear back-off retry logic for PDF downloads, ensuring compliance with arXiv's usage policies while handling transient network failures.

The ArxivClient in the jamwithai/production-agentic-rag-course repository is designed to be a responsible consumer of the public arXiv API. By combining global rate limiting with intelligent retry mechanisms, the client prevents API abuse and maximizes download reliability in production environments.

Core Rate-Limiting Mechanism

The client implements a unified rate-limiting strategy that applies to both API queries and PDF downloads, governed by the ArxivSettings.rate_limit_delay configuration parameter.

Global Request Throttling in fetch_papers

Before every API call, the client checks the timestamp of the previous request stored in self._last_request_time. If the elapsed time is less than self.rate_limit_delay (default 3 seconds), it calculates the remaining wait time and pauses execution asynchronously.

This logic appears in src/services/arxiv/client.py within both fetch_papers and fetch_papers_with_query:

# src/services/arxiv/client.py – rate-limit before a request

if self._last_request_time is not None:
    time_since_last = time.time() - self._last_request_time
    if time_since_last < self.rate_limit_delay:
        sleep_time = self.rate_limit_delay - time_since_last
        await asyncio.sleep(sleep_time)   # ← enforced pause

self._last_request_time = time.time()

This ensures sequential API calls maintain consistent spacing, preventing the client from overwhelming arXiv's servers regardless of how quickly the application code invokes methods.

PDF Download Rate Limiting

The _download_with_retry method unconditionally applies the rate limit before entering the retry loop. This guarantees that even fresh download attempts respect the global delay policy:

# src/services/arxiv/client.py – PDF download respects the same delay

async def _download_with_retry(...):
    # Respect rate limits for every download attempt

    await asyncio.sleep(self.rate_limit_delay)
    for attempt in range(max_retries):
        ...

Retry Logic for PDF Downloads

While API query failures surface immediately to the caller, PDF downloads implement automatic retry logic to handle transient network issues. This distinction recognizes that HTTP timeouts during large binary transfers are common and often self-resolving.

Linear Back-Off Strategy

The retry mechanism uses linear back-off rather than exponential. The wait time for each retry attempt equals download_retry_delay_base multiplied by the attempt count (1×, 2×, 3×, etc.), creating predictable retry intervals.

The implementation in src/services/arxiv/client.py handles both httpx.TimeoutException and httpx.HTTPError:

# src/services/arxiv/client.py – retry loop

max_retries = self._settings.download_max_retries
for attempt in range(max_retries):
    try:
        async with httpx.AsyncClient(timeout=float(self.timeout_seconds)) as client:
            async with client.stream("GET", url) as response:
                response.raise_for_status()
                # write chunks …

        return True                      # success → exit

    except httpx.TimeoutException as e:
        if attempt < max_retries - 1:
            wait_time = self._settings.download_retry_delay_base * (attempt + 1)
            await asyncio.sleep(wait_time)   # ← back-off

        else:
            raise PDFDownloadTimeoutError(...)
    except httpx.HTTPError as e:
        if attempt < max_retries - 1:
            wait_time = self._settings.download_retry_delay_base * (attempt + 1)
            await asyncio.sleep(wait_time)
        else:
            raise PDFDownloadException(...)

Error Handling and Max Attempts

The default configuration allows 3 retry attempts defined by ArxivSettings.download_max_retries in src/config.py. Unexpected exceptions (non-HTTP errors) are immediately re-raised without retry, preventing infinite loops on unrecoverable failures like disk permission errors or malformed URLs.

Configuration and Customization

All rate-limiting and retry parameters are centralized in src/config.py within the ArxivSettings class:

• rate_limit_delay: Minimum seconds between requests (default: 3)
• download_max_retries: Maximum PDF download attempts (default: 3)
• download_retry_delay_base: Base seconds for linear back-off calculation

The factory function in src/services/arxiv/factory.py instantiates the client with these settings loaded from environment variables or defaults, making the behavior configurable without code changes.

Usage Example

Integrate the client into your application using the factory method. The rate limiting and retries operate transparently:

from src.services.arxiv.factory import make_arxiv_client

# Create a fully-configured client (settings are read from the .env or defaults)

client = make_arxiv_client()

# Fetch the latest 5 AI papers (rate-limited automatically)

papers = await client.fetch_papers(max_results=5)

# Download the first paper's PDF (retries up to 3 times on failure)

pdf_path = await client.download_pdf(papers[0])

The client can be injected into higher-level services like MetadataFetcher without the caller managing rate-limits or retry policies.

Summary

• Global rate limiting enforces a 3-second minimum delay between all API requests by tracking self._last_request_time in src/services/arxiv/client.py.
• PDF downloads retry up to 3 times using linear back-off for httpx.TimeoutException and httpx.HTTPError.
• Configuration is centralized in ArxivSettings (src/config.py) and loaded via the factory pattern.
• API query failures surface immediately without retry, while download failures are retried to handle transient network issues.

Frequently Asked Questions

What is the default rate limit delay?

The default rate limit delay is 3 seconds between requests, defined in ArxivSettings.rate_limit_delay within src/config.py. This applies to both API queries and PDF downloads.

Why does the client use linear back-off instead of exponential back-off?

The implementation deliberately uses linear back-off (wait time = download_retry_delay_base × attempt count) to provide predictable, moderate delays between retry attempts. This approach balances resource conservation with quick recovery for transient PDF download failures typical in academic paper retrieval workflows.

Are API query failures retried?

No. The fetch_papers and fetch_papers_with_query methods raise exceptions immediately upon failure. According to the source code design, API query errors are considered rare and should surface to the caller, whereas PDF downloads are retried because network timeouts during large file transfers are common and often self-resolving.

How can I customize the retry settings?

Override the defaults by modifying environment variables referenced in ArxivSettings (defined in src/config.py) or by adjusting the download_max_retries and download_retry_delay_base parameters before passing settings to make_arxiv_client() in src/services/arxiv/factory.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}