How to Handle Rate Limiting Across Multiple Data Providers in OpenBB

OpenBB implements a uniform, provider-level throttling mechanism using configurable sleep parameters, asynchronous delays, and LRU caching to manage rate limits across dozens of external APIs including FRED, Intrinio, SEC, and Nasdaq.

OpenBB accesses external data services that each enforce unique request-rate policies. The OpenBB-finance/OpenBB repository provides a consistent framework to handle rate limiting across multiple data providers through configurable delays and intelligent caching, ensuring reliable data retrieval without triggering API caps.

Provider-Specific Sleep Parameters

Most provider models expose a sleep query parameter that instructs the fetcher how long to pause between consecutive HTTP calls. This value is configurable when constructing query objects, allowing fine-tuning based on each provider's documented limits.

Intrinio and FRED Series (Async)

In openbb_platform/providers/intrinio/openbb_intrinio/models/fred_series.py, the model defines a sleep field and applies asyncio.sleep before each request:

sleep: float | None = Field(
    description="Time to sleep between requests to avoid rate limiting."
)
...
await asyncio.sleep(query.sleep or 1.0)

SEC Form 4 Filings (Async)

The SEC utility in openbb_platform/providers/sec/openbb_sec/utils/form4.py deliberately throttles to stay under the SEC's 10-request-per-second limit:

await asyncio.sleep(1.125)

Nasdaq Company Filings (Sync)

For synchronous fetchers like Nasdaq, the code uses time.sleep as shown in openbb_platform/providers/nasdaq/openbb_nasdaq/models/company_filings.py:

time.sleep(2)

The Abstract Fetcher Base Class

All providers inherit from openbb_core.provider.abstract.fetcher.Fetcher. This base class centralizes error handling and ensures the request-retry loop remains consistent across providers. While concrete rate-limit logic lives in each provider, the abstract class guarantees that any sleep value propagates through the request pipeline:

# openbb_platform/core/openbb_core/provider/abstract/fetcher.py

class Fetcher(Generic[Q, R]):
    async def request(self, query: Q) -> R:
        # provider-specific sleep is applied before the HTTP call

        ...

Caching to Reduce API Traffic

OpenBB employs an LRU cache for provider responses to minimize outbound calls. By reusing recent results, the platform automatically lowers request volume, further mitigating rate-limit violations. The caching implementation appears in openbb_platform/core/tests/provider/utils/test_lru.py:

@lru_cache(maxsize=128)
def cached_fetch(...):
    ...

Uniform Error Handling for Rate Limits

When a provider detects a 429 (Too Many Requests) response, it raises a clear exception containing the phrase "You may be experiencing rate limiting". This pattern appears in FRED models such as openbb_platform/providers/fred/openbb_fred/models/retail_prices.py:

raise HTTPError("You may also be experiencing rate limiting.")

The uniform wording helps downstream CLI and UI components present consistent guidance to users.

Best Practices for Configuring Rate Limits

Follow this workflow to handle rate limiting across multiple data providers in OpenBB:

1. Inspect provider documentation in openbb_platform/providers/<provider>/README.md to identify official rate limits.
2. Set the sleep parameter in the query model to a safe cushion (e.g., sleep=2 for Nasdaq, sleep=1 for Intrinio).
3. Enable caching for repeated queries to avoid identical API calls within short time frames.
4. Handle EmptyDataError or HTTPError exceptions; retry with increased sleep values if throttling occurs.

Practical Code Examples

Calling Intrinio FRED Series with Custom Throttling

from openbb_platform.providers.intrinio.openbb_intrinio.models.fred_series import (
    FredSeriesQueryParams,
)

# Request the "GDP" series, pause 1.5 seconds between calls

params = FredSeriesQueryParams(symbol="GDP", sleep=1.5)
result = await FredSeriesFetcher().fetch(params)
print(result.df.head())

Source: openbb_platform/providers/intrinio/openbb_intrinio/models/fred_series.py

Fetching SEC Form 4 Filings Within Rate Limits

from openbb_platform.providers.sec.openbb_sec.utils.form4 import fetch_form4

# The utility internally sleeps 1.125 seconds after each request

records = await fetch_form4(cik="0000320193")  # Apple Inc.

print(records[:3])

Source: openbb_platform/providers/sec/openbb_sec/utils/form4.py

Nasdaq Company Filings with Synchronous Delay

from openbb_platform.providers.nasdaq.openbb_nasdaq.models.company_filings import (
    CompanyFilingsFetcher,
    CompanyFilingsQueryParams,
)

params = CompanyFilingsQueryParams(symbol="AAPL")

# The fetcher will pause 2 seconds before the request

data = CompanyFilingsFetcher().fetch(params)
print(data.df.head())

Source: openbb_platform/providers/nasdaq/openbb_nasdaq/models/company_filings.py

Using LRU Cache to Avoid Unnecessary Calls

from functools import lru_cache

@lru_cache(maxsize=64)
def get_fred_series(symbol: str):
    from openbb_platform.providers.fred.openbb_fred.models.retail_prices import (
        RetailPricesFetcher,
        RetailPricesQueryParams,
    )
    params = RetailPricesQueryParams(symbol=symbol)
    return RetailPricesFetcher().fetch(params)

# Subsequent calls within the cache window hit memory, not the API

df1 = get_fred_series("CPIAUCSL")
df2 = get_fred_series("CPIAUCSL")  # cached

Source: openbb_platform/providers/fred/openbb_fred/models/retail_prices.py

Summary

• Configurable sleep parameters in query models control delays between HTTP requests for each provider.
• Async and sync implementations use asyncio.sleep or time.sleep based on the fetcher type, with hardcoded values like 1.125 seconds for SEC and 2 seconds for Nasdaq.
• The abstract Fetcher class ensures consistent propagation of throttling settings across all providers.
• LRU caching reduces API traffic by storing recent responses, minimizing the risk of hitting rate caps.
• Uniform error messages containing "You may be experiencing rate limiting" enable consistent user guidance across the platform.

Frequently Asked Questions

How does OpenBB handle different rate limits for each data provider?

OpenBB handles different rate limits through provider-specific sleep parameters defined in each model's query params. According to the source code in openbb_platform/providers/intrinio/openbb_intrinio/models/fred_series.py and related files, each provider implements its own delay logic while inheriting common error handling from the abstract Fetcher base class.

Can I configure the delay between API requests in OpenBB?

Yes, most provider models expose a sleep field that accepts float values representing seconds to pause between requests. When constructing query objects like FredSeriesQueryParams, you can pass sleep=1.5 or similar values to customize throttling behavior based on the provider's documented limits.

What error message does OpenBB show when rate limiting occurs?

When detecting 429 responses or similar throttling scenarios, OpenBB raises HTTPError exceptions containing the specific phrase "You may be experiencing rate limiting". This pattern appears in files like openbb_platform/providers/fred/openbb_fred/models/retail_prices.py, enabling consistent error handling across CLI and UI components.

Does OpenBB cache API responses to avoid hitting rate limits?

Yes, OpenBB implements LRU caching as shown in openbb_platform/core/tests/provider/utils/test_lru.py. The @lru_cache decorator stores recent provider responses in memory, allowing subsequent identical queries to return cached data instead of making new API calls, effectively reducing outbound request volume.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}