How to Handle Rate Limiting Errors from AI Providers with Retry Logic in Open Notebook

Open Notebook automatically detects rate-limit responses from AI providers, converts them to structured RateLimitError exceptions, and implements configurable retry logic with exponential backoff to ensure robust API communication.

When building AI-powered applications with Open Notebook, encountering 429 "Too Many Requests" errors from providers like OpenAI or Anthropic is inevitable. The application uses a FastAPI backend that communicates with AI services through the Esperanto library, implementing a sophisticated error classification and retry system. This architecture ensures that transient rate limits don't disrupt user workflows while providing clear feedback when persistent limits are reached.

Understanding the Error Classification Pipeline

All provider-specific exceptions flow through a centralized classification system before reaching your application logic. This design allows Open Notebook to normalize different provider error formats into consistent exception types.

Mapping Provider Exceptions to RateLimitError

In open_notebook/utils/error_classifier.py, the classify_error function contains a rule set that identifies rate-limit patterns across different AI providers:

# open_notebook/utils/error_classifier.py

(
    ["rate limit", "rate_limit", "429", "too many requests", "quota exceeded"],
    RateLimitError,
    "Rate limit exceeded. Please wait a moment and try again.",
),

When the exception text matches any of these keywords, the function returns a RateLimitError instance along with a user-friendly message. This classification runs throughout the graph layers in open_notebook/graphs/*, ensuring consistent error handling regardless of which provider generated the original exception.

The RateLimitError Exception Type

The concrete exception class lives in open_notebook/exceptions.py, providing a distinct type for catching rate-limit specific failures:

# open_notebook/exceptions.py

class RateLimitError(OpenNotebookError):
    """Raised when a rate limit is exceeded."""
    pass

Inheriting from OpenNotebookError allows the application to distinguish rate-limit errors from other AI provider failures, enabling targeted retry strategies while maintaining a clear exception hierarchy.

HTTP Response Handling in FastAPI

Open Notebook registers a dedicated exception handler in api/main.py to ensure rate-limit errors translate to proper HTTP responses:

# api/main.py

@app.exception_handler(RateLimitError)
async def rate_limit_error_handler(request: Request, exc: RateLimitError):
    return JSONResponse(
        status_code=429,
        content={"detail": str(exc)},
        headers=_cors_headers(request),
    )

This handler automatically converts any uncaught RateLimitError into a 429 Too Many Requests response, preserving CORS headers for frontend consumption. The result is a clean API contract that informs clients exactly when they need to back off.

Built-in Retry Logic for Embeddings

Open Notebook implements generic retry logic for transient failures in the embedding pipeline, which automatically catches and retries rate-limit errors.

Configuration Constants

The retry behavior is defined in open_notebook/utils/embedding.py:

# open_notebook/utils/embedding.py

EMBEDDING_MAX_RETRIES = 3          # how many attempts per batch

EMBEDDING_RETRY_DELAY = 2          # seconds to wait before a retry

The Retry Loop Implementation

Each batch of texts processes through a retry loop that catches RateLimitError (classified from provider exceptions) and waits before retrying:

# open_notebook/utils/embedding.py

for attempt in range(1, EMBEDDING_MAX_RETRIES + 1):
    try:
        batch_embeddings = await embedding_model.aembed(batch)
        all_embeddings.extend(batch_embeddings)
        break
    except Exception as e:
        # Log and optionally back-off before next attempt

        if attempt < EMBEDDING_MAX_RETRIES:
            logger.debug(
                f"Embedding batch {batch_idx+1}/{total_batches} attempt {attempt} failed: {e}. "
                "Retrying..."
            )
            await asyncio.sleep(EMBEDDING_RETRY_DELAY)
        else:
            raise RuntimeError(...)

Because classify_error converts low-level provider exceptions into RateLimitError before they reach this loop, the embedding pipeline automatically handles rate-limit hits from any provider without provider-specific code.

Implementing Custom Retry Logic for LLM Calls

For operations outside the embedding pipeline, you can implement custom retry logic with exponential backoff using a pattern similar to the embedding code:

import asyncio
from open_notebook.exceptions import RateLimitError
from loguru import logger

async def call_with_rate_limit_retry(coro, *, max_retries=5, base_delay=1.0):
    """Execute `coro` (an awaitable) and retry on RateLimitError."""
    for attempt in range(1, max_retries + 1):
        try:
            return await coro()
        except RateLimitError as exc:
            if attempt == max_retries:
                logger.error("Rate‑limit retries exhausted")
                raise
            delay = base_delay * (2 ** (attempt - 1))   # exponential back‑off

            logger.warning(
                f"Rate limit hit (attempt {attempt}/{max_retries}); sleeping {delay}s"
            )
            await asyncio.sleep(delay)

Example Usage with Language Models

Apply this wrapper to LLM calls using the model manager from open_notebook/ai/models.py:

from open_notebook.ai.models import model_manager
from open_notebook.utils.error_classifier import classify_error

async def generate_chat_response(messages):
    async def _invoke():
        lm = await model_manager.get_language_model()
        return await lm.chat(messages)

    try:
        return await call_with_rate_limit_retry(_invoke)
    except Exception as e:
        # Convert any remaining exception into a user‑friendly message

        exc_class, user_msg = classify_error(e)
        raise exc_class(user_msg) from e

This pattern ensures that temporary rate limits trigger automatic retries with exponential backoff, while persistent failures surface as user-friendly errors after exhaustion.

Configuration and Tuning

Open Notebook exposes several configuration points for adjusting retry behavior:

• Environment Variables: Tune the embedding retry behavior without code changes:

  • OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE – controls the size of each batch sent to the provider
  • EMBEDDING_MAX_RETRIES and EMBEDDING_RETRY_DELAY – defined in open_notebook/utils/embedding.py and editable if you need different defaults

• Provider-Specific Headers: Some providers return a Retry-After header in their 429 responses. While the generic retry loop uses fixed delays, you can extend call_with_rate_limit_retry to parse this header and pass dynamic delays for more accurate backoff timing.

Summary

• Error Classification: The classify_error function in open_notebook/utils/error_classifier.py maps provider-specific rate-limit messages to the standard RateLimitError exception.
• Exception Handling: RateLimitError in open_notebook/exceptions.py provides a distinct type for catching these failures, while api/main.py converts them to HTTP 429 responses.
• Automatic Retries: The embedding pipeline in open_notebook/utils/embedding.py implements built-in retry logic with configurable EMBEDDING_MAX_RETRIES and EMBEDDING_RETRY_DELAY constants.
• Custom Implementation: Use the call_with_rate_limit_retry pattern with exponential backoff for LLM operations and other AI provider calls.
• User Experience: The system surfaces clear "Rate limit exceeded" messages while automatically backing off and retrying, minimizing disruption during temporary provider throttling.

Frequently Asked Questions

How does Open Notebook detect rate limiting errors from different AI providers?

Open Notebook uses the classify_error function in open_notebook/utils/error_classifier.py to scan exception messages for keywords like "rate limit", "429", "too many requests", and "quota exceeded". When any provider returns an error containing these strings, the function normalizes it into a RateLimitError with a consistent user-friendly message, regardless of whether the underlying provider is OpenAI, Anthropic, or a local Ollama instance.

What is the default retry behavior for embedding operations?

The embedding pipeline in open_notebook/utils/embedding.py automatically retries failed batches up to 3 times (EMBEDDING_MAX_RETRIES = 3) with a fixed delay of 2 seconds (EMBEDDING_RETRY_DELAY = 2) between attempts. This catches rate-limit errors after they've been classified by the error classifier, ensuring that brief provider throttling doesn't fail the entire embedding job.

Can I customize the retry logic for specific AI providers?

Yes. While Open Notebook provides built-in retry logic for embeddings, you can implement custom retry strategies for LLM calls or other operations using the call_with_rate_limit_retry pattern shown in the implementation examples. This allows you to adjust max_retries, implement exponential backoff instead of fixed delays, or read provider-specific Retry-After headers to customize backoff timing per provider.

What HTTP status code does Open Notebook return when rate limits are exceeded?

Open Notebook returns HTTP 429 Too Many Requests when rate-limit errors exhaust all retries or occur in contexts without retry logic. The FastAPI exception handler in api/main.py catches RateLimitError instances and returns a JSON response with status code 429 and a detail message explaining that the rate limit was exceeded, allowing frontend clients to implement their own backoff strategies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}