# How ML Intern Implements Retry Logic for Transient LLM Errors and Rate Limiting

> ML Intern implements robust retry logic for LLM errors. It retries failed calls up to three times with progressive back-off and handles rate limiting to ensure smooth operation.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: internals
- Published: 2026-04-24

---

**ML Intern retries failed LLM calls up to three times using progressive back-off delays of 5, 15, and 30 seconds, automatically detecting transient errors including HTTP 429 rate limits and server outages before escalating permanent failures to the user.**

The huggingface/ml-intern repository protects its agent loop with a deterministic retry mechanism designed specifically for temporary LLM service interruptions. This implementation distinguishes between transient network issues and permanent configuration errors, ensuring that rate limiting and momentary capacity problems do not terminate user sessions. Understanding this retry logic for transient LLM errors and rate limiting reveals how the system maintains conversation flow despite external API instability.

## Core Retry Configuration

The retry behavior is governed by two module-level constants defined in [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py). **`_MAX_LLM_RETRIES`** is set to `3` (line 119), establishing a hard ceiling on retry attempts before the system propagates the error to the user.

```python

# From agent/core/agent_loop.py, lines 119-121

_MAX_LLM_RETRIES = 3
_LLM_RETRY_DELAYS = [5, 15, 30]

```

**`_LLM_RETRY_DELAYS`** (lines 120-121) defines the progressive back-off schedule in seconds. The first retry waits 5 seconds, the second waits 15 seconds, and the final attempt waits 30 seconds. This linear-to-exponential ramp prevents aggressive re-polling while giving temporary outages time to resolve.

## Detecting Transient Errors

Before any retry executes, the system validates that the exception is recoverable. The **`_is_transient_error`** function (lines 124-138) inspects error messages for specific patterns indicating temporary conditions:

- **Timeout indicators**: `timeout`, `timed out`
- **Rate limiting**: HTTP status `429`, `rate limit`, `rate_limit`
- **Server errors**: HTTP statuses `503`, `502`, `500`
- **Capacity issues**: `overloaded`, `capacity`, connection reset/refused, EOF, broken pipe

Only errors matching these patterns trigger the retry path. Configuration errors or authentication failures bypass this logic and fail immediately.

## Retry Implementation in LLM Calls

The retry loop is embedded directly into the LLM invocation helpers, ensuring consistent behavior across streaming and non-streaming execution paths.

### Streaming Requests (`_call_llm_streaming`)

In **`_call_llm_streaming`** (lines 1000-1035), the implementation wraps the API call in a `for` loop iterating over the maximum retry count. When an exception occurs, the code invokes `_is_transient_error` to classify the failure:

```python

# Simplified representation of the retry block in _call_llm_streaming

for _llm_attempt in range(_MAX_LLM_RETRIES):
    try:
        # ... LLM API call ...

        break  # Success exits the loop

    except Exception as exc:
        if _llm_attempt < _MAX_LLM_RETRIES - 1 and _is_transient_error(exc):
            delay = _LLM_RETRY_DELAYS[_llm_attempt]
            # Log warning and emit tool_log event to UI

            await session.send_event("tool_log", f"LLM connection error, retrying in {delay}s...")
            await asyncio.sleep(delay)
            continue
        raise  # Non-transient or final attempt re-raises

```

### Non-Streaming Requests (`_call_llm_non_streaming`)

**`_call_llm_non_streaming`** (lines 992-1024) mirrors this exact structure for synchronous-style invocations. Both paths share the same retry budget and delay schedule, ensuring uniform resilience regardless of how the response is consumed.

## Handling Configuration Errors Before Retry Budget

ML Intern implements an **effort-healing shortcut** that executes before the transient retry logic consumes an attempt. Located around lines 1015-1022 in `_call_llm_streaming`, this block checks for effort-configuration errors (e.g., requesting "thinking" mode on a model that does not support it).

If the error matches an effort-configuration problem, the system heals the request parameters and retries **once** without incrementing the `_llm_attempt` counter. This preserves the three-attempt budget for genuine transient errors while automatically fixing incompatible parameter combinations.

## User Feedback and Event Emission

During each retry cycle, the agent emits a `tool_log` event via the **`session.send_event`** method. This broadcasts the back-off countdown to the frontend, allowing the UI to display messages like "LLM connection error, retrying in 15s..." to keep users informed of temporary delays.

Warning-level logs are also generated server-side, creating an audit trail of which specific error patterns triggered retries.

## Code Examples

### Example 1: Direct Streaming Invocation

When using the internal streaming helper, retry logic executes automatically:

```python
from agent.core.agent_loop import _call_llm_streaming

# Existing Session object required for context and event emission

messages = [{"role": "user", "content": "Explain transformer architecture."}]
tools = []
llm_params = {
    "model": "gpt-4o-mini",
    "temperature": 0.7,
}

# Automatically handles up to 3 retries with 5s/15s/30s delays

result = await _call_llm_streaming(session, messages, tools, llm_params)

```

### Example 2: Manual Retry Implementation

For custom tooling outside the agent loop, reproduce the delay schedule exactly:

```python
import asyncio

_MAX_LLM_RETRIES = 3
_RETRY_DELAYS = [5, 15, 30]

async def resilient_llm_call(session, messages, tools, llm_params):
    for attempt in range(_MAX_LLM_RETRIES):
        try:
            response = await acompletion(
                messages=messages, 
                tools=tools, 
                **llm_params
            )
            return response
        except Exception as exc:
            if attempt < _MAX_LLM_RETRIES - 1 and _is_transient_error(exc):
                delay = _RETRY_DELAYS[attempt]
                print(f"Transient error ({exc}); retrying in {delay}s …")
                await asyncio.sleep(delay)
            else:
                raise

```

## Summary

- **Three-attempt limit**: ML Intern caps retries at `_MAX_LLM_RETRIES = 3` before surfacing errors to users.
- **Progressive back-off**: Delays escalate from 5 to 15 to 30 seconds to allow service recovery.
- **Pattern-based detection**: The `_is_transient_error` function filters for HTTP 429, 503, timeouts, and capacity-related strings.
- **Dual coverage**: Both `_call_llm_streaming` and `_call_llm_non_streaming` implement identical retry blocks.
- **Effort healing**: Configuration errors are fixed and retried once without consuming the transient error budget.

## Frequently Asked Questions

### How does ML Intern distinguish between rate limits and permanent errors?

The system uses the **`_is_transient_error`** function in [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py) (lines 124-138) to scan exception messages for specific substrings including `429`, `rate limit`, `503`, `timeout`, and `capacity`. Messages containing these patterns trigger the retry logic, while authentication or validation failures propagate immediately.

### Can I configure the number of retries or delay durations?

Currently, **`_MAX_LLM_RETRIES`** and **`_LLM_RETRY_DELAYS`** are hardcoded constants at lines 119-121 of [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py). Modifying these values requires editing the source code, as the repository does not expose runtime configuration for these parameters.

### What happens if all three retry attempts fail?

After exhausting the retry budget in `_call_llm_streaming` or `_call_llm_non_streaming`, the original exception is re-raised and converted into a user-facing error event. The agent terminates the current operation and reports the failure through the session's event system rather than hanging indefinitely.

### Does effort healing count against the three retry attempts?

No. According to the implementation at lines 1015-1022, when an effort-configuration error is detected, the system heals the request parameters and retries **once** without incrementing the `_llm_attempt` counter. This ensures that fixing incompatible model parameters does not reduce the budget available for handling transient network errors.