How ML Intern Implements Retry Logic for Transient LLM Errors and Rate Limiting

ML Intern retries failed LLM calls up to three times using progressive back-off delays of 5, 15, and 30 seconds, automatically detecting transient errors including HTTP 429 rate limits and server outages before escalating permanent failures to the user.

The huggingface/ml-intern repository protects its agent loop with a deterministic retry mechanism designed specifically for temporary LLM service interruptions. This implementation distinguishes between transient network issues and permanent configuration errors, ensuring that rate limiting and momentary capacity problems do not terminate user sessions. Understanding this retry logic for transient LLM errors and rate limiting reveals how the system maintains conversation flow despite external API instability.

Core Retry Configuration

The retry behavior is governed by two module-level constants defined in agent/core/agent_loop.py. _MAX_LLM_RETRIES is set to 3 (line 119), establishing a hard ceiling on retry attempts before the system propagates the error to the user.

# From agent/core/agent_loop.py, lines 119-121

_MAX_LLM_RETRIES = 3
_LLM_RETRY_DELAYS = [5, 15, 30]

_LLM_RETRY_DELAYS (lines 120-121) defines the progressive back-off schedule in seconds. The first retry waits 5 seconds, the second waits 15 seconds, and the final attempt waits 30 seconds. This linear-to-exponential ramp prevents aggressive re-polling while giving temporary outages time to resolve.

Detecting Transient Errors

Before any retry executes, the system validates that the exception is recoverable. The _is_transient_error function (lines 124-138) inspects error messages for specific patterns indicating temporary conditions:

• Timeout indicators: timeout, timed out
• Rate limiting: HTTP status 429, rate limit, rate_limit
• Server errors: HTTP statuses 503, 502, 500
• Capacity issues: overloaded, capacity, connection reset/refused, EOF, broken pipe

Only errors matching these patterns trigger the retry path. Configuration errors or authentication failures bypass this logic and fail immediately.

Retry Implementation in LLM Calls

The retry loop is embedded directly into the LLM invocation helpers, ensuring consistent behavior across streaming and non-streaming execution paths.

Streaming Requests (_call_llm_streaming)

In _call_llm_streaming (lines 1000-1035), the implementation wraps the API call in a for loop iterating over the maximum retry count. When an exception occurs, the code invokes _is_transient_error to classify the failure:

# Simplified representation of the retry block in _call_llm_streaming

for _llm_attempt in range(_MAX_LLM_RETRIES):
    try:
        # ... LLM API call ...

        break  # Success exits the loop

    except Exception as exc:
        if _llm_attempt < _MAX_LLM_RETRIES - 1 and _is_transient_error(exc):
            delay = _LLM_RETRY_DELAYS[_llm_attempt]
            # Log warning and emit tool_log event to UI

            await session.send_event("tool_log", f"LLM connection error, retrying in {delay}s...")
            await asyncio.sleep(delay)
            continue
        raise  # Non-transient or final attempt re-raises

Non-Streaming Requests (_call_llm_non_streaming)

_call_llm_non_streaming (lines 992-1024) mirrors this exact structure for synchronous-style invocations. Both paths share the same retry budget and delay schedule, ensuring uniform resilience regardless of how the response is consumed.

Handling Configuration Errors Before Retry Budget

ML Intern implements an effort-healing shortcut that executes before the transient retry logic consumes an attempt. Located around lines 1015-1022 in _call_llm_streaming, this block checks for effort-configuration errors (e.g., requesting "thinking" mode on a model that does not support it).

If the error matches an effort-configuration problem, the system heals the request parameters and retries once without incrementing the _llm_attempt counter. This preserves the three-attempt budget for genuine transient errors while automatically fixing incompatible parameter combinations.

User Feedback and Event Emission

During each retry cycle, the agent emits a tool_log event via the session.send_event method. This broadcasts the back-off countdown to the frontend, allowing the UI to display messages like "LLM connection error, retrying in 15s..." to keep users informed of temporary delays.

Warning-level logs are also generated server-side, creating an audit trail of which specific error patterns triggered retries.

Code Examples

Example 1: Direct Streaming Invocation

When using the internal streaming helper, retry logic executes automatically:

from agent.core.agent_loop import _call_llm_streaming

# Existing Session object required for context and event emission

messages = [{"role": "user", "content": "Explain transformer architecture."}]
tools = []
llm_params = {
    "model": "gpt-4o-mini",
    "temperature": 0.7,
}

# Automatically handles up to 3 retries with 5s/15s/30s delays

result = await _call_llm_streaming(session, messages, tools, llm_params)

Example 2: Manual Retry Implementation

For custom tooling outside the agent loop, reproduce the delay schedule exactly:

import asyncio

_MAX_LLM_RETRIES = 3
_RETRY_DELAYS = [5, 15, 30]

async def resilient_llm_call(session, messages, tools, llm_params):
    for attempt in range(_MAX_LLM_RETRIES):
        try:
            response = await acompletion(
                messages=messages, 
                tools=tools, 
                **llm_params
            )
            return response
        except Exception as exc:
            if attempt < _MAX_LLM_RETRIES - 1 and _is_transient_error(exc):
                delay = _RETRY_DELAYS[attempt]
                print(f"Transient error ({exc}); retrying in {delay}s …")
                await asyncio.sleep(delay)
            else:
                raise

Summary

• Three-attempt limit: ML Intern caps retries at _MAX_LLM_RETRIES = 3 before surfacing errors to users.
• Progressive back-off: Delays escalate from 5 to 15 to 30 seconds to allow service recovery.
• Pattern-based detection: The _is_transient_error function filters for HTTP 429, 503, timeouts, and capacity-related strings.
• Dual coverage: Both _call_llm_streaming and _call_llm_non_streaming implement identical retry blocks.
• Effort healing: Configuration errors are fixed and retried once without consuming the transient error budget.

Frequently Asked Questions

How does ML Intern distinguish between rate limits and permanent errors?

The system uses the _is_transient_error function in agent/core/agent_loop.py (lines 124-138) to scan exception messages for specific substrings including 429, rate limit, 503, timeout, and capacity. Messages containing these patterns trigger the retry logic, while authentication or validation failures propagate immediately.

Can I configure the number of retries or delay durations?

Currently, _MAX_LLM_RETRIES and _LLM_RETRY_DELAYS are hardcoded constants at lines 119-121 of agent/core/agent_loop.py. Modifying these values requires editing the source code, as the repository does not expose runtime configuration for these parameters.

What happens if all three retry attempts fail?

After exhausting the retry budget in _call_llm_streaming or _call_llm_non_streaming, the original exception is re-raised and converted into a user-facing error event. The agent terminates the current operation and reports the failure through the session's event system rather than hanging indefinitely.

Does effort healing count against the three retry attempts?

No. According to the implementation at lines 1015-1022, when an effort-configuration error is detected, the system heals the request parameters and retries once without incrementing the _llm_attempt counter. This ensures that fixing incompatible model parameters does not reduce the budget available for handling transient network errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}