# How ML Intern Recovers When the LLM Context Window Is Exceeded

> Discover how ML Intern handles LLM context window overflows. It automatically recovers by summarizing older dialogue turns and retrying calls for seamless AI interaction.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: internals
- Published: 2026-04-24

---

**ML Intern automatically recovers from context window overflows by catching the `ContextWindowExceededError` from LiteLLM, triggering a context compaction that summarizes older dialogue turns, and retrying the LLM call with the condensed history.**

When autonomous agents handle long-running conversations, exceeding the LLM's token limit is inevitable. The open-source ML Intern project (huggingface/ml-intern) solves this through an intelligent recovery system that detects overflows and compresses conversation history without user intervention. This article examines the exact source code implementation that enables seamless ML Intern context window recovery through automatic compaction and retry logic.

## Detection: Trapping ContextWindowExceededError

ML Intern intercepts token limit violations at the API layer by wrapping all LiteLLM calls in exception handlers. In [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py), both streaming and non-streaming invocations of `acompletion` reside inside `try … except ContextWindowExceededError` blocks at lines 313, 407, 860, and 864.

When the exception fires, the inner handler propagates it upward:

```python
try:
    response = await acompletion(...)
except ContextWindowExceededError:
    raise  # propagated to the outer loop

```

This re-raising allows the outer agent loop to decide how to recover rather than failing immediately.

## Compaction: Summarizing Historic Dialogue

The outer iteration loop catches the propagated exception and invokes the recovery routine. At lines 860‑864 of [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py), the code calls `_compact_and_notify(session)` and immediately continues the loop:

```python
except ContextWindowExceededError:
    # context is too big → compact then retry

    await _compact_and_notify(session)
    continue  # start the next LLM attempt with a smaller context

```

The compaction logic resides in [`agent/context_manager/manager.py`](https://github.com/huggingface/ml-intern/blob/main/agent/context_manager/manager.py) (lines 346‑399) inside the `ContextManager.compact` method. This routine decides which messages to preserve and which to summarize:

- **Preserve** the system prompt, the very first user message, and the most recent untouched messages
- **Summarize** everything in between using the prompt defined in `_COMPACT_PROMPT`
- **Rebuild** the context as system → first user → summary → recent messages
- **Verify** the new token count with `litellm.token_counter`

```python

# in manager.py (ContextManager.compact)

summary, completion_tokens = await summarize_messages(
    messages_to_summarize,
    model_name=model_name,
    hf_token=hf_token,
    max_tokens=self.compact_size,
    tool_specs=tool_specs,
    prompt=_COMPACT_PROMPT,
)

# rebuild the message list

self.items = head + [Message(role="assistant", content=summary)] + recent_messages
self.running_context_usage = token_counter(model=model_name,
                                          messages=[m.model_dump() for m in self.items])

```

## Retry with Condensed Context

After `compact` updates `session.context_manager`, the outer loop repeats the LLM call. Because `token_counter` confirmed the new count is below the model's limit (defined in [`agent/core/llm_params.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/llm_params.py)), the request succeeds and the conversation continues uninterrupted. The user never sees a "context-overflow" error.

The compaction prompt itself is cached for efficiency via [`agent/core/prompt_caching.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/prompt_caching.py), ensuring the summarization step adds minimal latency. Additionally, a manual compaction endpoint exists at `/compact/{session_id}` in [`backend/session_manager.py`](https://github.com/huggingface/ml-intern/blob/main/backend/session_manager.py), allowing the UI to trigger cleanup proactively.

## Summary

- **Detection**: All LLM calls in [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py) wrap `acompletion` in `try … except ContextWindowExceededError` blocks to catch LiteLLM overflow errors at lines 313, 407, 860, and 864.
- **Compaction**: The `ContextManager.compact` method in [`agent/context_manager/manager.py`](https://github.com/huggingface/ml-intern/blob/main/agent/context_manager/manager.py) preserves critical messages (system, first user, recent), summarizes the middle section using `_COMPACT_PROMPT`, and rebuilds the context list.
- **Verification**: The system recalculates token usage with `litellm.token_counter` before storing the compacted result in `running_context_usage`.
- **Retry**: The outer loop catches the exception, triggers `_compact_and_notify`, and retries automatically via `continue`, making recovery invisible to users.

## Frequently Asked Questions

### What error triggers ML Intern's context recovery mechanism?

ML Intern catches the `ContextWindowExceededError` thrown by the LiteLLM library when the input tokens exceed the model's ceiling. This detection happens in [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py) at multiple points (lines 313, 407, 860, and 864) where LLM calls are wrapped in exception handlers.

### Which messages survive the compaction process?

The `compact` method keeps three categories of messages intact: the system prompt, the very first user message, and the most recent untouched messages. Everything between the first user message and the recent window is fed into a summarization LLM call using the `_COMPACT_PROMPT` template, producing a condensed assistant message that replaces the middle history.

### Can developers manually trigger context compaction?

Yes. While the system handles overflows automatically, [`backend/session_manager.py`](https://github.com/huggingface/ml-intern/blob/main/backend/session_manager.py) exposes a `/compact/{session_id}` endpoint that allows the UI or external services to force a compaction proactively. This endpoint uses the same `ContextManager.compact` logic invoked during automatic recovery.

### How does ML Intern verify the compacted context fits within limits?

After rebuilding the message list as `head + [summary] + recent_messages`, the code recalculates usage with `litellm.token_counter(model=model_name, messages=[...])`. This verification ensures the token count is below the threshold defined in [`agent/core/llm_params.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/llm_params.py) before the outer loop retries the LLM call.