How ML Intern Implements Context Compaction for Long Conversations

ML Intern automatically compacts conversation history by summarizing older messages when token usage exceeds 90% of the model's limit, preserving the system prompt, original task, and recent turns while replacing the middle section with a concise summary.

The huggingface/ml-intern repository implements an intelligent context management system that enables extended agent interactions without hitting language model token limits. At the core of this system is context compaction, a process that condenses aging conversation history into terse summaries when memory thresholds are approached.

The ContextManager Architecture

ML Intern delegates all conversation state management to the ContextManager class defined in agent/context_manager/manager.py. This class maintains a rolling chat history and tracks cumulative token consumption via the running_context_usage attribute.

Every time the LLM generates a response, the add_message() method captures the usage.total_tokens from the API response and updates the running counter. This continuous monitoring enables proactive memory management before the model's context window is exhausted.

How Automatic Context Compaction Works

When the accumulated token count approaches the model's maximum capacity, the manager executes a multi-step compaction pipeline:

Token Usage Tracking

The foundation of context compaction relies on accurate token accounting. In agent/context_manager/manager.py, the add_message() method stores the latest token count at lines 17-22:

def add_message(self, message: Message, total_tokens: int | None = None) -> None:
    self.items.append(message)
    if total_tokens is not None:
        self.running_context_usage = total_tokens

This running_context_usage value serves as the primary indicator for triggering compaction logic.

The Compaction Threshold

The manager defines a hard threshold at 90% of the model's token ceiling to ensure proactive compaction. The implementation at lines 33-41 calculates this boundary:

self.compaction_threshold = int(self.model_max_tokens * 0.9)

The needs_compaction property (lines 42-46) evaluates whether compaction is required by checking if current usage exceeds this threshold while ensuring at least one message exists in the history:

@property
def needs_compaction(self) -> bool:
    return (
        self.running_context_usage > self.compaction_threshold
        and len(self.items) > 0
    )

Triggering the Compaction Process

Each conversational turn, the agent loop invokes _compact_and_notify() in agent/core/agent_loop.py (lines 34-47). This function checks needs_compaction and calls ContextManager.compact() when necessary:

async def _compact_and_notify(session: Session) -> None:
    cm = session.context_manager
    old_token_count = cm.running_context_usage
    
    if cm.needs_compaction:
        await cm.compact(
            model_name=session.config.model_name,
            tool_specs=session.tool_router.get_tool_specs_for_llm(),
            hf_token=session.hf_token,
        )

Selecting Messages to Summarize

The compact() method implements a surgical preservation strategy to maintain conversation coherence. According to lines 56-78 in manager.py, the algorithm categorizes messages into three groups:

• Protected: The system prompt and the first user message (containing the original task instruction) remain untouched
• Recent: The tail of untouched_messages (defaulting to 5) stays intact to preserve immediate dialogue context
• Summarized: Everything between the first user message and the recent tail becomes the summarization target

This selection ensures the agent retains the overarching goal and recent context while condensing older procedural exchanges.

Generating the Summary

The selected messages are processed by summarize_messages() (lines 98-130), which executes a single-turn LLM call using the _COMPACT_PROMPT. The summarization operates with a strict token budget defined as compact_size (10% of the model's maximum tokens):

summary, _ = await summarize_messages(
    messages_to_summarize,
    model_name,
    hf_token,
    max_tokens=self.compact_size,
    tool_specs=tool_specs,
    prompt=_COMPACT_PROMPT,
)

The prompt specifically instructs the model to preserve decisions, rationales, and essential context, ensuring that downstream reasoning retains access to critical information despite the compression.

Rebuilding the Conversation History

After generating the summary, the manager reconstructs the message history at lines 92-104:

# Rebuild history: system + first user + summary + recent

self.items = ([system_msg] if system_msg else []) + \
             [first_user_msg, summarized_msg] + recent_messages

# Recompute token usage

self.running_context_usage = token_counter(
    model=model_name,
    messages=[m.model_dump() for m in self.items],
)

The token counter recalculates the actual usage post-compaction, ensuring accurate tracking for subsequent turns.

Notification and Event Emission

If the compaction successfully reduces token count, the agent loop emits a compacted Server-Sent Event (SSE) via lines 48-58 in agent_loop.py:

if cm.running_context_usage != old_token_count:
    await session.send_event(Event(
        event_type="compacted",
        data={
            "old_tokens": old_token_count,
            "new_tokens": cm.running_context_usage
        },
    ))

This notification enables the frontend to display a "context compacted" banner, informing users that history compression has occurred.

Manual vs. Automatic Compaction

While the agent loop handles compaction automatically, ML Intern also exposes explicit control via the /compact/{session_id} endpoint defined in backend/routes/agent.py (lines 72-78):

@router.post("/compact/{session_id}")
async def compact_session(session_id: str):
    await session_manager.compact(session_id)
    return {"status": "compaction queued"}

This endpoint utilizes the same ContextManager.compact() method as the automatic system, ensuring identical behavior whether compaction is triggered by the 90% threshold or explicit user request. The frontend can invoke this via standard HTTP POST requests when users click a compact button in the web interface.

Implementation Details and Code Examples

Core Compaction Routine

The following simplified implementation illustrates the logic found in agent/context_manager/manager.py:

async def compact(self, model_name: str, tool_specs: list[dict] | None = None,
                 hf_token: str | None = None) -> None:
    if not self.needs_compaction:
        return

    # Identify protected messages

    system_msg = self.items[0] if self.items[0].role == "system" else None
    first_user_idx = next(i for i, m in enumerate(self.items) if m.role == "user")
    first_user_msg = self.items[first_user_idx]
    
    # Preserve recent context

    recent = self.items[-self.untouched_messages:]
    
    # Select middle section for summarization

    to_summarize = self.items[first_user_idx + 1 : -self.untouched_messages]
    
    # Generate summary

    summary, _ = await summarize_messages(
        to_summarize, model_name, hf_token,
        max_tokens=self.compact_size,
        tool_specs=tool_specs,
        prompt=_COMPACT_PROMPT,
    )
    summarized_msg = Message(role="assistant", content=summary)
    
    # Reconstruct conversation

    self.items = ([system_msg] if system_msg else []) + \
                 [first_user_msg, summarized_msg] + recent
    
    # Update token count

    self.running_context_usage = token_counter(
        model=model_name,
        messages=[m.model_dump() for m in self.items],
    )

Automatic Compaction Check

The agent integration pattern from agent/core/agent_loop.py demonstrates how compaction fits into the conversation loop:

async def _compact_and_notify(session: Session) -> None:
    cm = session.context_manager
    old = cm.running_context_usage
    
    await cm.compact(
        model_name=session.config.model_name,
        tool_specs=session.tool_router.get_tool_specs_for_llm(),
        hf_token=session.hf_token,
    )
    
    if cm.running_context_usage < old:
        await session.send_event(Event(
            event_type="compacted",
            data={"old_tokens": old, "new_tokens": cm.running_context_usage},
        ))

Summary

• ML Intern uses the ContextManager class in agent/context_manager/manager.py to monitor token usage continuously via running_context_usage.
• Automatic compaction triggers when token count exceeds 90% of the model's maximum capacity, as determined by the needs_compaction property.
• Surgical preservation keeps the system prompt, first user message, and 5 most recent messages intact while summarizing the intervening conversation.
• Summarization uses 10% of the model's token budget to generate concise summaries that preserve decisions and rationales via summarize_messages().
• Event notifications inform the frontend of successful compactions through SSE events emitted by _compact_and_notify() in agent/core/agent_loop.py.
• Manual override is available via the /compact/{session_id} endpoint in backend/routes/agent.py, using identical logic to automatic compaction.

Frequently Asked Questions

What triggers automatic context compaction in ML Intern?

Automatic context compaction triggers when ContextManager.running_context_usage exceeds 90% of the model's maximum token limit. This threshold is defined in the compaction_threshold property at lines 33-41 of agent/context_manager/manager.py. The system checks this condition after every LLM turn via the needs_compaction property, ensuring proactive compression before the context window overflows.

Which messages are preserved during context compaction?

The compaction algorithm preserves three specific segments: the system prompt (if present), the first user message (containing the original task), and the 5 most recent messages (configurable via untouched_messages). Everything between the first user message and this recent tail is summarized and replaced. This selection strategy maintains the agent's understanding of the core objective while keeping immediate conversational context intact.

How does ML Intern's summarization maintain conversation coherence?

The summarize_messages() function utilizes a specialized _COMPACT_PROMPT that explicitly instructs the LLM to preserve decisions, rationales, and essential context while condensing the conversation. By allocating 10% of the model's token budget to the summary, the system ensures sufficient space for the LLM to capture high-level narrative elements rather than merely truncating messages, allowing the agent to retain storyline continuity despite compression.

Can users manually trigger context compaction?

Yes, users can manually initiate compaction through the /compact/{session_id} HTTP endpoint defined in backend/routes/agent.py (lines 72-78). This endpoint queues a COMPACT operation that executes the same ContextManager.compact() method used by the automatic system. The frontend typically exposes this functionality via a dedicated button that sends a POST request, causing the agent to immediately summarize eligible messages regardless of the current token usage percentage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}