# How Prompt Caching Optimizes Token Usage in ML Intern

> Discover how ml intern uses prompt caching to slash Anthropic API costs by up to 90%. Learn to optimize token usage with the with_prompt_caching helper for efficient ML applications.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: internals
- Published: 2026-04-24

---

**ML Intern reduces Anthropic API costs by up to 90% on static prompt sections using the `with_prompt_caching` helper to inject ephemeral cache breakpoints into tool definitions and system messages.**

ML Intern implements an intelligent **prompt caching** strategy to minimize redundant token costs when communicating with large language models. The repository leverages LiteLLM's Anthropic integration to cache static portions of prompts—specifically tool specifications and system messages—across multiple turns in a conversation. This optimization is transparently handled by a single helper function that modifies request payloads only for Anthropic models while remaining a no-op for other providers like OpenAI or Hugging Face inference endpoints.

## How Prompt Caching Works in ML Intern

When interacting with Anthropic models through LiteLLM, ML Intern utilizes **cache control blocks** to mark specific portions of a prompt as cacheable for five minutes. The system targets two high-volume static sections that would otherwise be re-sent on every request:

- **Tool block**: The complete list of function definitions available to the agent
- **System message**: The rendered system prompt containing instructions and context

By applying `{"cache_control": {"type": "ephemeral"}}` to these sections, subsequent requests within the TTL window trigger a cache hit. Anthropic charges only a cache-read fee—approximately 10% of the standard input token price—rather than processing the full token count again. This covers roughly 4,000 to 5,000 static tokens that remain consistent across conversation turns.

## The Prompt Caching Implementation

The core logic resides in [`agent/core/prompt_caching.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/prompt_caching.py) within the `with_prompt_caching` function. This helper checks if the target model is Anthropic-based; if not, it returns the original messages and tools unchanged.

For Anthropic models, the function performs two specific mutations:

1. **Tool caching**: It clones the tools list and appends a cache control block to the last tool specification.
2. **System message caching**: If the first message has role `system`, it wraps the string content into a structured text block containing the cache control directive.

The function returns fresh copies of both lists, ensuring that shared state in the `ContextManager` remains immutable while still benefiting from cache savings.

### Source Code Structure

The implementation guarantees thread safety and state isolation by creating new list instances rather than mutating inputs in place:

```python
def with_prompt_caching(messages: list[Any],
                        tools: list[dict] | None,
                        model_name: str | None) -> tuple[list[Any], list[dict] | None]:
    # Bail out for non‑Anthropic models

    if not model_name or "anthropic" not in model_name:
        return messages, tools

    # ── Cache the tool block ──

    if tools:
        new_tools = list(tools)
        last = dict(new_tools[-1])
        last["cache_control"] = {"type": "ephemeral"}   # ← cache breakpoint

        new_tools[-1] = last
        tools = new_tools

    # ── Cache the system prompt ──

    if messages:
        first = messages[0]
        role = first.get("role") if isinstance(first, dict) else getattr(first, "role", None)
        if role == "system":
            content = (first.get("content")
                       if isinstance(first, dict)
                       else getattr(first, "content", None))
            if isinstance(content, str) and content:
                cached_block = [{
                    "type": "text",
                    "text": content,
                    "cache_control": {"type": "ephemeral"},
                }]
                new_first = {"role": "system", "content": cached_block}
                messages = [new_first] + list(messages[1:])
    return messages, tools

```

## Where Prompt Caching Is Applied

The `with_prompt_caching` helper is invoked immediately before every LLM call throughout the ML Intern codebase, ensuring consistent optimization across all agent operations.

**Research sub-agent**: In [`agent/tools/research_tool.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/research_tool.py) (lines 53-55), the helper processes messages and tool specifications before passing them to `acompletion`:

```python
_msgs, _tools = with_prompt_caching(messages,
                                    tool_specs if tool_specs else None,
                                    llm_params.get("model"))
response = await acompletion(messages=_msgs, tools=_tools, ...)

```

**Main agent loop**: The primary interaction loop in [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py) (lines 300-301) applies the same transformation to each turn's payload.

**Context manager**: When persisting conversation snapshots in [`agent/context_manager/manager.py`](https://github.com/huggingface/ml-intern/blob/main/agent/context_manager/manager.py) (line 119), the helper ensures cached blocks are present in stored context.

## Practical Usage Examples

### Manual Application of Prompt Caching

To manually apply caching to a request payload:

```python
from agent.core.prompt_caching import with_prompt_caching

# Original request payload

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain prompt caching."},
]
tools = [
    {"type": "function", "function": {"name": "search", "description": "..."}},
    {"type": "function", "function": {"name": "summarize", "description": "..."}},
]

# Apply caching for an Anthropic model

cached_messages, cached_tools = with_prompt_caching(
    messages,
    tools,
    model_name="anthropic/claude-3-5-sonnet-20240620"
)

print(cached_messages[0])   # system message now contains cache_control

print(cached_tools[-1])     # last tool now contains cache_control

```

This transforms the system message into a structured text block with `cache_control` and appends the cache directive to the final tool entry, preparing the request for reduced-cost cached processing.

### Integration in Agent Workflows

Higher-level agents integrate caching transparently by wrapping their LLM calls:

```python
async def call_llm(messages, tools, model):
    # Insert caching if the model supports it

    msgs, t = with_prompt_caching(messages, tools, model)
    # Forward the possibly‑modified payload to LiteLLM

    return await acompletion(messages=msgs, tools=t, model=model)

```

All downstream agents calling this function automatically benefit from token savings without implementing provider-specific logic.

## Summary

- ML Intern implements **prompt caching** through the `with_prompt_caching` helper in [`agent/core/prompt_caching.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/prompt_caching.py) to reduce Anthropic API costs.
- The system caches **tool definitions** (last tool) and **system messages** (first message) using Anthropic's ephemeral cache control blocks.
- Cached sections incur only ~10% of standard input token costs when reused within the 5-minute TTL window.
- The implementation is a **no-op for non-Anthropic models**, ensuring compatibility with OpenAI and Hugging Face routers without code changes.
- The helper is invoked before every LLM call in the research tool, main agent loop, and context manager to ensure continuous optimization.

## Frequently Asked Questions

### Which models support prompt caching in ML Intern?

Only Anthropic models trigger the caching logic. The `with_prompt_caching` function checks if the model name contains "anthropic" and returns unmodified payloads for all other providers, including OpenAI and Hugging Face inference endpoints.

### How much does prompt caching reduce API costs?

According to Anthropic's pricing structure, cache hits cost approximately 10% of the standard input token price. For ML Intern's typical workloads, this saves roughly 4,000 to 5,000 tokens per request that would otherwise be re-processed at full price, resulting in substantial cost reductions during multi-turn conversations.

### Why is the cache control applied to the last tool and first message?

Anthropic's API requires cache breakpoints at specific positions to maximize hit rates. Placing the cache control on the **last tool** ensures the complete tool schema is cached as a single block, while applying it to the **first system message** captures the static instructions. These locations represent the largest unchanging portions of typical ML Intern requests.

### Does prompt caching affect conversation state or history?

No. The `with_prompt_caching` function creates fresh copies of the messages and tools lists before modification. This design prevents mutation of shared state in the `ContextManager`, preserving the integrity of conversation history while still enabling cache benefits for identical content across separate API calls.