Implementing Context Engineering Tools for Enhanced Claude Interactions: A Technical Guide

Anthropic provides three API primitives—compaction, tool-result clearing, and structured memory—that enable developers to maintain long-running Claude agents processing over 300,000 tokens without exhausting the 1 million token context window.

The anthropics/claude-cookbooks repository demonstrates production-ready patterns for managing Claude’s context window in complex agent workflows. This guide analyzes the source code behind a synthetic biology research agent that ingests a 320,000-token corpus, showing you how to implement conversation compaction, tool-result clearing, and persistent memory tools using the actual API configurations and Python handlers from the repository.

Core Context Management Primitives

The Anthropic API exposes three beta features specifically designed for context engineering. According to the implementation in tool_use/context_engineering/context_engineering_tools.ipynb, these primitives can be combined to create agents that never hit the token ceiling while retaining critical reasoning.

Conversation Compaction (compact_20260112)

Compaction summarizes the entire conversation history into a condensed representation when the context approaches a specified threshold. When triggered—typically at 150,000 tokens—the API replaces the full message history with a summary that preserves essential findings while discarding low-signal tokens.

The configuration accepts a trigger token count and custom instructions for what to preserve:

compaction_cfg = {
    "compact_20260112": {
        "trigger": 150_000,
        "instructions": "Summarise all findings; keep organism names and numeric results.",
    }
}

As implemented in the research agent example, this primitive reduces the context window from approximately 335,000 tokens down to roughly 70,000 tokens while maintaining the scientific conclusions necessary for subsequent analysis.

Tool-Result Clearing (clear_tool_uses_20250919)

Tool-result clearing replaces large tool_result content—such as file reads or API responses—with tiny placeholders while preserving the tool_use block to maintain provenance. This is distinct from compaction because it operates on individual tool results rather than the entire conversation.

The primitive supports selective retention via the keep parameter (number of recent results to preserve) and exclude_tools (tool names that should never be cleared):

clearing_cfg = {
    "clear_tool_uses_20250919": {
        "trigger": 100_000,
        "keep": 3,
        "exclude_tools": ["memory"],
    }
}

In the claude-cookbooks demonstration, enabling clearing alone keeps the research agent’s context under 180,000 tokens despite reading eight large review documents totaling 320,000 tokens.

Persistent Memory Tool (memory_20250818)

The memory primitive moves structured data outside the context window entirely, enabling multi-session persistence. Unlike the automatic compaction and clearing features, memory is invoked explicitly as a tool call that CRUDs files in a sandboxed directory.

The MemoryToolHandler class in tool_use/memory_tool.py implements six commands (view, create, str_replace, insert, delete, rename) with path traversal protection:

def _validate_path(self, path: str) -> Path:
    if not path.startswith("/memories"):
        raise ValueError("Path must start with /memories")
    relative_path = path[len("/memories"):].lstrip("/")
    full_path = (self.memory_root / relative_path).resolve()
    full_path.relative_to(self.memory_root.resolve())
    return full_path

This validation ensures agents can only access files within the designated memory root, preventing directory traversal attacks while allowing arbitrary note-taking across sessions.

Agent Architecture Deep Dive

The research agent architecture separates concerns between data provision, execution orchestration, and state management. Understanding these components is essential for implementing context engineering in production applications.

Synthetic Corpus (research_corpus.py)

The agent operates against a deterministic, in-memory corpus defined in tool_use/context_engineering/research_corpus.py. This dictionary maps virtual file paths to content strings totaling approximately 320,000 tokens:

CORPUS: dict[str, str] = {
    "/research/celegans_review.md": """# Model Organism Review: C. elegans ...""",

    "/research/drosophila_review.md": """# Model Organism Review: Drosophila ...""",

    # … 6 additional organisms …

}

Using synthetic data eliminates external I/O latency and guarantees reproducible token counts for testing context management strategies.

Execution Loop (context_engineering_tools.ipynb)

The run_research_session function orchestrates the agent loop, handling API calls with optional context management configurations:

def run_research_session(
    initial_prompt: str,
    *,
    context_management: dict | None = None,
    betas: list[str] | None = None,
    memory_handler=None,
    max_turns: int = 12,
) -> SessionResult:
    tools = list(BASE_TOOLS)
    if memory_handler is not None:
        tools.append(MEMORY_TOOL_SPEC)
    
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=4096,
        system=SYSTEM_PROMPT,
        messages=messages,
        tools=tools,
        context_management=context_management,
        betas=betas,
    )

The function tracks token_trajectory and events to visualize when compaction or clearing triggers fire during the session.

Practical Implementation Examples

The following patterns demonstrate how to configure the Anthropic API to handle increasingly complex context management scenarios.

Baseline Without Context Management

Running the agent without optimization quickly exhausts the available context:

from context_engineering_tools import run_research_session

baseline = run_research_session(
    RESEARCH_TASK,
    label="baseline",
    max_turns=12,
)

Result: The agent reaches approximately 335,000 tokens in five turns, storing every file read and reasoning step in the context window.

Implementing Tool-Result Clearing

To reduce memory pressure, configure the clear_tool_uses_20250919 beta identifier:

clearing_cfg = {
    "clear_tool_uses_20250919": {
        "trigger": 100_000,
        "keep": 3,
        "exclude_tools": ["memory"],
    }
}

clearing = run_research_session(
    RESEARCH_TASK,
    label="clearing",
    context_management=clearing_cfg,
)

Result: Earlier read_file results are replaced with placeholders, keeping the context under 180,000 tokens while preserving the record that files were accessed.

Combining Clearing and Compaction

For maximum efficiency, merge both configurations to handle both large tool outputs and accumulated reasoning:

compaction_cfg = {
    "compact_20260112": {
        "trigger": 150_000,
        "instructions": "Summarise all findings; keep organism names and numeric results.",
    }
}
combined_cfg = {**clearing_cfg, **compaction_cfg}

combined = run_research_session(
    RESEARCH_TASK,
    label="combined",
    context_management=combined_cfg,
)

Result: After turn three, the context compacts to approximately 70,000 tokens. Subsequent tool calls stay below 200,000 tokens despite continuing the full research synthesis.

Persisting State Across Sessions

Use the MemoryToolHandler to maintain notes between distinct API sessions:

from tool_use.memory_tool import MemoryToolHandler

mem_handler = MemoryToolHandler()  # Writes to ./memory_storage/memories

# Session 1: Record findings

session1 = run_research_session(RESEARCH_TASK, memory_handler=mem_handler)
mem_handler.execute(
    command="create",
    path="/memories/batch1_notes.txt",
    file_text="\n".join(session1.notes),
)

# Session 2: Retrieve previous context

session2 = run_research_session(
    "Continue analysis using previous notes",
    memory_handler=mem_handler,
)

The memory tool writes to the filesystem, allowing arbitrarily large notes to persist indefinitely without consuming API tokens.

Summary

• Conversation compaction (compact_20260112) triggers at a token threshold to summarize the entire message history, reducing context size by 50-80% while preserving reasoning.
• Tool-result clearing (clear_tool_uses_20250919) replaces large tool outputs with placeholders, configured via keep and exclude_tools parameters to protect recent or critical results.
• Memory tools (memory_20250818) implement external state persistence through the MemoryToolHandler class, supporting CRUD operations validated against path traversal.
• The claude-cookbooks implementation demonstrates processing 320,000 tokens of synthetic research data while maintaining context under 75,000 tokens using combined primitives.

Frequently Asked Questions

What is the difference between compaction and tool-result clearing in the Anthropic API?

Compaction operates on the entire conversation history, generating a summary that replaces all previous messages when the token count exceeds the trigger threshold. Tool-result clearing specifically targets individual tool_result blocks, replacing the content of old tool calls with placeholders while keeping the tool_use record intact. Use compaction when the conversation itself grows too large; use clearing when tool outputs (like file reads or API responses) dominate the token count but the conversation history remains manageable.

How do I prevent the memory tool from accessing files outside its designated directory?

The MemoryToolHandler in tool_use/memory_tool.py implements path validation via the _validate_path method, which checks that all paths start with /memories and uses pathlib.Path.resolve() with relative_to() to ensure the resolved path remains within the memory_root. Any attempt to traverse to parent directories raises a ValueError before file operations execute.

Can I use context management features with any Claude model?

Context engineering primitives like compact_20260112 and clear_tool_uses_20250919 are beta features that must be explicitly enabled via the betas or context_management parameters in the API request. According to the claude-cookbooks source, these features work with Claude Sonnet and Claude Opus models where the context window limit is a concern, but you should verify current availability in the Anthropic documentation as beta identifiers are versioned with date suffixes (e.g., _20260112).

What happens if a compaction trigger fires during a complex multi-step reasoning task?

When the compact_20260112 trigger fires, the API immediately summarizes the conversation up to that point according to your instructions parameter. As demonstrated in the research agent notebook, providing specific instructions like "keep organism names and numeric results" ensures that critical in-progress reasoning is preserved in the summary. The agent continues from the compacted state, maintaining awareness of previous conclusions despite the truncated history.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}