# How the Context Ablation Experiment Validates Essential Context Components for Agent Performance

> Discover how context ablation experiments prove tool definitions, results, reasoning, and conversation history are essential for AI agent performance. Learn what truly matters for reliable task completion.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: deep-dive
- Published: 2026-08-23

---

**The context ablation experiment systematically removes specific components from an AI agent's interaction history to prove that tool definitions, tool results, reasoning traces, and conversation history are each essential for reliable task completion.**

The context ablation experiment in Chapter 1 of the `bojieli/ai-agent-book` repository isolates the impact of individual context components on agent reasoning. By running identical tasks under five distinct `ContextMode` configurations, the experiment demonstrates that stripping any single element—from tool schemas to conversation history—degrades performance or causes complete task failure. This systematic approach, implemented in [`chapter1/context/run_experiment_1_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/run_experiment_1_1.py), provides quantitative evidence for which parts of the communication history actually matter.

## The Five Context Modes Tested

The experiment defines five distinct ablation arms in the `ContextMode` enum (located in [`chapter1/context/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/agent.py)). Each arm removes a specific layer of context while keeping others intact, allowing precise measurement of individual component contributions.

### FULL Mode: The Performance Baseline

The `FULL` mode retains all context components: tool definitions, tool results, provider-generated reasoning, and complete conversation history.

According to the source code in [`chapter1/context/run_experiment_1_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/run_experiment_1_1.py), this mode verifies three critical contracts:

- `tools_present_every_turn`: Tool definitions are included in every API request
- `history_present_after_first_turn`: Previous conversation turns are retained after the initial exchange
- `reasoning_retained_after_first_turn`: Provider reasoning traces persist in the context window

With full context available, the agent achieves `canonical_answer_correct = True`, establishing the baseline for successful task completion.

### NO_TOOL_CALLS: Removing Tool Definitions

When running in `NO_TOOL_CALLS` mode, the experiment strips all `"tools"` and `"tool_choice"` fields from API requests before sending them to the provider.

The contract verification in `evaluate_context_contract()` asserts `tools_absent_every_turn` to confirm complete removal. This ablation demonstrates catastrophic failure: the agent produces **zero tool actions** (`tool_action_count == 0`) because it literally cannot see the available functions. Without tool definitions, the agent fails to generate the correct answer, proving that **tool schemas are prerequisite for functional tool use**.

### NO_TOOL_RESULTS: Hiding Execution Feedback

The `NO_TOOL_RESULTS` mode retains tool definitions and allows the agent to invoke tools, but replaces the `"content"` of tool result messages with `"[Tool result hidden due to context mode]"`.

The contract checks confirm:

- `tool_calls_retained`: The agent can still see that it made a tool call
- `tool_results_hidden`: The actual execution output is masked

This creates a specific failure pattern: the agent enters a **repetition loop** (`has_repeated_tool_action == True`). Because it cannot see the results of its previous invocations, it calls the same tool repeatedly with identical parameters, unable to progress toward the solution. This demonstrates that **tool execution feedback is necessary for adaptive reasoning**.

### NO_REASONING: Stripping Provider Reasoning Traces

In `NO_REASONING` mode, all reasoning content generated by the provider is filtered from the conversation history before the next turn. The contract verifies that `provider_generated_reasoning` exists in the raw output but `reasoning_removed_from_history` confirms it never reaches subsequent context.

The impact manifests as **degraded canonical correctness** (`without_reasoning_degraded == True`). While the agent may still execute tool actions, the quality of decision-making deteriorates because it lacks access to its own chain-of-thought from previous steps. This proves that **reasoning traces are not debug output but functional dependencies** for multi-step tasks.

### NO_HISTORY: Eliminating Prior Conversation Turns

The `NO_HISTORY` mode implements a sliding window that retains only the static system prompt and the current user message, dropping all intermediate turns. The contract `only_static_prefix_and_user_every_turn` validates this isolation while `tools_still_present` confirms that tool definitions remain available.

Despite having access to tools, the agent exhibits **repeated tool actions** (`has_repeated_tool_action == True`) because it cannot reference previous steps or results. This mirrors the `NO_TOOL_RESULTS` failure mode but stems from different root causes—loss of temporal continuity rather than masked execution output—demonstrating that **conversation history provides essential episodic memory** for coherent task execution.

## Running the Context Ablation Experiment

You can reproduce these findings using the orchestration script provided in the repository.

### Executing the Full Five-Arm Experiment

Run all context modes against a specific provider and model:

```bash
python chapter1/context/run_experiment_1_1.py \
  --provider kimi \
  --model kimi-k3 \
  --max-iterations 5 \
  --output-dir ./validation/example_run

```

This generates an [`evidence.json`](https://github.com/bojieli/ai-agent-book/blob/main/evidence.json) file containing detailed results for each arm, including contract verification results and behavioral metrics.

### Analyzing Specific Ablation Results

Inspect individual experiment arms programmatically to verify specific hypotheses:

```python
import json
from pathlib import Path

evidence = json.loads(Path("./validation/example_run/evidence.json").read_text())
no_tool_results_arm = next(
    a for a in evidence["arms"] if a["mode"] == "no_tool_results"
)

print("Tool calls retained:", no_tool_results_arm["context_contract"]["tool_calls_retained"])
print("Tool results hidden:", no_tool_results_arm["context_contract"]["tool_results_hidden"])
print("Repeated tool actions:", no_tool_results_arm["behavior"]["has_repeated_tool_action"])

```

### Using Contract Evaluation for Custom Tests

The `evaluate_context_contract()` function is exposed for validating custom agent implementations:

```python
from chapter1.context.run_experiment_1_1 import evaluate_context_contract, ContextMode

# Assuming `turns` is a list of recorded API interactions from your custom run

contract = evaluate_context_contract(ContextMode.NO_REASONING.value, turns)
print(contract["reasoning_removed_from_history"])
print(contract["provider_generated_reasoning"])

```

## Key Source Files

Understanding the experiment architecture requires examining these specific files:

- **[`chapter1/context/run_experiment_1_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/run_experiment_1_1.py)**: Contains the `evaluate_context_contract()`, `summarize_arm()`, and `analyze()` functions that orchestrate the five-arm experiment and verify context integrity.
- **[`chapter1/context/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/agent.py)**: Implements `ContextAwareAgent` and the `ContextMode` enum that controls context filtering logic.
- **[`chapter1/context/test_experiment_1_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/test_experiment_1_1.py)**: Unit tests verifying that each ablation arm behaves according to specification.
- **[`chapter1/context/tests/test_agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter1/context/tests/test_agent.py)**: Tests for the underlying agent logic ensuring context mode implementations are correct.

## Summary

The context ablation experiment provides empirical validation that agent performance depends on the complete interaction history, not just the current prompt:

- **Tool definitions** (`NO_TOOL_CALLS`): Removal prevents all tool use, resulting in zero actions and task failure.
- **Tool results** (`NO_TOOL_RESULTS`): Hiding execution output causes repetition loops as the agent cannot learn from previous invocations.
- **Reasoning traces** (`NO_REASONING`): Stripping chain-of-thought degrades decision quality even when tools remain accessible.
- **Conversation history** (`NO_HISTORY`): Eliminating prior turns destroys episodic memory, causing the agent to repeat steps unnecessarily.
- **Full context** (`FULL`): The only configuration achieving consistent canonical correctness, proving that every component serves a functional purpose.

## Frequently Asked Questions

### What is a context ablation experiment?

A context ablation experiment is a controlled study that systematically removes specific components from an AI agent's input context to measure the impact on task performance. In the `bojieli/ai-agent-book` implementation, the experiment uses five distinct `ContextMode` settings to isolate the contributions of tool definitions, tool results, reasoning traces, and conversation history. This approach reveals which context elements are essential versus merely present.

### Which context component causes the most severe performance degradation when removed?

Removing **tool definitions** (`NO_TOOL_CALLS` mode) causes total task failure with zero tool actions, representing the most severe degradation. While other ablations cause repetition or quality degradation, the absence of tool schemas prevents the agent from invoking capabilities entirely. This demonstrates that tool definitions are the foundational prerequisite for functional agent behavior.

### How does the experiment verify that specific context components are actually removed?

The experiment uses contract verification functions defined in `evaluate_context_contract()` within [`run_experiment_1_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_experiment_1_1.py). These contracts assert specific conditions such as `tools_absent_every_turn`, `tool_results_hidden`, or `only_static_prefix_and_user_every_turn`. By programmatically inspecting the actual API request payloads sent to the provider, the experiment confirms that ablations are implemented correctly and not merely simulated.

### Can the context ablation framework test custom agent implementations?

Yes. The `evaluate_context_contract()` function accepts a `ContextMode` value and a list of recorded API turns, allowing you to validate custom agents against the same contracts. Import the function from `chapter1.context.run_experiment_1_1` and pass your agent's interaction history to verify compliance with specific context preservation requirements or to test how your agent behaves under specific ablation scenarios.