# Context Engineering in AI Agents: KV Cache and Prompt Engineering Best Practices

> Master context engineering for AI agents. Learn KV cache and prompt engineering best practices to boost performance and cut costs.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: best-practices
- Published: 2026-07-30

---

**Implementing stable system prompts and ordered tool definitions can achieve over 90% KV-cache hit rates, reducing Time-to-First-Token by approximately 30% and lowering API costs by up to 20%.**

Context engineering shapes the messages an LLM-based agent receives to maximize computational reuse across turns. This guide examines concrete techniques from the `bojieli/ai-agent-book` repository that demonstrate how to optimize **context engineering in AI agents** through strategic KV-cache utilization and prompt stability patterns.

## How KV-Cache Reduces Latency and Cost

**KV-cache** (key-value cache) is an OpenAI-style optimization that stores the hidden-state for a stable token prefix. When subsequent requests share the exact same prompt prefix, the model skips recomputing those tokens, dramatically cutting **Time-to-First-Token (TTFT)**.

In [`chapter2/kv-cache/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/kv-cache/agent.py), the repository implements a `KVCacheAgent` class that demonstrates this mechanism through various operational modes. The cache only works when the *exact* token sequence of the prefix is reused; any deviation forces a full recompute. When cached tokens are billed at a discounted rate (typically 10% of standard pricing), this translates to direct cost savings alongside performance gains.

## Critical Factors That Invalidate KV-Cache

Understanding what breaks the cache is essential for effective context engineering. The repository deliberately implements several anti-pattern modes to illustrate these failure points.

### Dynamic System Prompts

Adding per-turn mutations like timestamps or session IDs to the system prompt invalidates the cache. In `KVCacheMode.DYNAMIC_SYSTEM`, the code injects a dynamic timestamp into the system message (lines 27-34 in [`agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agent.py)), causing a complete cache miss even when the core task remains unchanged.

### Tool Ordering Variations

The ReAct engine parses tools by position; shuffling them yields a different token sequence each turn. The `KVCacheMode.SHUFFLED_TOOLS` mode randomly permutes `tool_defs` before each request, demonstrating how unstable tool ordering destroys prefix consistency.

### Per-Turn Dynamic Data

Appending user-specific fields such as credit balances or profile information to the system prompt changes the token string each iteration. `KVCacheMode.DYNAMIC_PROFILE` illustrates this by adding a recomputed "credits" line every turn, effectively flushing the cached state.

### Sliding-Window Context Management

While dropping older messages reduces token length, it also discards cached prefix tokens. The `KVCacheMode` implementation includes a *sliding_window* mode that keeps only the most recent N messages, preventing KV-cache reuse across turns despite maintaining semantic continuity.

### Unstructured Text Formats

Switching from structured JSON to free-form text often introduces variable tokenization (whitespace, punctuation changes) between runs. `KVCacheMode.TEXT_FORMAT` demonstrates this instability by using raw-text prompts that fail to maintain bit-for-bit identical prefixes.

## Implementing Cache-Friendly Agent Architecture

To maximize cache utilization, the repository recommends the `KVCacheMode.CORRECT` pattern. This approach maintains an immutable system prompt containing static task descriptions, stable tool definitions, and consistent formatting.

The following example demonstrates the correct implementation:

```python
from agent import KVCacheAgent, KVCacheMode

agent = KVCacheAgent(
    api_key="YOUR_KIMI_OR_OPENROUTER_KEY",
    mode=KVCacheMode.CORRECT,            # ← immutable system prompt

    model="kimi-k2.6",
    root_dir="/path/to/repo",
    verbose=True,
)

task = "Analyze and summarize the projects in chapter1 and chapter2."
result = agent.execute_task(task, max_iterations=30)
print(result["metrics"].cache_hits, "cache hits")

```

Key architectural principles from [`chapter2/kv-cache/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/kv-cache/main.py) include:

1. **Define a single, immutable system prompt** that contains task descriptions and tool definitions
2. **Place dynamic data in user messages** rather than mutating the system context
3. **Maintain stable tool ordering** across all agent iterations
4. **Reuse identical token prefixes** (e.g., the same "You are an AI assistant..." paragraph) across sessions

## Measuring Cache Performance and Cost Impact

The repository provides utilities to quantify optimization effectiveness. The `run_comparison` function in [`chapter2/kv-cache/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/kv-cache/main.py) executes tasks across all cache modes simultaneously, while `run_report` generates performance analyses from stored JSON results.

To compare all modes and visualize the cost of breaking the cache:

```python
from chapter2.kv_cache.main import run_comparison

run_comparison(
    api_key="YOUR_KEY",
    task="Summarize the repository's chapters.",
    cache_price_ratio=0.1,   # cached tokens cost 10% of normal tokens

)

```

The helper function `_billable_tokens` computes adjusted costs using the `cache_price_ratio` parameter (default 0.1), enabling precise ROI calculations for optimization efforts. After running experiments, generate offline reports without API calls:

```python
from chapter2.kv_cache.main import run_report

run_report(inputs=["result_*.json", "comparison_*.json"])

```

These utilities track **cached token counts** and **cache-hit rates** via the `print_comparison_table` function, providing concrete metrics to verify that engineering changes achieve the target >90% hit rate.

## Summary

- **KV-cache** stores hidden states for identical token prefixes, reducing TTFT by ~30% after the first turn
- **Stable system prompts** are mandatory for cache reuse; avoid timestamps, shuffled tools, or dynamic profile data in the system message
- **Tool definition ordering** must remain constant across iterations to maintain prefix consistency
- **Sliding-window contexts** and **unstructured text formats** inadvertently discard cached computations
- The `KVCacheMode.CORRECT` implementation in [`chapter2/kv-cache/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/kv-cache/agent.py) demonstrates the architectural pattern for sustained high cache-hit rates
- Cost savings of up to 20% are achievable when cached tokens are billed at reduced rates

## Frequently Asked Questions

### What is the difference between KV-cache and prompt-cache?

**KV-cache** operates at the token level, storing intermediate key-value states for partial prompts, while **prompt-cache** stores entire prompt-completion pairs. KV-cache remains effective when prompt prefixes stay stable even if suffixes change, making it more flexible for multi-turn conversational agents.

### Why does tool ordering affect KV-cache performance?

LLMs process tool definitions as positional token sequences. Shuffling tool order changes the exact byte representation of the system prompt, creating a different prefix hash that the KV-cache cannot match. The repository's `KVCacheMode.SHUFFLED_TOOLS` demonstrates that random `tool_defs` permutations cause 100% cache misses.

### How much can context engineering reduce API costs?

According to the metrics implemented in [`chapter2/kv-cache/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/kv-cache/main.py), proper context engineering achieves **>90% cache-hit rates** after the initial turn. When providers bill cached tokens at 10% of standard rates (the default `cache_price_ratio` used in the repository), this yields **up to 20% total cost reduction** for multi-turn ReAct agents.

### Where should dynamic user data go if not in the system prompt?

Dynamic information such as user credits, timestamps, or session-specific variables should be placed in **user messages** rather than the system prompt. The `KVCacheMode.DYNAMIC_PROFILE` anti-pattern in the repository illustrates how adding "credits" lines to the system context breaks the cache, whereas keeping the system prompt static preserves computational reuse.