Context Engineering in AI Agents: KV Cache and Prompt Engineering Best Practices

Implementing stable system prompts and ordered tool definitions can achieve over 90% KV-cache hit rates, reducing Time-to-First-Token by approximately 30% and lowering API costs by up to 20%.

Context engineering shapes the messages an LLM-based agent receives to maximize computational reuse across turns. This guide examines concrete techniques from the bojieli/ai-agent-book repository that demonstrate how to optimize context engineering in AI agents through strategic KV-cache utilization and prompt stability patterns.

How KV-Cache Reduces Latency and Cost

KV-cache (key-value cache) is an OpenAI-style optimization that stores the hidden-state for a stable token prefix. When subsequent requests share the exact same prompt prefix, the model skips recomputing those tokens, dramatically cutting Time-to-First-Token (TTFT).

In chapter2/kv-cache/agent.py, the repository implements a KVCacheAgent class that demonstrates this mechanism through various operational modes. The cache only works when the exact token sequence of the prefix is reused; any deviation forces a full recompute. When cached tokens are billed at a discounted rate (typically 10% of standard pricing), this translates to direct cost savings alongside performance gains.

Critical Factors That Invalidate KV-Cache

Understanding what breaks the cache is essential for effective context engineering. The repository deliberately implements several anti-pattern modes to illustrate these failure points.

Dynamic System Prompts

Adding per-turn mutations like timestamps or session IDs to the system prompt invalidates the cache. In KVCacheMode.DYNAMIC_SYSTEM, the code injects a dynamic timestamp into the system message (lines 27-34 in agent.py), causing a complete cache miss even when the core task remains unchanged.

Tool Ordering Variations

The ReAct engine parses tools by position; shuffling them yields a different token sequence each turn. The KVCacheMode.SHUFFLED_TOOLS mode randomly permutes tool_defs before each request, demonstrating how unstable tool ordering destroys prefix consistency.

Per-Turn Dynamic Data

Appending user-specific fields such as credit balances or profile information to the system prompt changes the token string each iteration. KVCacheMode.DYNAMIC_PROFILE illustrates this by adding a recomputed "credits" line every turn, effectively flushing the cached state.

Sliding-Window Context Management

While dropping older messages reduces token length, it also discards cached prefix tokens. The KVCacheMode implementation includes a sliding_window mode that keeps only the most recent N messages, preventing KV-cache reuse across turns despite maintaining semantic continuity.

Unstructured Text Formats

Switching from structured JSON to free-form text often introduces variable tokenization (whitespace, punctuation changes) between runs. KVCacheMode.TEXT_FORMAT demonstrates this instability by using raw-text prompts that fail to maintain bit-for-bit identical prefixes.

Implementing Cache-Friendly Agent Architecture

To maximize cache utilization, the repository recommends the KVCacheMode.CORRECT pattern. This approach maintains an immutable system prompt containing static task descriptions, stable tool definitions, and consistent formatting.

The following example demonstrates the correct implementation:

from agent import KVCacheAgent, KVCacheMode

agent = KVCacheAgent(
    api_key="YOUR_KIMI_OR_OPENROUTER_KEY",
    mode=KVCacheMode.CORRECT,            # ← immutable system prompt

    model="kimi-k2.6",
    root_dir="/path/to/repo",
    verbose=True,
)

task = "Analyze and summarize the projects in chapter1 and chapter2."
result = agent.execute_task(task, max_iterations=30)
print(result["metrics"].cache_hits, "cache hits")

Key architectural principles from chapter2/kv-cache/main.py include:

1. Define a single, immutable system prompt that contains task descriptions and tool definitions
2. Place dynamic data in user messages rather than mutating the system context
3. Maintain stable tool ordering across all agent iterations
4. Reuse identical token prefixes (e.g., the same "You are an AI assistant..." paragraph) across sessions

Measuring Cache Performance and Cost Impact

The repository provides utilities to quantify optimization effectiveness. The run_comparison function in chapter2/kv-cache/main.py executes tasks across all cache modes simultaneously, while run_report generates performance analyses from stored JSON results.

To compare all modes and visualize the cost of breaking the cache:

from chapter2.kv_cache.main import run_comparison

run_comparison(
    api_key="YOUR_KEY",
    task="Summarize the repository's chapters.",
    cache_price_ratio=0.1,   # cached tokens cost 10% of normal tokens

)

The helper function _billable_tokens computes adjusted costs using the cache_price_ratio parameter (default 0.1), enabling precise ROI calculations for optimization efforts. After running experiments, generate offline reports without API calls:

from chapter2.kv_cache.main import run_report

run_report(inputs=["result_*.json", "comparison_*.json"])

These utilities track cached token counts and cache-hit rates via the print_comparison_table function, providing concrete metrics to verify that engineering changes achieve the target >90% hit rate.

Summary

• KV-cache stores hidden states for identical token prefixes, reducing TTFT by ~30% after the first turn
• Stable system prompts are mandatory for cache reuse; avoid timestamps, shuffled tools, or dynamic profile data in the system message
• Tool definition ordering must remain constant across iterations to maintain prefix consistency
• Sliding-window contexts and unstructured text formats inadvertently discard cached computations
• The KVCacheMode.CORRECT implementation in chapter2/kv-cache/agent.py demonstrates the architectural pattern for sustained high cache-hit rates
• Cost savings of up to 20% are achievable when cached tokens are billed at reduced rates

Frequently Asked Questions

What is the difference between KV-cache and prompt-cache?

KV-cache operates at the token level, storing intermediate key-value states for partial prompts, while prompt-cache stores entire prompt-completion pairs. KV-cache remains effective when prompt prefixes stay stable even if suffixes change, making it more flexible for multi-turn conversational agents.

Why does tool ordering affect KV-cache performance?

LLMs process tool definitions as positional token sequences. Shuffling tool order changes the exact byte representation of the system prompt, creating a different prefix hash that the KV-cache cannot match. The repository's KVCacheMode.SHUFFLED_TOOLS demonstrates that random tool_defs permutations cause 100% cache misses.

How much can context engineering reduce API costs?

According to the metrics implemented in chapter2/kv-cache/main.py, proper context engineering achieves >90% cache-hit rates after the initial turn. When providers bill cached tokens at 10% of standard rates (the default cache_price_ratio used in the repository), this yields up to 20% total cost reduction for multi-turn ReAct agents.

Where should dynamic user data go if not in the system prompt?

Dynamic information such as user credits, timestamps, or session-specific variables should be placed in user messages rather than the system prompt. The KVCacheMode.DYNAMIC_PROFILE anti-pattern in the repository illustrates how adding "credits" lines to the system context breaks the cache, whereas keeping the system prompt static preserves computational reuse.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}