How CacheAligner Improves KV Cache Hit Rates by Stabilizing Prefixes in Headroom

CacheAligner is a detector-only transform that sits at the start of the Headroom pipeline, scanning system prompts for volatile tokens—such as UUIDs, timestamps, and JWTs—to warn developers about unstable KV-cache prefixes and track a stable canonical hash without mutating messages.

LLM providers like OpenAI and Anthropic cache the first N tokens of a request in a key-value (KV) cache for fast look-ups. When that prefix shifts by even a single token, the cache invalidates and the request pays full prompt cost. In the chopratejas/headroom repository, CacheAligner solves this by keeping the system-prompt prefix as stable as possible across turns.

Why Stable Prefixes Control KV-Cache Hit Rates

Providers store the initial portion of a prompt as cached KV pairs. Dynamic values injected into the system message—UUIDs, ISO-8601 timestamps, JWTs, or hex hashes—make the prefix unstable. A one-token change causes a cache miss, which strips away the provider’s read-discount for repeated prefixes. According to the Headroom source code, volatile content in the system prompt is the primary driver of unnecessary cache misses.

Detecting Volatile Tokens in System Prompts

CacheAligner scans only system messages to flag content likely to change between calls. The detection logic in headroom/transforms/cache_aligner.py relies on structural parsers rather than naive regexes to identify four common categories:

• UUIDs — detected via _is_uuid
• ISO-8601 timestamps — detected via _is_iso8601
• JWT shapes — detected via _is_jwt_shape
• Hex hashes — detected via _is_hex_hash

The helper detect_volatile_content tokenizes the message with a simple whitespace split (_split_tokens) and returns a list of VolatileFinding objects, each containing a label and a sample of the detected token.

Emitting Warnings and Alignment Metrics

When volatile tokens are found, CacheAligner logs a concise warning— for example, CacheAligner: detected volatile content in system prompt ...—and appends the same text to the warnings field of the TransformResult. This immediate feedback tells developers that the prefix is unstable and should be cleaned up, typically by moving dynamic values out of the system prompt and into the latest user turn.

The transform also exposes a quantitative health indicator through get_alignment_score. This method returns a 0–100 metric, deducting 10 points for every volatile finding. A high score above 80 correlates with a stable prefix and therefore a high probability of KV-cache hits.

Tracking Prefix Stability with Canonical Hashes

After scanning, CacheAligner builds a canonical string from all system-message contents and computes a short hash via compute_short_hash. This value is stored in the CachePrefixMetrics dataclass defined in headroom/config.py and inserted as a marker (stable_prefix_hash:<hash>) into the result.

Downstream observability can compare this hash against the previous request’s hash (_previous_prefix_hash). If the values differ, the prefix_changed flag becomes True, giving operations teams clear visibility into when cache misses are caused by real prefix drift rather than unrelated factors.

Why CacheAligner Never Mutates Messages

CacheAligner is explicitly designed as a detector-only transform. Its apply method copies the input messages using deep_copy_messages from headroom/utils.py and returns them completely unchanged, leaving transforms_applied empty. This invariant guarantees that the provider’s cache prefix is never rewritten by Headroom. Once developers remove volatile content based on the warning, every subsequent request sees exactly the same token sequence for the cached prefix, yielding consistent cache hits.

This design choice is critical because mutation could inadvertently shift tokens or change the prefix boundary, defeating the purpose of cache alignment.

Pipeline Placement and Automatic Execution

CacheAligner is automatically inserted as the first step in the default transform pipeline. In headroom/transforms/pipeline.py, the _build_default_transforms method places CacheAligner at index zero, ensuring it runs before any other transform that might alter message structure. This ordering preserves the integrity of prefix analysis against the original request.

Measuring the Impact on Token Costs

Once volatile tokens are removed from the system prompt, the provider can reuse cached key-value pairs for the unchanged prefix. Based on the PrefixFreezeConfig description in headroom/config.py, this alignment can reduce the prompt token bill by up to approximately 90 percent, reflecting the typical read-discount offered by providers like Anthropic for prefix caching.

Practical Example: Detecting Cache Misalignment

The following example demonstrates how to use CacheAligner directly and within a pipeline:

from headroom.transforms.cache_aligner import align_for_cache
from headroom.transforms.pipeline import create_pipeline

# Prepare a request with a volatile UUID in the system prompt

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant. Session ID: 550e8400-e29b-41d4-a716-446655440000"
    },
    {"role": "user", "content": "Tell me a joke."},
]

# Run CacheAligner (detect-only); messages remain unchanged

cleaned_messages, prefix_hash = align_for_cache(messages)
print("Stable prefix hash:", prefix_hash)

# Use inside the default Headroom pipeline

pipeline = create_pipeline()
result = pipeline.apply(messages, model="gpt-4")

# Check alignment health

score = pipeline.transforms[0].get_alignment_score(messages)
print("Alignment score:", score)  # 90-100 indicates a stable prefix

In this snippet, align_for_cache is a thin wrapper around CacheAligner.apply that returns the unchanged messages alongside the stable hash. The warning emitted by CacheAligner appears in both logs and the result.warnings field, enabling teams to surface cache-alignment issues to dashboards.

Summary

• CacheAligner is a detector-only transform in headroom/transforms/cache_aligner.py that scans system messages for volatile tokens.
• It detects UUIDs, ISO-8601 timestamps, JWT shapes, and hex hashes using structural parsers.
• It emits warnings and populates TransformResult.warnings so developers can remove dynamic content from the system prompt.
• It computes a canonical stable_prefix_hash, stores it in CachePrefixMetrics, and sets prefix_changed=True when the hash drifts.
• It never mutates messages, preserving the exact token sequence required for provider KV-cache hits.
• It sits at the start of the default pipeline in headroom/transforms/pipeline.py and exposes a 0–100 get_alignment_score for observability.

Frequently Asked Questions

What happens if CacheAligner finds volatile content in the system prompt?

CacheAligner logs a warning and adds the same message to the warnings field of the TransformResult. It does not modify the prompt. Developers should move detected dynamic values—such as UUIDs or timestamps—out of the system message and into the user turn to stabilize the prefix.

How does CacheAligner track whether the KV-cache prefix changed between requests?

After scanning system messages, CacheAligner builds a canonical string of their contents and computes a short hash via compute_short_hash. This hash is stored in CachePrefixMetrics and compared against _previous_prefix_hash. If the values differ, the prefix_changed flag is set to True.

Why is CacheAligner designed as a detector-only transform instead of rewriting prompts?

If CacheAligner mutated messages, it could accidentally alter the token sequence or prefix boundary in ways that conflict with the provider's caching logic. By copying messages with deep_copy_messages and returning them unchanged, CacheAligner guarantees that once volatile content is manually removed, the cached prefix remains bit-for-bit identical across requests.

Where is CacheAligner positioned in the Headroom transform pipeline?

CacheAligner is inserted as the very first transform in the default pipeline, as defined by _build_default_transforms in headroom/transforms/pipeline.py. This ensures prefix analysis runs against the original request before any other transform can restructure the messages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}