Prompt Cache Affinity in Grok2API: How It Works with Codex and Claude Code

Prompt cache affinity is a session-binding mechanism that ties a request's cached prompt to a specific backend account and egress node using a sticky store, ensuring deterministic reuse across Codex and Claude Code sessions.

The chenyme/grok2api repository implements prompt cache affinity to maintain secure, stateful connections between client sessions and backend infrastructure. This mechanism ensures that cached prompts remain isolated to specific accounts while enabling efficient reuse across multiple turns in AI coding assistants.

What Is Prompt Cache Affinity?

Prompt cache affinity refers to the architectural pattern of binding a cached prompt (or "turn") to a specific egress account and compute node for the duration of a session. Rather than treating each request as stateless, the system extracts a unique affinity key from incoming headers and uses it to route subsequent requests to the same backend resources. This guarantees that cached tokens remain available and are never shared across different user accounts or sessions.

How Codex and Claude Code Supply the Affinity Key

Client applications provide the affinity key through HTTP headers that contain structured metadata about the current conversation turn.

Codex Turn Metadata

For Codex sessions, the client sends the X-Codex-Turn-Metadata header containing a JSON payload. In backend/internal/infra/provider/cli/normalize.go, the API parses this header to extract the prompt_cache_key field, which serves as the primary affinity identifier.

// Extracted from header parsing logic
turnMeta := request.Header.Get("X-Codex-Turn-Metadata")
var meta struct {
    PromptCacheKey string `json:"prompt_cache_key"`
    WindowID       string `json:"window_id"`
}
json.Unmarshal([]byte(turnMeta), &meta)
affinityKey := meta.PromptCacheKey

Claude Code Session Headers

Claude Code follows an equivalent pattern, supplying the prompt_cache_key via a similar metadata header (e.g., X-Claude-Turn-Metadata). The parsing logic in backend/internal/infra/provider/cli/normalize.go handles both variants, ensuring the affinity key is extracted regardless of which client initiates the request.

The Sticky Store: Binding Keys to Accounts

Once extracted, the affinity key is stored in the sticky store implemented in backend/internal/infra/runtime/memory/store.go. This component maintains the mapping between cache keys and backend accounts.

The StickyStore.Bind method records the association between the affinity key, the selected account ID, and an expiration timestamp:

// Binding the affinity key to a specific account
if affinityKey != "" {
    sticky.Bind(ctx, affinityKey, accountID, time.Now(), expiresAt)
}

Subsequent requests carrying the same key trigger StickyStore.Get, which retrieves the bound account ID to ensure the cached prompt is accessed only from the original backend context:

// Retrieving the bound account for cache reuse
boundID, ok, _ := sticky.Get(ctx, affinityKey, time.Now())
if ok {
    // Route to the same account to leverage cached tokens
}

Deterministic Node Selection

The gateway selector enforces affinity at the node level through deterministic hashing. In backend/internal/application/gateway/selector.go, the selectNode function hashes the affinity key using sha256.Sum256 to choose a specific egress node from the available pool.

This deterministic selection ensures that all requests sharing the same prompt_cache_key route to the identical backend instance, maximizing cache hit rates while preventing distribution of sensitive cached data across multiple nodes.

Security Boundaries and Fallback Handling

The system implements strict security controls to prevent cross-account cache leakage. As noted in adapter_test.go, the code explicitly avoids "fabricating a random conv-id without a session key" because doing so would break server affinity and force cached_tokens to zero.

When the affinity key is missing or the bound account has expired, the request falls back to the soft identity path. This preserves basic account affinity but does not reuse cached prompts, ensuring that stale or unauthorized cache entries are never served.

Implementation Flow in Code

The following Go snippet illustrates the complete flow for a Codex request, from header extraction to sticky store binding:

package main

import (
    "crypto/sha256"
    "encoding/json"
    "time"
)

// 1. Extract metadata from Codex header
turnMeta := request.Header.Get("X-Codex-Turn-Metadata")
var meta struct {
    PromptCacheKey string `json:"prompt_cache_key"`
}
json.Unmarshal([]byte(turnMeta), &meta)

// 2. Validate and bind affinity key
affinityKey := meta.PromptCacheKey
if affinityKey != "" {
    // backend/internal/infra/runtime/memory/store.go
    sticky.Bind(ctx, affinityKey, accountID, time.Now(), expiresAt)
    
    // 3. Deterministic node selection via selector.go
    hash := sha256.Sum256([]byte(affinityKey))
    node := selectNode(pool, hash)
}

This pattern is replicated for Claude Code in backend/internal/infra/provider/cli/adapter.go, where the affinity key is attached to outbound requests before routing.

Summary

• Prompt cache affinity binds cached prompts to specific accounts and nodes using a prompt_cache_key extracted from client headers.
• Codex sends the key via X-Codex-Turn-Metadata, while Claude Code uses an equivalent header structure.
• The sticky store (backend/internal/infra/runtime/memory/store.go) persists key-to-account mappings with expiration timestamps via Bind and Get methods.
• Deterministic node selection uses SHA-256 hashing of the affinity key in backend/internal/application/gateway/selector.go to route requests consistently.
• Security safeguards prevent cache leakage by falling back to soft identity when keys are missing, as enforced in the adapter layer.

Frequently Asked Questions

What happens if the prompt_cache_key is missing from the request?

If the prompt_cache_key is absent or the sticky store entry has expired, the system falls back to the soft identity path. This maintains account-level routing but disables prompt cache reuse, ensuring cached_tokens remains at zero and preventing potential data leakage between sessions.

How does prompt cache affinity improve performance?

By binding a prompt_cache_key to a specific account and node, subsequent requests with the same key route to the same backend instance where the prompt is already cached. This eliminates redundant processing and token generation, reducing latency for multi-turn conversations in Codex and Claude Code sessions.

Is the affinity key shared between different users?

No. The sticky store explicitly binds each affinity key to a specific account ID in backend/internal/infra/runtime/memory/store.go. The system guarantees isolation by checking the bound account before reusing cached tokens, ensuring that prompts from one user are never served to another, even if they somehow obtain the same cache key.

Which files handle the header parsing for Codex?

The primary header parsing occurs in backend/internal/infra/provider/cli/normalize.go, which extracts the X-Codex-Turn-Metadata header and unmarshals the JSON to retrieve the prompt_cache_key. The adapter logic in backend/internal/infra/provider/cli/adapter.go subsequently attaches this key to outbound requests, while backend/internal/transport/http/inference/prompt_cache_test.go contains test cases validating the Codex turn behavior.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}