How to Handle KV Cache Mismatches and Session Recovery in ds4

When ds4 detects a KV cache mismatch during checkpoint loading, it invalidates the live session via ds4_session_invalidate(), removes the corrupt cache file, and falls back to a fresh pre-fill to ensure inference integrity.

The ds4 inference engine persists attention states and token histories in a disk-based KV cache to enable fast session resumption. When incoming prompts diverge from cached checkpoints or stored headers become corrupted, understanding how to handle KV cache mismatches and session recovery is critical for maintaining reliable production deployments. The system implements a strict verification pipeline and centralized invalidation mechanism to prevent corrupted states from propagating into live inference.

How ds4 Detects KV Cache Mismatches

The engine validates cached checkpoints through a four-stage verification pipeline implemented in ds4_kvstore.c. Any failure during these checks triggers immediate session invalidation and cache eviction.

Header Validation

The verification process begins in ds4_kvstore_read_header(), which checks magic bytes (KV_CACHE_MAGIC0-2), the cache version (KV_CACHE_VERSION), and payload ABI compatibility (KV_CACHE_PAYLOAD_ABI) at lines 20-24. If the header structure does not match the current binary's expectations, the load aborts before reading any KV data, preventing ABI mismatches from corrupting the session.

Text-Prefix Hash Verification

After header validation, ds4 computes a SHA-1 hash of the stored text prefix using ds4_kvstore_sha1_bytes_hex() and compares it against the hash derived from the beginning of the incoming prompt (lines 12622-12632). A mismatch indicates the cached content does not correspond to the current request, forcing the engine to treat the checkpoint as stale.

Byte-Prefix Matching

Even with matching hashes, ds4 performs a byte-level prefix check via ds4_kvstore_byte_prefix_match() (lines 6670-6672) to ensure the cached text is a literal prefix of the incoming prompt. If the prompt diverges earlier than the cached sequence, the checkpoint is rejected and the session is invalidated.

Payload Deserialization Checks

Finally, ds4_session_load_payload() streams the raw KV ring and model-specific state from disk, verifying that the bytes read exactly match the payload size recorded in the header (lines 12484-12512). Discrepancies here indicate file truncation or corruption, triggering immediate invalidation and file removal.

Session Invalidation Mechanisms

When validation fails, ds4 triggers ds4_session_invalidate() to clear internal state flags such as checkpoint_valid (lines 65508-65520 in ds4.c). This function serves as the central safety mechanism across three primary code paths.

KV-Cache Loading Failures

In ds4_kvstore_try_load_text() (lines 12415-12435), failures during header reading, hash comparison, or prefix matching immediately call ds4_session_invalidate(s->session) followed by unlink(path) to delete the corrupt file from the cache directory.

Session Rewrite Failures

During request processing in ds4_server.c (lines 3885-3894), if ds4_session_rewrite_from_common() returns DS4_SESSION_REWRITE_REBUILD_NEEDED, the server discards the live KV state and attempts to load an older checkpoint. If none are usable, the session is invalidated and the engine performs a full pre-fill.

Explicit User Actions

Both ds4_cli.c (line 1372) and ds4_agent.c (line 4273) invoke ds4_session_invalidate() when users clear chat history or when decoding steps abort due to runtime errors. This ensures stale KV states do not persist across user operations or error conditions.

Implementing Session Recovery

The recovery flow follows a strict hierarchy: invalidate the current session, attempt to locate a newer compatible checkpoint, and fall back to full pre-fill if necessary. The following examples demonstrate loading cached checkpoints and handling mismatches programmatically.

/* --------------------------------------------------------------
   Example: Load a cached KV checkpoint and recover from a mismatch.
   -------------------------------------------------------------- */
#include "ds4.h"
#include "ds4_kvstore.h"

void run_prompt(ds4_engine *engine, const char *prompt) {
    ds4_session *session = NULL;
    ds4_session_create(&session, engine, 4096);   // allocate a session

    // Try to recover from KV cache
    ds4_tokens effective = {0};
    ds4_kvstore_load_result result = {0};
    int loaded = ds4_kvstore_try_load_text(
        &engine->kvstore,          // KV store singleton
        engine,                    // engine for tokenization
        session,                   // live session to populate
        prompt,                    // incoming prompt text
        &effective,                // will contain the prompt tokens
        &result,                   // optional diagnostics
        NULL,                      // no extra trailer hooks
        false);                    // no response-protocol flag

    if (loaded > 0) {
        /* Cache hit – resume from the stored state. */
        printf("Cache hit: resumed at %d tokens\n", loaded);
        ds4_session_sync(session, &effective, NULL, 0);
    } else {
        /* Cache miss or mismatch – fallback to full prefill. */
        ds4_tokens full = {0};
        ds4_tokenize_rendered_chat(engine, prompt, &full);
        ds4_session_sync(session, &full, NULL, 0);
    }

    /* ... continue generation ... */

    ds4_session_free(session);
}

/* --------------------------------------------------------------
   Example: Explicitly invalidate a session after a runtime error.
   -------------------------------------------------------------- */
void decode_loop(ds4_session *s) {
    int token = ds4_session_argmax(s);
    if (token < 0) {
        /* Something went wrong – discard the corrupted KV state. */
        ds4_session_invalidate(s);
        return;
    }
    /* Normal processing ... */
}

Key Source Files and Functions

File	Purpose	Important Symbols
ds4_kvstore.c	Core implementation of on-disk KV cache, loading, verification, and eviction	ds4_kvstore_try_load_text, ds4_kvstore_read_header, ds4_kvstore_byte_prefix_match, ds4_kvstore_sha1_bytes_hex
ds4.c	Session lifecycle, invalidation logic, and low-level graph handling	ds4_session_invalidate, ds4_session_sync, ds4_session_load_payload
ds4.h	Public API declarations for session management and KV cache operations	ds4_session_invalidate, ds4_session_rewrite_from_common, ds4_session_sync
ds4_server.c	Server-side request handling and cache integration	Cache loading error paths, rewrite failure handling
ds4_cli.c / ds4_agent.c	User-facing CLI and agent interfaces	Calls to ds4_session_invalidate after history truncation or decode failures

Summary

• Four-stage verification (header, SHA-1 hash, byte-prefix match, payload size) ensures cache integrity before resuming sessions.
• ds4_session_invalidate() serves as the central safety mechanism, clearing checkpoint_valid flags and forcing clean-state recovery.
• Automatic cleanup removes corrupt cache files via unlink(path) to prevent future false hits.
• Hierarchical recovery attempts newer checkpoints after invalidation, falling back to full pre-fill only when necessary.
• Explicit invalidation in CLI and agent code ensures user actions and runtime errors do not leave stale KV states.

Frequently Asked Questions

What triggers a KV cache mismatch in ds4?

A KV cache mismatch occurs when ds4_kvstore_try_load_text() detects header corruption, SHA-1 hash mismatches between stored and incoming text, failed byte-prefix comparisons, or payload size discrepancies during deserialization. Any of these validation failures trigger ds4_session_invalidate() and removal of the corrupt file.

How does ds4 recover from a corrupted KV checkpoint?

The engine first invalidates the live session to clear internal state flags, then attempts to load an older compatible checkpoint. If no valid checkpoints exist, ds4 falls back to a full pre-fill of the prompt through ds4_session_sync(), rebuilding the KV state from scratch at the cost of additional computation.

What is the role of ds4_session_invalidate()?

ds4_session_invalidate() is the central invalidation routine defined in ds4.c (lines 65508-65520) that marks a session as invalid by clearing flags like checkpoint_valid. This ensures subsequent calls to ds4_session_sync() recognize the need for a clean-state rebuild rather than attempting to reuse corrupted KV memory.

Where is the cache verification logic implemented?

The core verification pipeline resides in ds4_kvstore.c, including header validation (ds4_kvstore_read_header), cryptographic hash checks (ds4_kvstore_sha1_bytes_hex), prefix matching (ds4_kvstore_byte_prefix_match), and payload loading (ds4_session_load_payload). Session invalidation triggers are distributed across ds4.c, ds4_server.c, ds4_cli.c, and ds4_agent.c.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}