# How KV Cache Persistence Works in ds4-server for Conversation State

> Learn how KV cache persistence in ds4-server saves conversation token-text to durable checkpoint files. Resume chats instantly without recomputing prompts using SHA-1 hashed filenames and smart eviction.

- Repository: [Salvatore Sanfilippo/ds4](https://github.com/antirez/ds4)
- Tags: internals
- Published: 2026-08-05

---

**KV cache persistence in ds4-server stores conversation token-text as durable checkpoint files, enabling the model to resume chats without recomputing prompts by using SHA-1 hashed filenames and a budget-aware eviction policy.**

The **ds4-server** from the antirez/ds4 repository implements a sophisticated key-value cache system that preserves conversation state across server restarts. This **KV cache persistence** mechanism eliminates redundant computation for repeated prompts by storing token sequences on disk and reloading them on demand. The implementation spans multiple source files, with the core logic residing in [`ds4_kvstore.c`](https://github.com/antirez/ds4/blob/main/ds4_kvstore.c) and integration points in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c).

## Understanding the KV Cache Architecture

The persistence layer treats each conversation prefix as an independent checkpoint. When a client sends a prompt, the server checks whether a matching checkpoint exists before running inference. If found, the server restores the precomputed token sequence directly into the engine's buffer.

This design provides two critical benefits:

- **Latency reduction** – Avoids re-tokenizing and re-processing prompts that have been seen before
- **Resource efficiency** – Shares computation across sessions and server restarts

The checkpoint system operates transparently to clients while exposing configuration options through command-line flags defined in [`ds4_help.c`](https://github.com/antirez/ds4/blob/main/ds4_help.c).

## Initializing the KV Store

Server startup triggers the KV store initialization sequence. The `ds4_kvstore_open(kc, dir, budget_mb, reject_different_quant, opt)` function in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) (approximately line 9360) establishes the persistence directory and allocates the management structures.

```bash
ds4-server \
  --model my-model.gguf \
  --kv-dir /var/ds4/kvcache \
  --kv-budget 4096      # 4 GiB budget

```

The `budget_mb` parameter enforces a hard limit on total disk usage. When specified with `--kv-budget`, the server will not consume more than the allocated megabytes across all checkpoint files.

Default behavior is controlled by `ds4_kvstore_default_options` in [`ds4_kvstore.c`](https://github.com/antirez/ds4/blob/main/ds4_kvstore.c) (lines 64-71), which specifies:

- **Token thresholds** – Minimum and maximum token counts worth caching
- **Alignment requirements** – Memory layout constraints for efficient loading
- **Eviction heuristics** – Weighting factors for retention decisions

These defaults can be overridden via the `--kv-options` flag for specialized workloads.

## Checkpoint File Format and Storage

Each checkpoint follows a strict binary format defined in [`ds4_kvstore.c`](https://github.com/antirez/ds4/blob/main/ds4_kvstore.c). The structure begins with three magic bytes "KVC" and a version number:

```c
// Magic constants from ds4_kvstore.c (lines 25-29)
#define KV_CACHE_MAGIC0 'K'
#define KV_CACHE_MAGIC1 'V'
#define KV_CACHE_MAGIC2 'C'
#define KV_CACHE_VERSION 1
#define KV_CACHE_PAYLOAD_ABI 1

```

The 20-byte header contains:

| Field | Purpose |
|-------|---------|
| Model ID | Prevents loading checkpoints from incompatible models |
| Quantization bits | Ensures format consistency with current engine |
| Reason code | Classification of why the checkpoint was created |
| Flags | Boolean options affecting interpretation |
| Token count | Length of the stored sequence |
| Hit count | Usage statistics for eviction decisions |
| Context size | Maximum context window when stored |

Following the header, the **payload** holds the raw token-text. The **payload ABI** field enables future format changes without breaking existing cache directories—older versions are simply rejected and regenerated.

### Writing Checkpoints

After each generation completes, `ds4_kvstore_store_len(kc, tokens)` in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) (lines 9397-9404) persists the state. This function:

1. Serializes the header with current metadata
2. Writes the token-text payload
3. Invokes registered trailer hooks for protocol extensions

```c
// Simplified storage flow from custom code
ds4_kvstore_options opts = ds4_kvstore_default_options();
ds4_kvstore_open(&kc, "/tmp/kv", 1024, false, opts);

/* After generating tokens */
size_t token_len = ds4_engine_token_len(engine);
ds4_kvstore_store_len(&kc, token_len);   // persists conversation state

```

## Loading and Restoring Checkpoints

The retrieval mechanism relies on **deterministic hashing** of prompt text. The server computes a SHA-1 digest using `ds4_kvstore_sha1_bytes_hex` and constructs a filename `<sha>.kv`. This guarantees that identical prompts map to identical files regardless of session boundaries.

The load operation in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) (lines 9625-9630) follows this path:

```c
ds4_kvstore_load_result lr = {0};
ds4_kvstore_trailer_hooks hooks = kv_cache_tool_map_hooks(server, NULL);

int loaded = ds4_kvstore_try_load_text(
                &server->kv,
                server->engine,
                slot->session,    // chat session identifier
                &lr,
                &hooks);

if (loaded) {
    // Tokens restored to engine; continue generation from cached state
}

```

The validation sequence checks:

- Magic bytes and version compatibility
- Model ID match against current engine
- Quantization bit consistency
- Payload ABI compatibility

Failed validation causes silent rejection, triggering fresh computation and a new checkpoint write.

## Budget-Aware Eviction Policy

The KV store enforces its size limit through `ds4_kvstore_evict` in [`ds4_kvstore.c`](https://github.com/antirez/ds4/blob/main/ds4_kvstore.c) (lines 560-585). When the total directory size exceeds `budget_mb`, the system selects removal candidates using a composite **eviction score**:

```

eviction_score = f(hit_count, age, continuation_factor)

```

The **continuation factors** (`KV_CACHE_CONTINUED_PREFIX_*` constants) reward prefixes that frequently begin multi-turn conversations. This heuristic recognizes that conversation starters have higher reuse potential than arbitrary mid-dialogue states.

Eviction proceeds until the cache returns below budget, typically removing:

- Old checkpoints with zero or low hit counts
- Large checkpoints with poor hit-to-size ratios
- Non-continuation prefixes when alternatives exist

## Trailer Hooks for Protocol Extensions

The core KV format remains stable across ds4 versions, but protocols may need to attach metadata. The **trailer hook system** enables this extension without format changes.

In [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) (lines 9448-9468), the function `kv_cache_tool_map_hooks` creates a hook table mapping tool IDs to trailer data. This table travels with checkpoint operations:

```c
ds4_kvstore_trailer_hooks hooks = kv_cache_tool_map_hooks(server, NULL);

// Passed to both store and load operations
ds4_kvstore_store_len(&kc, tokens);        // hooks serialize tool metadata
ds4_kvstore_try_load_text(&kc, ..., &hooks); // hooks deserialize and apply

```

Use cases include:

- **Tool calling** – Preserving which tools were available when the checkpoint was created
- **Authentication** – Attaching session credentials for multi-tenant deployments
- **Auditing** – Embedding provenance information for compliance requirements

## Graceful Shutdown and Durability

Server termination calls `ds4_kvstore_close(kc)` in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) (lines 9368-9370). This routine:

- Flushes pending writes to disk
- Releases file handles and memory
- Preserves all checkpoint files for subsequent restarts

The on-disk cache survives process crashes and system reboots. On restart, `ds4_kvstore_open` discovers existing files and rebuilds its in-memory index without re-reading full payloads, achieving near-instant availability of cached conversations.

## Configuration Reference

| Flag | Default | Description |
|------|---------|-------------|
| `--kv-dir` | (none) | Directory path for checkpoint storage |
| `--kv-budget` | 0 (unlimited) | Maximum cache size in megabytes |
| `--kv-options` | (defaults) | JSON override for KV store options |

Environment-specific tuning typically sets `--kv-budget` to 50-100% of available disk space, depending on conversation diversity and retention requirements.

## Summary

- **KV cache persistence** in ds4-server stores conversation token-text as durable checkpoint files, eliminating redundant computation across sessions and restarts
- **SHA-1 hashing** of prompts provides deterministic filename generation, enabling automatic cache hits for identical inputs
- **20-byte binary headers** with magic bytes, version fields, and validation checksums ensure format safety and model compatibility
- **Budget-aware eviction** combines hit frequency, age, and continuation heuristics to retain the most valuable checkpoints within configured limits
- **Trailer hooks** allow protocol-specific extensions without modifying the core checkpoint format
- **Graceful shutdown** preserves all state to disk, with fast restoration on subsequent server startup

## Frequently Asked Questions

### What happens if the KV cache directory fills up?

When the total size exceeds `--kv-budget`, the eviction algorithm in `ds4_kvstore_evict` automatically removes the lowest-scoring checkpoints until the cache returns below limit. The scoring function prioritizes recent, frequently accessed, and conversation-starting prefixes. Evicted files are deleted from disk and their entries removed from the in-memory index.

### Can checkpoints from different model versions be loaded?

No. The `reject_different_quant` parameter and model ID field in the header prevent cross-model loading. When `ds4_kvstore_try_load_text` detects a mismatch in quantization bits or model identifier, it rejects the checkpoint and the server regenerates it with the current engine configuration.

### How does the server handle concurrent access to checkpoint files?

The implementation relies on atomic file operations at the OS level. Writes use temporary files followed by atomic rename operations, ensuring that readers never observe partial checkpoints. The in-memory index in `ds4_kvstore` protects its data structures with appropriate locking during concurrent lookups and updates.

### What is the performance impact of KV persistence?

Checkpoint storage adds minimal latency—typically milliseconds for sequences under 10K tokens—because writes are sequential and batched. Loading provides significant speedups, often 10-100x faster than recomputation for large contexts, depending on model size and hardware. The tradeoff is disk space consumption controlled by `--kv-budget`.