# KV Cache Eviction Strategy in MLA for Long Context Sequences: A Deep Dive into DeepSeek-V3

> Discover DeepSeek-V3's KV cache eviction strategy using MLA to manage long context sequences. Learn how the implicit sliding-window automatically handles memory without explicit deletion. Optimized for efficiency.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: deep-dive
- Published: 2026-02-26

---

**The Multi-Head Latent Attention (MLA) layer in DeepSeek-V3 implements an implicit sliding-window eviction strategy that automatically discards older tokens once the sequence exceeds `max_seq_len`, keeping memory usage bounded without explicit deletion logic.**

The **KV cache eviction strategy in MLA** is critical for handling long context sequences in large language models. In the DeepSeek-V3 architecture, the MLA layer pre-allocates fixed-size buffers during model instantiation and manages cache entries through positional indexing rather than dynamic memory allocation. This design ensures predictable memory consumption during autoregressive generation while maintaining attention over the most recent tokens.

## How MLA Manages KV Cache Allocation in DeepSeek-V3

The MLA implementation in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) allocates cache buffers when the `Transformer` class is instantiated, creating distinct storage strategies for naïve and optimized execution paths.

### Cache Buffer Initialization

During `__init__`, the model registers fixed-size tensors as buffers using `torch.zeros` with dimensions derived from `ModelArgs`. These buffers persist across forward passes but are marked with `persistent=False` to exclude them from state dict serialization:

```python

# From inference/model.py lines 440-445

self.register_buffer(
    "k_cache",
    torch.zeros(args.max_batch_size, args.max_seq_len,
                self.n_local_heads, self.qk_head_dim),
    persistent=False)
self.register_buffer(
    "v_cache",
    torch.zeros(args.max_batch_size, args.max_seq_len,
                self.n_local_heads, self.v_head_dim),
    persistent=False)

```

### Naïve vs. Optimized Cache Structures

The MLA layer supports two cache configurations:

- **Naïve implementation**: Maintains separate `k_cache` and `v_cache` tensors with full head dimensions (`qk_head_dim` and `v_head_dim` respectively)
- **Optimized implementation**: Uses a compressed `kv_cache` combined with a positional embedding cache `pe_cache` to reduce memory footprint (see [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) lines 443-444)

Both implementations rely on the same **sliding-window eviction strategy** based on positional indexing.

## The Sliding-Window KV Cache Eviction Strategy

Rather than implementing explicit cache management algorithms like LRU or FIFO, MLA uses an implicit eviction mechanism tied to tensor indexing boundaries.

### Write-Back Logic and Position Tracking

During each forward pass in the `forward` method, the layer writes newly computed keys, values, and positional embeddings into the cache at indices determined by `start_pos` and `end_pos`:

```python

# From inference/model.py lines 484-485

self.kv_cache[:bsz, start_pos:end_pos] = self.kv_norm(kv)
self.pe_cache[:bsz, start_pos:end_pos] = k_pe.squeeze(2)

```

The `start_pos` parameter represents the current position in the sequence, while `end_pos = start_pos + seqlen` marks the boundary of new tokens.

### Implicit Eviction Beyond max_seq_len

The **KV cache eviction strategy** operates automatically when processing sequences longer than `max_seq_len`:

1. **Bounded buffers**: The cache tensors are pre-allocated with fixed dimensions `[max_batch_size, max_seq_len, ...]`
2. **Index-based filtering**: When `start_pos` exceeds `max_seq_len`, the slice `start_pos:end_pos` falls outside the tensor bounds
3. **Silent discarding**: PyTorch slice assignment silently ignores out-of-bounds indices, effectively discarding older tokens without explicit deletion logic

This creates a **sliding window** that retains only the most recent `max_seq_len` tokens. As the generation loop in [`inference/generate.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/generate.py) (lines 60-71) increments `prev_pos` (passed as `start_pos`) with each token generation, older cache entries automatically fall out of scope.

## Configuring Cache Limits in ModelArgs

The eviction boundary is controlled through `ModelArgs` in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (line 56), which defines `max_seq_len` with a default value of 16384 (4096×4):

```python
@dataclass
class ModelArgs:
    max_seq_len: int = 16384  # Default cache window size

    max_batch_size: int = 8
    # ... additional arguments

```

Reducing `max_seq_len` decreases memory consumption but shortens the effective context window before eviction occurs.

## Code Examples

### Setting Up Constrained KV Cache

Configure a model with a limited cache size to enforce earlier eviction:

```python
from inference.model import ModelArgs, Transformer

# Configure for 8192-token cache limit (half the default)

args = ModelArgs(
    max_seq_len=8192,
    max_batch_size=4,
    n_layers=32,
    n_heads=32,
    dim=4096,
    qk_rope_head_dim=64,
    qk_nope_head_dim=64,
    kv_lora_rank=128,
    q_lora_rank=0,
    dtype="bfloat16"
)

model = Transformer(args)

# The KV cache buffers are now fixed at [4, 8192, ...]

```

### Autoregressive Generation with Sliding Window

Demonstrate how the cache evicts old tokens during long generation:

```python
from inference.generate import generate
import torch

# Create input tokens

tokens = torch.randint(0, args.vocab_size, (1, 10), device="cuda")

# Generate 2000 tokens with a cache limit of 8192

# Once generation exceeds 8192 tokens, older cache entries are evicted

output = generate(
    model,
    [tokens[0].tolist()],
    max_new_tokens=2000,
    eos_id=args.eos_token_id,
    temperature=0.9
)

# The model only attends to the most recent 8192 positions

# Earlier positions were silently discarded from kv_cache and pe_cache

```

## Summary

- **Fixed-size allocation**: The MLA layer pre-allocates KV cache buffers at model initialization using `max_seq_len` dimensions, ensuring predictable memory usage.
- **Implicit sliding window**: The **KV cache eviction strategy** relies on tensor indexing rather than explicit deletion; when `start_pos` exceeds `max_seq_len`, out-of-bounds writes silently fail, discarding older tokens.
- **Position-based management**: The generation loop increments `start_pos` (as `prev_pos`) with each token, automatically shifting the effective cache window to retain only the most recent tokens.
- **Configuration control**: The eviction threshold is set via `ModelArgs.max_seq_len` in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py), allowing trade-offs between context length and memory consumption.

## Frequently Asked Questions

### How does the MLA layer determine which tokens to evict from the KV cache?

The MLA layer does not implement explicit token selection logic for eviction. Instead, it relies on the bounds of the pre-allocated cache tensors. When the `start_pos` parameter exceeds `max_seq_len`, the slice assignment `self.kv_cache[:bsz, start_pos:end_pos]` targets indices outside the tensor dimensions. PyTorch silently ignores these out-of-bounds operations, effectively preventing storage of older tokens and creating an implicit **sliding-window eviction** that retains only the most recent `max_seq_len` positions.

### What is the difference between the naïve and optimized KV cache implementations in DeepSeek-V3?

The naïve implementation allocates separate `k_cache` and `v_cache` buffers with full head dimensions (`qk_head_dim` and `v_head_dim`), consuming more memory but maintaining straightforward tensor operations. The optimized implementation uses a compressed `kv_cache` combined with a separate `pe_cache` for positional embeddings, reducing memory footprint through latent attention compression. Both implementations, however, employ the identical **KV cache eviction strategy** based on fixed-size buffers and positional indexing limits defined by `max_seq_len`.

### Can the KV cache size be adjusted without retraining the model?

Yes, the KV cache size can be modified at inference time by changing the `max_seq_len` parameter in `ModelArgs` before model instantiation. This parameter controls the second dimension of the cache buffers allocated in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py). Reducing `max_seq_len` decreases GPU memory consumption but shortens the effective context window, causing earlier eviction of historical tokens. Increasing `max_seq_len` (within available memory constraints) extends the retention window without requiring model retraining, as the eviction strategy remains purely a function of buffer geometry and indexing arithmetic.