# DeepSeek-V3 MLA KV Cache Memory Footprint for 128K Context Length

> Discover the DeepSeek-V3 MLA KV cache memory footprint for 128K context. Learn about BFloat16 and FP8 quantization options impacting GPU memory usage, perfect for optimizing large context models.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: performance
- Published: 2026-02-26

---

**For a 128,000-token context window, DeepSeek-V3's Multi-Head Latent Attention (MLA) consumes approximately 125 MiB of GPU memory for the KV cache in BFloat16 mode, or 62.5 MiB when using FP8 quantization.**

DeepSeek-V3 achieves efficient long-context inference through a compressed KV cache mechanism implemented in its Multi-Head Latent Attention layers. Unlike standard multi-head attention that stores full key and value heads for every token, the MLA architecture in the deepseek-ai/DeepSeek-V3 repository utilizes a low-rank projection with a fixed dimension of 512, drastically reducing the memory footprint required to support 128K context lengths.

## MLA KV Cache Tensor Structure

The KV cache in DeepSeek-V3 is stored as a compressed tensor with the exact shape:

```

(batch_size, max_seq_len, kv_lora_rank)

```

**Key architectural constants** from the inference source code:

- **`kv_lora_rank`**: Defined as **512** in [`inference/configs/config_236B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_236B.json) (lines 15–17). This compressed rank replaces the traditional `(num_heads × head_dim)` storage, which typically requires thousands of floating-point values per token.
- **Buffer initialization**: The cache is registered as a non-persistent buffer in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 440–445):

```python
self.register_buffer(
    "kv_cache",
    torch.zeros(args.max_batch_size, args.max_seq_len, self.kv_lora_rank),
    persistent=False,
)

```

During the forward pass, this buffer is populated with the normalized KV projection (`self.kv_norm(kv)`) and retained in the model's compute dtype for the duration of the sequence generation.

## Memory Calculation for 128K Context Length

The total memory footprint scales linearly with sequence length and depends on the precision mode. For a single batch with **128,000 tokens**:

| Data Type | Bytes per Element | Formula | Total Memory |
|-----------|-------------------|---------|--------------|
| **BFloat16** (`bf16`) | 2 | 128,000 × 512 × 2 | **125 MiB** |
| **FP8** | 1 | 128,000 × 512 × 1 | **62.5 MiB** |

**Byte calculation details:**
- **128,000** tokens (context length)
- **512** dimensions (`kv_lora_rank`)
- **2** bytes per element for BF16 (default inference mode)
- **1** byte per element for FP8 (optional quantization)

This represents a **32× reduction** compared to standard attention mechanisms, which would require caching 8,192+ dimensions per token (e.g., 128 heads × 64 dimensions) for equivalent model capacity.

## Precision Modes and GPU Efficiency

DeepSeek-V3 supports two compute dtypes for the KV cache, controlled via the `fp8` inference flag:

- **BFloat16**: The default 2-byte precision providing full dynamic range for the 512-dimensional latent vectors.
- **FP8**: 1-byte quantization that halves the memory footprint, enabling 128K context inference with only **62.5 MiB** of cache memory per layer.

You can programmatically calculate the cache requirement for any context length using the architectural constants from the source:

```python
def mla_kv_cache_mb(seq_len, kv_rank=512, dtype="bf16"):
    """Calculate DeepSeek-V3 MLA KV cache size in MiB."""
    bytes_per_elem = 2 if dtype == "bf16" else 1
    total_bytes = seq_len * kv_rank * bytes_per_elem
    return total_bytes / (1024 * 1024)

# 128K context calculations

print(f"BF16: {mla_kv_cache_mb(128000, dtype='bf16'):.1f} MiB")
print(f"FP8:  {mla_kv_cache_mb(128000, dtype='fp8'):.1f} MiB")

```

```text
Expected output:
BF16: 125.0 MiB
FP8:  62.5 MiB

```

The `persistent=False` parameter in the buffer registration ensures these temporary inference tensors are excluded from model state dicts, preventing unnecessary disk I/O during checkpoint operations.

## Summary

- **DeepSeek-V3 MLA** compresses per-token KV representations into a fixed **512-dimensional** vector (`kv_lora_rank`), independent of the total number of attention heads.
- **128K context length** requires **125 MiB** per layer in BF16 or **62.5 MiB** in FP8, as defined in [`inference/configs/config_236B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_236B.json) and allocated in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py).
- The cache tensor shape is `(batch_size, max_seq_len, 512)`, using 2 bytes per element by default.
- Memory increases linearly: each additional 1,000 tokens consumes approximately **0.98 MiB** (BF16) or **0.49 MiB** (FP8).

## Frequently Asked Questions

### How does MLA achieve such a small memory footprint for 128K contexts?

Multi-Head Latent Attention decouples the key and value calculations from the full head dimension through a low-rank compression matrix. Instead of storing 8,192+ floating-point values per token (typical for 128-head models), MLA caches only **512** latent values, achieving a **16:1 compression ratio** while preserving attention quality via the `kv_lora_rank` projection defined in the model configuration.

### What is the `kv_lora_rank` parameter and where is it configured?

The **`kv_lora_rank`** is a hyperparameter set to **512** in [`inference/configs/config_236B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_236B.json) (lines 15–17). It defines the width of the compressed latent space for keys and values. This value is loaded during model initialization in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) and directly determines the third dimension of the `kv_cache` tensor, balancing memory efficiency against representational capacity.

### Can the KV cache size be reduced further for 128K inference?

Yes. Switching from BFloat16 (2 bytes) to **FP8** (1 byte) quantization halves the cache footprint from **125 MiB to 62.5 MiB** for 128K tokens. This is supported natively in the DeepSeek-V3 inference engine via dtype flags and is particularly effective for batch generation scenarios where cache memory becomes the bottleneck.

### Where in the codebase is the 128K context KV cache allocated?

The allocation occurs in **[`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py)** at lines 440–445, where `torch.zeros(args.max_batch_size, args.max_seq_len, self.kv_lora_rank)` creates the buffer. When `args.max_seq_len` is set to 128000, this line allocates the **125 MiB** (BF16) tensor on the GPU device, sized specifically for the compressed MLA representation rather than full attention heads.