# How Multi-Head Latent Attention (MLA) Reduces KV Cache Memory in DeepSeek-V3

> Discover how DeepSeek-V3's Multi-Head Latent Attention (MLA) slashes KV cache memory by compressing keys and values reducing per-token storage from thousands to hundreds of floats.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: internals
- Published: 2026-02-26

---

**Multi-Head Latent Attention (MLA) reduces KV cache memory by compressing keys and values into a low-rank latent representation, cutting per-token storage from thousands of floats to just hundreds, while standard Multi-Head Attention (MHA) stores full per-head tensors.**

DeepSeek-V3 replaces the traditional Multi-Head Attention mechanism with Multi-Head Latent Attention to achieve substantial GPU memory savings during inference. While standard MHA maintains separate key and value tensors for every attention head, MLA stores a compressed latent cache and a small rotary positional cache, enabling context windows up to 4× longer without exhausting memory limits.

## Understanding Standard MHA KV Cache Memory

In standard Multi-Head Attention implementations, the KV cache stores full tensors for both keys and values across all heads. According to the DeepSeek-V3 inference code in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py), the naïve MHA implementation allocates:

- `k_cache` with shape `(B, S, n_local_heads, qk_head_dim)`
- `v_cache` with shape `(B, S, n_local_heads, v_head_dim)`

Where `B` is batch size, `S` is sequence length, and `n_local_heads` represents the number of attention heads. For a typical configuration with 16 heads and 192-dimensional keys, this results in **3,072 floats per token** just for the key cache alone, plus additional memory for values.

## How MLA Compresses Cache with Low-Rank Representation

Multi-Head Latent Attention fundamentally changes the caching strategy by storing a compressed **latent representation** rather than full head-specific tensors. The implementation in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 43-45) defines:

- `kv_cache` with shape `(B, S, kv_lora_rank)`
- `pe_cache` for rotary positional embeddings with shape `(B, S, qk_rope_head_dim)`

The `kv_lora_rank` parameter defaults to **512** in DeepSeek-V3 configurations, compared to the 3,072+ dimensions required by standard MHA. This represents approximately **6× reduction** in cache memory per token for the key-value storage alone.

### The Absorb Implementation and Computation Flow

MLA employs an "absorb" implementation that reconstructs full attention scores from the compressed cache during forward passes. As implemented in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 84-88):

1. The latent keys are normalized using `self.kv_norm(kv)`
2. Queries split into non-positional (`q_nope`) and rotary (`q_pe`) components
3. `q_nope` multiplies the compressed `kv_cache`
4. `q_pe` multiplies the separate `pe_cache`
5. Final attention scores combine both contributions

This approach avoids storing full per-head tensors while maintaining the expressiveness of multi-head attention through low-rank projections.

## Memory Savings Analysis

The reduction from Multi-Head Latent Attention translates to concrete GPU memory savings that enable longer context windows. Comparing the configurations found in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 75-78):

**Standard MHA memory per token:**
- Keys: 16 heads × 192 dims = 3,072 floats
- Values: 16 heads × 128 dims = 2,048 floats
- **Total: ~5,120 floats per token**

**MLA memory per token:**
- Latent KV: 512 floats (`kv_lora_rank`)
- Rotary PE: 64 floats (`qk_rope_head_dim`)
- **Total: ~576 floats per token**

This represents approximately **3–5× lower memory consumption**, allowing DeepSeek-V3 to support context lengths up to 4× longer than standard implementations while remaining within GPU memory constraints, as documented in the repository README (lines 45-49).

## Summary

Multi-Head Latent Attention reduces KV cache memory in DeepSeek-V3 through these key mechanisms:

- **Low-rank compression**: Stores a 512-dimensional latent representation instead of full per-head tensors (3,072+ dimensions)
- **Decoupled rotary cache**: Separates positional embeddings into a small `pe_cache` rather than duplicating them across heads
- **Absorb computation**: Reconstructs attention scores on-the-fly from compressed caches without materializing full key/value tensors
- **3–5× memory reduction**: Enables context window expansion up to 4× while maintaining inference efficiency

## Frequently Asked Questions

### What is the exact memory difference between MLA and standard MHA in DeepSeek-V3?

In DeepSeek-V3, standard MHA would require approximately 5,120 floats per token (3,072 for keys and 2,048 for values across 16 heads), while MLA requires only 576 floats per token (512 for the latent KV cache plus 64 for rotary positional embeddings). This represents roughly 89% reduction in KV cache memory per token.

### How does the "absorb" implementation maintain attention quality with compressed caches?

The absorb implementation in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) maintains quality by splitting queries into two components: `q_nope` (non-positional) interacts with the compressed latent cache, while `q_pe` (rotary) interacts with the separate positional cache. The final attention scores sum both contributions, preserving the expressive power of multi-head attention while operating on compressed representations.

### Where is the MLA cache configured in the DeepSeek-V3 codebase?

The MLA cache dimensions are configured in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) lines 43-45, where `kv_cache` and `pe_cache` are registered as buffers with shapes `(B, S, kv_lora_rank)` and `(B, S, qk_rope_head_dim)` respectively. The specific rank values (typically 512 for `kv_lora_rank`) are defined in the model configuration files such as [`inference/configs/config_671B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_671B.json).

### Can MLA support longer context windows than standard MHA?

Yes, MLA enables significantly longer context windows—up to 4× the original sequence length—because the 3–5× reduction in memory per token frees substantial GPU memory that would otherwise be consumed by the KV cache. This allows DeepSeek-V3 to process longer sequences while maintaining inference efficiency, as documented in the repository README (lines 45-49).