How Multi-Head Latent Attention (MLA) Reduces KV Cache Memory in DeepSeek-V3

Multi-Head Latent Attention (MLA) reduces KV cache memory by compressing keys and values into a low-rank latent representation, cutting per-token storage from thousands of floats to just hundreds, while standard Multi-Head Attention (MHA) stores full per-head tensors.

DeepSeek-V3 replaces the traditional Multi-Head Attention mechanism with Multi-Head Latent Attention to achieve substantial GPU memory savings during inference. While standard MHA maintains separate key and value tensors for every attention head, MLA stores a compressed latent cache and a small rotary positional cache, enabling context windows up to 4× longer without exhausting memory limits.

Understanding Standard MHA KV Cache Memory

In standard Multi-Head Attention implementations, the KV cache stores full tensors for both keys and values across all heads. According to the DeepSeek-V3 inference code in inference/model.py, the naïve MHA implementation allocates:

• k_cache with shape (B, S, n_local_heads, qk_head_dim)
• v_cache with shape (B, S, n_local_heads, v_head_dim)

Where B is batch size, S is sequence length, and n_local_heads represents the number of attention heads. For a typical configuration with 16 heads and 192-dimensional keys, this results in 3,072 floats per token just for the key cache alone, plus additional memory for values.

How MLA Compresses Cache with Low-Rank Representation

Multi-Head Latent Attention fundamentally changes the caching strategy by storing a compressed latent representation rather than full head-specific tensors. The implementation in inference/model.py (lines 43-45) defines:

• kv_cache with shape (B, S, kv_lora_rank)
• pe_cache for rotary positional embeddings with shape (B, S, qk_rope_head_dim)

The kv_lora_rank parameter defaults to 512 in DeepSeek-V3 configurations, compared to the 3,072+ dimensions required by standard MHA. This represents approximately 6× reduction in cache memory per token for the key-value storage alone.

The Absorb Implementation and Computation Flow

MLA employs an "absorb" implementation that reconstructs full attention scores from the compressed cache during forward passes. As implemented in inference/model.py (lines 84-88):

1. The latent keys are normalized using self.kv_norm(kv)
2. Queries split into non-positional (q_nope) and rotary (q_pe) components
3. q_nope multiplies the compressed kv_cache
4. q_pe multiplies the separate pe_cache
5. Final attention scores combine both contributions

This approach avoids storing full per-head tensors while maintaining the expressiveness of multi-head attention through low-rank projections.

Memory Savings Analysis

The reduction from Multi-Head Latent Attention translates to concrete GPU memory savings that enable longer context windows. Comparing the configurations found in inference/model.py (lines 75-78):

Standard MHA memory per token:

• Keys: 16 heads × 192 dims = 3,072 floats
• Values: 16 heads × 128 dims = 2,048 floats
• Total: ~5,120 floats per token

MLA memory per token:

• Latent KV: 512 floats (kv_lora_rank)
• Rotary PE: 64 floats (qk_rope_head_dim)
• Total: ~576 floats per token

This represents approximately 3–5× lower memory consumption, allowing DeepSeek-V3 to support context lengths up to 4× longer than standard implementations while remaining within GPU memory constraints, as documented in the repository README (lines 45-49).

Summary

Multi-Head Latent Attention reduces KV cache memory in DeepSeek-V3 through these key mechanisms:

• Low-rank compression: Stores a 512-dimensional latent representation instead of full per-head tensors (3,072+ dimensions)
• Decoupled rotary cache: Separates positional embeddings into a small pe_cache rather than duplicating them across heads
• Absorb computation: Reconstructs attention scores on-the-fly from compressed caches without materializing full key/value tensors
• 3–5× memory reduction: Enables context window expansion up to 4× while maintaining inference efficiency

Frequently Asked Questions

What is the exact memory difference between MLA and standard MHA in DeepSeek-V3?

In DeepSeek-V3, standard MHA would require approximately 5,120 floats per token (3,072 for keys and 2,048 for values across 16 heads), while MLA requires only 576 floats per token (512 for the latent KV cache plus 64 for rotary positional embeddings). This represents roughly 89% reduction in KV cache memory per token.

How does the "absorb" implementation maintain attention quality with compressed caches?

The absorb implementation in inference/model.py maintains quality by splitting queries into two components: q_nope (non-positional) interacts with the compressed latent cache, while q_pe (rotary) interacts with the separate positional cache. The final attention scores sum both contributions, preserving the expressive power of multi-head attention while operating on compressed representations.

Where is the MLA cache configured in the DeepSeek-V3 codebase?

The MLA cache dimensions are configured in inference/model.py lines 43-45, where kv_cache and pe_cache are registered as buffers with shapes (B, S, kv_lora_rank) and (B, S, qk_rope_head_dim) respectively. The specific rank values (typically 512 for kv_lora_rank) are defined in the model configuration files such as inference/configs/config_671B.json.

Can MLA support longer context windows than standard MHA?

Yes, MLA enables significantly longer context windows—up to 4× the original sequence length—because the 3–5× reduction in memory per token frees substantial GPU memory that would otherwise be consumed by the KV cache. This allows DeepSeek-V3 to process longer sequences while maintaining inference efficiency, as documented in the repository README (lines 45-49).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}