DeepSeek-V3 MLA KV Cache Memory Footprint for 128K Context Length

For a 128,000-token context window, DeepSeek-V3's Multi-Head Latent Attention (MLA) consumes approximately 125 MiB of GPU memory for the KV cache in BFloat16 mode, or 62.5 MiB when using FP8 quantization.

DeepSeek-V3 achieves efficient long-context inference through a compressed KV cache mechanism implemented in its Multi-Head Latent Attention layers. Unlike standard multi-head attention that stores full key and value heads for every token, the MLA architecture in the deepseek-ai/DeepSeek-V3 repository utilizes a low-rank projection with a fixed dimension of 512, drastically reducing the memory footprint required to support 128K context lengths.

MLA KV Cache Tensor Structure

The KV cache in DeepSeek-V3 is stored as a compressed tensor with the exact shape:

(batch_size, max_seq_len, kv_lora_rank)

Key architectural constants from the inference source code:

• kv_lora_rank: Defined as 512 in inference/configs/config_236B.json (lines 15–17). This compressed rank replaces the traditional (num_heads × head_dim) storage, which typically requires thousands of floating-point values per token.
• Buffer initialization: The cache is registered as a non-persistent buffer in inference/model.py (lines 440–445):

self.register_buffer(
    "kv_cache",
    torch.zeros(args.max_batch_size, args.max_seq_len, self.kv_lora_rank),
    persistent=False,
)

During the forward pass, this buffer is populated with the normalized KV projection (self.kv_norm(kv)) and retained in the model's compute dtype for the duration of the sequence generation.

Memory Calculation for 128K Context Length

The total memory footprint scales linearly with sequence length and depends on the precision mode. For a single batch with 128,000 tokens:

Data Type	Bytes per Element	Formula	Total Memory
BFloat16 (bf16)	2	128,000 × 512 × 2	125 MiB
FP8	1	128,000 × 512 × 1	62.5 MiB

Byte calculation details:

• 128,000 tokens (context length)
• 512 dimensions (kv_lora_rank)
• 2 bytes per element for BF16 (default inference mode)
• 1 byte per element for FP8 (optional quantization)

This represents a 32× reduction compared to standard attention mechanisms, which would require caching 8,192+ dimensions per token (e.g., 128 heads × 64 dimensions) for equivalent model capacity.

Precision Modes and GPU Efficiency

DeepSeek-V3 supports two compute dtypes for the KV cache, controlled via the fp8 inference flag:

• BFloat16: The default 2-byte precision providing full dynamic range for the 512-dimensional latent vectors.
• FP8: 1-byte quantization that halves the memory footprint, enabling 128K context inference with only 62.5 MiB of cache memory per layer.

You can programmatically calculate the cache requirement for any context length using the architectural constants from the source:

def mla_kv_cache_mb(seq_len, kv_rank=512, dtype="bf16"):
    """Calculate DeepSeek-V3 MLA KV cache size in MiB."""
    bytes_per_elem = 2 if dtype == "bf16" else 1
    total_bytes = seq_len * kv_rank * bytes_per_elem
    return total_bytes / (1024 * 1024)

# 128K context calculations

print(f"BF16: {mla_kv_cache_mb(128000, dtype='bf16'):.1f} MiB")
print(f"FP8:  {mla_kv_cache_mb(128000, dtype='fp8'):.1f} MiB")

Expected output:
BF16: 125.0 MiB
FP8:  62.5 MiB

The persistent=False parameter in the buffer registration ensures these temporary inference tensors are excluded from model state dicts, preventing unnecessary disk I/O during checkpoint operations.

Summary

• DeepSeek-V3 MLA compresses per-token KV representations into a fixed 512-dimensional vector (kv_lora_rank), independent of the total number of attention heads.
• 128K context length requires 125 MiB per layer in BF16 or 62.5 MiB in FP8, as defined in inference/configs/config_236B.json and allocated in inference/model.py.
• The cache tensor shape is (batch_size, max_seq_len, 512), using 2 bytes per element by default.
• Memory increases linearly: each additional 1,000 tokens consumes approximately 0.98 MiB (BF16) or 0.49 MiB (FP8).

Frequently Asked Questions

How does MLA achieve such a small memory footprint for 128K contexts?

Multi-Head Latent Attention decouples the key and value calculations from the full head dimension through a low-rank compression matrix. Instead of storing 8,192+ floating-point values per token (typical for 128-head models), MLA caches only 512 latent values, achieving a 16:1 compression ratio while preserving attention quality via the kv_lora_rank projection defined in the model configuration.

What is the kv_lora_rank parameter and where is it configured?

The kv_lora_rank is a hyperparameter set to 512 in inference/configs/config_236B.json (lines 15–17). It defines the width of the compressed latent space for keys and values. This value is loaded during model initialization in inference/model.py and directly determines the third dimension of the kv_cache tensor, balancing memory efficiency against representational capacity.

Can the KV cache size be reduced further for 128K inference?

Yes. Switching from BFloat16 (2 bytes) to FP8 (1 byte) quantization halves the cache footprint from 125 MiB to 62.5 MiB for 128K tokens. This is supported natively in the DeepSeek-V3 inference engine via dtype flags and is particularly effective for batch generation scenarios where cache memory becomes the bottleneck.

Where in the codebase is the 128K context KV cache allocated?

The allocation occurs in inference/model.py at lines 440–445, where torch.zeros(args.max_batch_size, args.max_seq_len, self.kv_lora_rank) creates the buffer. When args.max_seq_len is set to 128000, this line allocates the 125 MiB (BF16) tensor on the GPU device, sized specifically for the compressed MLA representation rather than full attention heads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}