# How to Debug OOM Errors When Running DeepSeek-V3 671B Parameter Model Inference

> Debug OOM errors for DeepSeek-V3 671B model inference by reducing sequence length and batch size, switching to bf16 dtype, and profiling memory layer by layer to pinpoint the issue.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: tutorial
- Published: 2026-02-26

---

**Reduce `max_seq_len` and `max_batch_size` in `ModelArgs`, switch to `bf16` dtype to eliminate FP8 scaling tensors, and profile memory after each layer in `Transformer.forward` to identify the exact allocation causing the crash.**

The DeepSeek-V3 671B parameter model ([`config_671B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/config_671B.json)) represents the largest checkpoint in the DeepSeek-V3 family, requiring substantial GPU memory during inference. When running this model, **out-of-memory (OOM) errors** typically stem from three sources: model weights, the KV cache, and intermediate activations. Understanding how to debug these OOM errors when running 671B parameter model inference requires analyzing the specific allocation patterns in the `deepseek-ai/DeepSeek-V3` source code.

## Understanding Memory Consumption in DeepSeek-V3

Before debugging, identify which component exceeds your GPU capacity. The 671B model stores the majority of memory across three distinct areas defined in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py):

| Source | Storage Content | Allocation Location |
|--------|----------------|---------------------|
| **Model weights** | FP8 or BF16 quantized linear layers, embeddings, and MoE experts | `Linear`, `ColumnParallelLinear`, and `RowParallelLinear` classes in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 31-77) |
| **KV cache** | Attention context including `k`, `v`, and rotary-position embeddings for all processed tokens | `MLA` class creates `kv_cache` and `pe_cache` buffers in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 39-45) |
| **Intermediate activations** | Temporary tensors from forward pass operations (RMSNorm, MoE routing, attention projections) | Throughout `MLA.forward` and `MoE.forward` in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 61-99) |

When the sum of these allocations exceeds available GPU memory, PyTorch throws a CUDA OOM error. The following debugging steps progress from least invasive (configuration changes) to more invasive (code modifications).

## Debugging Steps for OOM Errors

### Verify the Hardware Budget

First, confirm your GPU capacity before adjusting model parameters:

```bash
nvidia-smi

```

This displays total and free memory per GPU. A single 24GB GPU cannot accommodate the 671B FP8 model (approximately 30GB for weights) combined with a full KV cache. If hardware constraints are absolute, proceed to reduce memory contributors.

### Reduce Maximum Sequence Length

The KV cache size grows linearly with `max_seq_len`. The default configuration sets `args.max_seq_len = 16384` (4096 × 4) in `ModelArgs` within [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 55-57).

```python
import json
from model import ModelArgs, Transformer

# Load configuration

cfg_path = "inference/configs/config_671B.json"
args = ModelArgs(**json.load(open(cfg_path)))

# Reduce sequence length to halve KV cache memory

args.max_seq_len = 8192
model = Transformer(args).cuda()

```

**Effect:** Reducing `max_seq_len` from 16384 to 8192 cuts the KV cache memory allocation by 50%, preventing OOMs triggered by long input contexts.

### Lower Batch Size

`max_batch_size` controls concurrent prompt processing, defaulting to 8 in `ModelArgs` (line 55). Each batch entry maintains independent KV cache buffers.

```python
args.max_batch_size = 1  # Process single prompt at a time

```

**Effect:** Setting `max_batch_size = 1` eliminates parallel cache allocations, saving `batch_size × seq_len × hidden_dim` bytes of GPU memory.

### Switch Data Type from FP8 to BF16

The model supports `bf16` and `fp8` via `ModelArgs.dtype`. While FP8 reduces weight storage, it introduces per-block scaling tensors that consume additional memory. For debugging, force `bf16` to eliminate these scaling tensors and isolate whether the OOM stems from cache or quantization overhead.

```python
args.dtype = "bf16"  # Forces Linear.dtype = torch.bfloat16 in model.py lines 60-62

```

**Effect:** Weight memory roughly doubles to approximately 60GB for 671B parameters, but removes FP8 scaling tensor overhead. If OOM persists with `bf16`, the KV cache is the primary memory consumer.

### Enable Activation Quantization

When operating in FP8 mode, enable block-wise activation quantization via `act_quant` in [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py) (lines 38-57). The `linear` wrapper in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 52-60) automatically invokes `act_quant` when `gemm_impl == "fp8"`.

No additional code required—simply ensure `args.dtype = "fp8"` and `gemm_impl` remains at its default `"fp8"` value.

### Profile Memory Per Layer

Insert instrumentation after each transformer block to identify exact allocation spikes:

```python
import torch

def mem_report(stage):
    torch.cuda.synchronize()
    allocated = torch.cuda.memory_allocated() / 1e9
    reserved = torch.cuda.memory_reserved() / 1e9
    print(f"[{stage}] allocated={allocated:.2f}GB reserved={reserved:.2f}GB")

# Usage inside Transformer.forward after each layer:

for i, layer in enumerate(self.layers):
    h = layer(h, start_pos, freqs_cis, mask)
    mem_report(f"after layer {i}")

```

Run once; the output reveals the specific layer where memory exceeds capacity.

### CPU Offloading for KV Cache

For advanced scenarios with limited GPU memory, move KV cache tensors to CPU between generation steps. The cache buffers in `MLA` class are standard PyTorch tensors.

```python

# Inside MLA.forward, after attention computation

self.kv_cache = self.kv_cache.cpu()
self.pe_cache = self.pe_cache.cpu()
torch.cuda.empty_cache()

```

**Effect:** Frees GPU memory between steps at the cost of PCIe transfer latency, enabling long-sequence generation on constrained hardware.

### Validate Checkpoint Loading

Duplicate weight loading occurs if mismatched safetensors shards are present. Ensure you load only the shard matching your world size:

```python

# Correct: single GPU, single shard

load_model(model, "ckpt_dir/model0-mp1.safetensors")

```

Verify no additional `model0-mp2.safetensors` or similar files are inadvertently loaded.

### Check Distributed Launch Options

Accidental multi-process launches on single GPU cause duplicate allocations. Verify environment variables before running:

```bash
echo $WORLD_SIZE $RANK $LOCAL_RANK

```

For single-GPU inference, ensure `WORLD_SIZE=1`. Values greater than 1 with only one visible device indicate configuration errors that trigger OOM.

## Practical Code Examples

### Minimal Script for Limited GPU Memory

Run 671B inference on a single 24GB GPU by aggressively limiting sequence length and batch size:

```python
import json
import torch
from model import ModelArgs, Transformer
from generate import generate
from transformers import AutoTokenizer

# Load and modify config

cfg_path = "inference/configs/config_671B.json"
args = ModelArgs(**json.load(open(cfg_path)))
args.max_seq_len = 8192      # Reduce from 16384

args.max_batch_size = 1      # Single prompt only

args.dtype = "bf16"          # Avoid FP8 scaling tensors

# Initialize model

model = Transformer(args).cuda()
model.eval()

# Generate

tokenizer = AutoTokenizer.from_pretrained("ckpt_dir")
prompt = "Explain the architecture of DeepSeek-V3."
tokens = [tokenizer.encode(prompt)]
output = generate(model, tokens, max_new_tokens=64,
                  eos_id=tokenizer.eos_token_id, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))

```

### Memory Debugging Instrumentation

Add this profiler to `Transformer.forward` in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) to identify which layer causes OOM:

```python
import torch

def log_memory(stage):
    torch.cuda.synchronize()
    allocated = torch.cuda.memory_allocated() / 1e9
    reserved = torch.cuda.memory_reserved() / 1e9
    print(f"[MEMORY] {stage}: allocated={allocated:.2f}GB, reserved={reserved:.2f}GB")

# Insert inside Transformer.forward:

log_memory("start forward")
for i, layer in enumerate(self.layers):
    h = layer(h, start_pos, freqs_cis, mask)
    log_memory(f"after layer {i}")

```

### Running with FP8 Activation Quantization

Enable FP8 weight storage with automatic activation quantization on a 40GB GPU:

```python
args.dtype = "fp8"          # Use FP8 weights

args.max_seq_len = 4096     # Conservative cache size

args.max_batch_size = 1
model = Transformer(args).cuda()

# Activation quantization happens automatically via act_quant() in kernel.py

# when linear() is called with gemm_impl="fp8"

```

## Key Files Reference

Understanding these source files is essential for effective debugging:

| File | Purpose | Critical Sections |
|------|---------|-------------------|
| [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) | Core architecture (`Transformer`, `MLA`, `MoE`, parallel linear layers) | `ModelArgs` definition (lines 55-58), KV cache allocation in `MLA.__init__` (lines 39-45), forward pass logic (lines 61-99) |
| [`inference/generate.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/generate.py) | Token generation loop and prompt handling | Prompt tensor construction (lines 51-78), generation loop with early stopping (lines 60-71) |
| [`inference/configs/config_671B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_671B.json) | Hyperparameters for 671B checkpoint | `dtype`, `max_seq_len`, `num_experts` configuration |
| [`inference/kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py) | FP8 quantization kernels and GEMM wrappers | `act_quant` for activation quantization (lines 38-57), `weight_dequant` (lines 89-110), `fp8_gemm` (lines 75-96) |

## Summary

- **Verify hardware capacity** with `nvidia-smi` before attempting to load the 671B checkpoint, which requires approximately 30GB for FP8 weights alone.
- **Reduce `max_seq_len`** in `ModelArgs` to linearly decrease KV cache memory usage, or set `max_batch_size = 1` to eliminate parallel cache allocations.
- **Switch to `bf16`** dtype to remove FP8 per-block scaling tensors when debugging whether quantization overhead causes OOM.
- **Enable activation quantization** by keeping `dtype="fp8"` and `gemm_impl="fp8"`, which automatically invokes `act_quant()` in [`kernel.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/kernel.py) to reduce activation memory.
- **Profile per-layer memory** by inserting `torch.cuda.memory_allocated()` calls in `Transformer.forward` to identify exactly which layer triggers the allocation spike.
- **Validate checkpoint loading** to ensure only the correct safetensors shard is loaded, and verify `WORLD_SIZE=1` for single-GPU inference to prevent duplicate allocations.

## Frequently Asked Questions

### What is the minimum GPU memory required to run DeepSeek-V3 671B inference?

The 671B parameter model requires approximately **30GB of GPU memory** for FP8 quantized weights alone, plus additional memory for the KV cache and intermediate activations. For practical inference with a reasonable sequence length (4096-8192 tokens), you need at least **40-80GB of GPU memory** (e.g., A100 80GB or multiple GPUs via tensor parallelism). Single 24GB consumer GPUs cannot fit the full model without aggressive CPU offloading or quantization beyond the standard FP8 format.

### Why does switching from FP8 to BF16 sometimes reduce OOM errors?

While FP8 reduces the storage size of model weights by 50% compared to BF16, it introduces **per-block scaling tensors** that consume additional memory during the `linear` layer operations in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py). When you set `args.dtype = "bf16"`, the model loads weights as `torch.bfloat16` without the extra scaling metadata, which can actually reduce total memory pressure if the KV cache is small. However, BF16 doubles the base weight memory to approximately 60GB, so this switch only helps when debugging quantization-related overhead or when the OOM occurs during activation computation rather than weight loading.

### How does the KV cache cause OOM errors during long generation sequences?

The KV cache stores key and value tensors for every token processed, with size proportional to `batch_size × seq_len × hidden_dim`. In the `MLA` class within [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) (lines 39-45), these buffers are allocated as `kv_cache` and `pe_cache` tensors. When generating long sequences (approaching the `max_seq_len` default of 16384), the cache grows linearly with each new token until it exhausts remaining GPU memory. Reducing `max_seq_len` in `ModelArgs` directly limits the maximum cache size, while setting `max_batch_size = 1` prevents parallel cache allocations from compounding the memory pressure.