# How to Handle Out-of-Memory Errors with Large Context Windows in llama.cpp

> Learn how llama.cpp handles out of memory errors with large context windows. Discover how it detects and reports allocation issues for KV cache, compute buffers, and output tensors.

- Repository: [ggml/llama.cpp](https://github.com/ggml-org/llama.cpp)
- Tags: how-to-guide
- Published: 2026-02-27

---

**llama.cpp detects out-of-memory conditions during buffer allocation for the KV-cache, compute buffers, and output tensors, immediately throwing `std::runtime_error` and logging specific error messages rather than attempting automatic recovery.**

When working with large language models in [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/ggml-org/llama.cpp), increasing the context window size (`n_ctx`) linearly increases RAM consumption for the KV-cache and temporary compute buffers. If your system cannot satisfy these allocation requests, the library aborts with explicit error logs in [`src/llama-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-context.cpp) and [`src/llama-kv-cache.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-kv-cache.cpp) rather than crashing silently or implementing dynamic fallbacks.

## Where llama.cpp Detects Memory Allocation Failures

The codebase implements defensive checks at critical allocation points. These checks do not attempt recovery; they validate allocation success and abort immediately upon failure to prevent undefined behavior.

| Situation | Code Location | Behavior |
|-----------|--------------|----------|
| **Output or compute buffer allocation failure** | [`src/llama-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-context.cpp) (lines 1840, 2024) | Logs `LLAMA_LOG_ERROR` with messages like `"failed to allocate output buffer"` or `"failed to allocate compute buffers"` and throws `std::runtime_error` |
| **KV-cache creation failure** | [`src/llama-kv-cache.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-kv-cache.cpp) (line 193) | Throws `std::runtime_error("failed to allocate buffer for kv cache")` during initialization |
| **Per-slot context capacity exceeded** | [`tools/server/server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server-context.cpp) (line 1242) | Logs via `SLT_DBG` with message `"stopped due to running out of context capacity"` and stops decoding for that specific slot |
| **Prompt cache limit reached** | [`tools/server/server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server-context.cpp) (lines 803-811) | Respects the `cache_ram_mib` parameter; refuses to allocate beyond the specified limit and evicts older entries |

## Calculating Memory Requirements for Large Contexts

The **KV-cache** dominates memory consumption for long contexts. Before increasing `n_ctx`, estimate requirements using the formula: `n_ctx * n_embd * 2 * 4 bytes`, where `n_embd` is the model's hidden dimension. Raising the context from the default 2048 tokens to 8192 tokens quadruples the KV-cache footprint, often consuming several gigabytes of additional RAM.

## Strategies to Prevent OOM Errors

### Choose a Realistic Context Size (`n_ctx`)

The default CLI value is **2048 tokens**. Use `-c 4096` or `-c 8192` (or set `cparams.n_ctx` in the API) only after verifying available RAM against the calculation above. The allocation occurs in `llama_init_from_model` in [`src/llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama.cpp), which propagates any `std::runtime_error` from the underlying buffer allocation code.

### Limit the Prompt Cache Size

Pass `--cache-ram-mib <MiB>` when starting the server. As implemented in [`tools/server/server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server-context.cpp) (lines 803-811), this creates a bounded `server_prompt_cache` that refuses to grow beyond the specified limit, forcing eviction of older entries rather than allocating new memory.

```bash
./server -m model.gguf --cache-ram-mib 2048   # 2 GiB hard limit

```

### Enable Context Shifting

The server enables **context shifting** by default (disable with `--no-shift`). When active, the server retains only the most recent `n_ctx` tokens in the KV-cache, discarding older entries to prevent unbounded growth. This prevents OOM during long conversations but may lose earlier context.

### Reduce Batch Size (`n_batch`)

Temporary compute buffers scale with batch size. Reduce peak allocation during forward passes by setting `--batch-size 256` (CLI) or `cparams.n_batch = 256` (API). This parameter controls how many tokens are processed per forward pass in the compute graph.

### Use Quantized Model Weights

Load models with **8-bit or 4-bit quantization** (`--ggml-type Q8_0` or `Q4_0`) to reduce the memory footprint of model weights. This leaves more available RAM for the KV-cache when running with large `n_ctx` values.

## Implementation Examples

### Creating a Context with Safe Parameters (C++ API)

```cpp
llama_context_params cparams = llama_context_default_params();
cparams.n_ctx   = 4096;          // desired context length
cparams.n_batch = 256;           // smaller batch reduces peak RAM
cparams.cache_type = LLAMA_CACHE_TYPE_SRAM;   // keep KV cache in RAM

llama_context * ctx = llama_init_from_model(model, cparams);
if (!ctx) {
    // llama_init_from_model throws on allocation failure – catch it
    std::cerr << "Failed to create context – possibly OOM" << std::endl;
    std::exit(1);
}

```

*Source references:* Allocation checks in [`src/llama-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-context.cpp) (lines 1840, 2024); initialization entry point in [`src/llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama.cpp).

### Server-Side Slot Capacity Handling

```cpp
// inside tools/server/server-context.cpp (slot decode loop)
if (slot.prompt.n_tokens() + 1 >= slot.n_ctx) {
    SLT_DBG(slot,
        "stopped due to running out of context capacity, "
        "prompt.n_tokens() = %d, task.n_tokens = %d, n_ctx = %d\n",
        slot.prompt.n_tokens(), slot.task->n_tokens(), slot.n_ctx);
    break;   // abort generation for this request
}

```

*Source reference:* [`tools/server/server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server-context.cpp) at line 1242.

## Summary

- **OOM is fatal**: The library throws `std::runtime_error` and aborts the operation; no automatic recovery or context shrinking occurs.
- **Check logs**: Error messages `"failed to allocate output buffer"`, `"failed to allocate compute buffers"`, or `"failed to allocate buffer for kv cache"` indicate specific allocation failures in [`src/llama-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-context.cpp) or [`src/llama-kv-cache.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-kv-cache.cpp).
- **Prevention**: Calculate `n_ctx * n_embd * 2 * 4 bytes` before increasing context size, use `--cache-ram-mib` to cap server cache, enable context shifting, reduce `n_batch`, and prefer quantized models.
- **Server behavior**: Per-slot OOM (exceeding `n_ctx`) stops decoding for that request only, while core buffer allocation failures terminate the process.

## Frequently Asked Questions

### Does llama.cpp support automatic context shrinking when memory is low?

No. The library does not implement automatic fallback mechanisms such as dynamically reducing `n_ctx` when allocation fails. As seen in [`src/llama-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-context.cpp) (lines 1840, 2024) and [`src/llama-kv-cache.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-kv-cache.cpp) (line 193), the code immediately throws `std::runtime_error` upon allocation failure. The caller must catch this exception and decide whether to retry with a smaller context size or terminate.

### What error message indicates the KV-cache failed to allocate?

The specific error message is `"failed to allocate buffer for kv cache"`, thrown as a `std::runtime_error` from [`src/llama-kv-cache.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/src/llama-kv-cache.cpp) at line 193. This occurs during the initialization of the `llama_context` when the system cannot allocate the continuous memory block required for keys and values across all layers.

### How do I calculate the exact RAM needed for a specific context window?

Use the formula `n_ctx * n_embd * 2 * 4 bytes`. Multiply the desired context length (`n_ctx`) by the model's embedding dimension (`n_embd`), then multiply by 2 (for key and value tensors) and by 4 (bytes per float32). For a 4096-dimensional model at 8192 context length, this requires approximately 256 MiB for the KV-cache alone, excluding compute buffers and model weights.

### Can I recover from an OOM error without restarting the process?

Partially. If using the server, a single slot exceeding its context capacity logs `"stopped due to running out of context capacity"` and stops decoding for that specific request (line 1242 in [`tools/server/server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server-context.cpp)), leaving other slots operational. However, core buffer allocation failures (compute buffers, output buffers, or initial KV-cache creation) throw exceptions that typically require catching at the application level and reinitializing the context with smaller parameters, as the `llama_context` will be in an invalid state.