How to Use Memory Mapping for Models Larger Than VRAM in llama.cpp

Enable memory-mapping with --mmap 1 (the default) to stream model weights directly from disk, allowing llama.cpp to run models that exceed your GPU VRAM and system RAM combined by loading only active pages on demand.

The llama.cpp inference engine (ggml-org/llama.cpp) implements POSIX and Windows memory-mapping to handle GGUF files that are too large for physical memory. Instead of loading the entire model into RAM, the system maps the file into the process address space and defers reads until specific tensor data is required by the compute batch.

How Memory Mapping Works

Llama.cpp uses host-side memory-mapped pages to hold model weights when they cannot fit entirely in GPU memory. The operating system loads only the pages that the CPU actually accesses, while the GPU receives tensors on-demand via staged uploads.

The implementation resides in src/llama-mmap.cpp, where the constructor (lines 14-46) creates the mapping using mmap() on POSIX systems or CreateFileMapping/MapViewOfFile on Windows. The llama_model_loader class in src/llama-model-loader.cpp (lines 500-560) instantiates these mappings based on the use_mmap parameter, disabling the feature automatically if direct_io is requested or if the platform lacks support.

Enabling Memory Mapping via CLI

The --mmap flag is defined in common/arg.cpp (line 2201) and defaults to enabled. Use the following patterns depending on your hardware constraints:

# Default behavior (memory-mapping enabled)

./build/bin/llama-cli -m models/llama-2-70b.Q4_0.gguf -n 256

# Explicitly enable (useful in scripts for clarity)

./build/bin/llama-cli -m models/llama-2-70b.Q4_0.gguf --mmap 1 -n 256

# Disable mapping when the model fits entirely in RAM/VRAM

./build/bin/llama-cli -m models/llama-2-70b.Q4_0.gguf --no-mmap -n 256

When --no-mmap is specified, the loader performs a full read of the GGUF file into RAM, which improves latency for small models but causes out-of-memory errors for oversized weights.

Programmatic Configuration in C++

To enable memory-mapping from within your application, set the use_mmap field in the parameters structure before initialization:

#include "llama/llama-cpp.h"

gpt_params params;
params.model = "models/llama-2-70b.Q4_0.gguf";
params.use_mmap = true;               // Enable memory-mapping
params.use_direct_io = false;         // Keep false unless using O_DIRECT

// The loader creates llama_mmap objects internally based on this flag
llama_context * ctx = llama_init_from_params(params);
if (!ctx) {
    fprintf(stderr, "Failed to load model\n");
    return 1;
}

This parameter propagates through src/llama.cpp (lines 777-779) into the llama_model_loader constructor, which determines whether to allocate llama_mmap objects for each model file.

Internal Architecture

Mapping Creation

For each shard of a multi-file model, the loader constructs a llama_mmap object inside src/llama-model-loader.cpp:

// Inside llama_model_loader::init_mappings()
if (use_mmap) {
    std::unique_ptr<llama_mmap> mapping =
        std::make_unique<llama_mmap>(file.get(), prefetch ? -1 : 0, is_numa);
    // Track ranges for later unmapping
    mmaps_used.emplace_back(mapping->size(), 0);
}

The llama_mmap class encapsulates platform-specific handles and calculates page-aligned offsets for efficient I/O.

Tensor Allocation and Streaming

During tensor loading in src/llama-model-loader.cpp (lines 1095-1112), the allocator checks whether a tensor buffer already resides in GPU memory. If not, it uses the mapped host buffer:

// Allocation path uses existing mapping
ggml_backend_tensor_alloc(buf_mmap, cur, data);

This avoids doubling memory consumption by referencing the file-backed pages directly instead of copying them into a separate heap allocation.

Partial Unmapping After GPU Upload

To reclaim host RAM after transferring weights to the GPU, the loader calls unmap_fragment (lines 63-81 in src/llama-mmap.cpp):

// After uploading tensor data to GPU
auto & mmap_used = mmaps_used[weight->idx];
mmap_used.first  = std::min(mmap_used.first,  weight->offs);
mmap_used.second = std::max(mmap_used.second, weight->offs + n_size);

// Later, when all tensors are on the GPU:
mapping->unmap_fragment(mmap_used.first, mmap_used.second);

This function aligns the range to system page boundaries and calls munmap (or the Windows equivalent), allowing the operating system to release the physical pages while keeping the file handle open.

Optional Memory Locking

To prevent the operating system from swapping mapped pages to disk (Linux), combine --mmap with --mlock:

./build/bin/llama-cli -m model.gguf --mmap 1 --mlock

The lock implementation in src/llama-mlock.cpp (lines 10-30) invokes mlock or VirtualLock to pin pages into physical RAM, ensuring deterministic latency at the cost of requiring sufficient system memory.

Configuration Guidelines

Choose your memory strategy based on total available memory versus model size:

• Model size ≤ available RAM + GPU VRAM: Use --no-mmap for faster loading and lower latency, as the full model resides in physical memory without disk I/O during inference.
• Model size > RAM or OOM during loading: Keep the default --mmap 1 to stream weights from disk on demand.
• Production servers requiring deterministic performance: Add --mlock to prevent swap thrashing, ensuring mapped pages stay resident.

Summary

• Memory-mapping (--mmap 1) is the default mechanism in llama.cpp for handling models larger than combined VRAM and RAM.
• The implementation spans src/llama-mmap.cpp (OS abstraction), src/llama-model-loader.cpp (integration logic), and common/arg.cpp (CLI parsing).
• Tensors stream from disk-backed pages to GPU on-demand, with optional unmap_fragment calls freeing host memory after upload.
• Disable with --no-mmap only when the entire model fits comfortably in physical memory to maximize throughput.

Frequently Asked Questions

What happens if I disable memory mapping with --no-mmap?

The loader performs a blocking read of the entire GGUF file into heap-allocated RAM using standard I/O. This eliminates disk latency during inference but requires sufficient physical memory to hold the complete model weights, causing immediate termination if allocation fails.

Does memory mapping slow down inference?

Initial token generation may exhibit higher latency due to disk I/O as the OS faults in the required pages. However, once working tensors are cached in RAM or GPU memory, subsequent access speeds match non-mapped loads. The impact is negligible for CPU inference and manageable for GPU offload when using SSD storage.

Can I use memory mapping with multiple GPU layers?

Yes. Memory-mapping operates independently of GPU layer distribution (--gpu-layers or -ngl). The host-side mapped pages serve as the source for tensor uploads to any number of GPU layers, and unmap_fragment selectively frees host memory only after the GPU buffers are populated.

How does --mlock differ from standard memory mapping?

Standard memory mapping allows the kernel to swap file-backed pages to disk under memory pressure. The --mlock flag forces the system to pin all mapped pages into physical RAM using mlock() (Linux) or VirtualLock (Windows), preventing swap-out. This requires the process to have appropriate privileges (CAP_IPC_LOCK on Linux) and sufficient RAM to hold the accessed portions of the model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}