How to Handle Hybrid CPU+GPU Inference for Large Models in llama.cpp

llama.cpp handles hybrid CPU+GPU inference for large models by splitting transformer layers across devices using n_gpu_layers, combining KV-cache and recurrent memory in llama_memory_hybrid, and automatically enabling pipeline parallelism across multiple GPUs.

Running very large transformer models on consumer hardware requires splitting computation between CPU and GPU resources. The llama.cpp framework implements sophisticated hybrid inference through layer offloading, unified memory management, and automatic multi-device scheduling. This approach allows models like Falcon-H1, Granite-Hybrid, and Qwen-3-Next to run efficiently even when individual layers exceed available VRAM.

Detecting Hybrid Model Architectures

Before allocating memory, llama.cpp classifies the model architecture to determine if it contains mixed attention and recurrent layers.

Identifying Hybrid Architectures

The llm_arch_is_hybrid() function in src/llama-arch.cpp checks the model type against known hybrid configurations:

bool llm_arch_is_hybrid(const llm_arch & arch) { 
    // Returns true for Falcon-H1, Granite-Hybrid, Qwen-3-Next, etc.
    return arch == LLM_ARCH_FALCON_H1 || 
           arch == LLM_ARCH_GRANITE_HYBRID ||
           arch == LLM_ARCH_QWEN3_NEXT;
}

Source: src/llama-arch.cpp#L25

This classification triggers the creation of hybrid memory structures rather than standard KV caches.

Configuring GPU Layer Offloading

The primary mechanism for hybrid CPU+GPU inference is selective layer placement controlled by the n_gpu_layers parameter.

Setting n_gpu_layers

In src/llama-model.h, the n_gpu_layers() method returns the number of layers to offload:

uint32_t n_gpu_layers() const;  // Returns value from -ngl flag or default (-1)

Source: src/llama-model.h#L535

The implementation in src/llama-model.cpp processes this value during model loading:

// Inside llama_model loading logic
if (hparams.n_gpu_layers < 0) {
    // -1 means all layers on CPU (default)
    hparams.n_gpu_layers = 0;
}

Source: src/llama-model.cpp#L7886

The offload_kqv and unified_kv Options

For large models, moving the KV cache to GPU memory significantly reduces CPU-GPU transfer overhead:

• offload_kqv: Moves the KQV (Key-Query-Value) matrices to GPU memory
• unified_kv: Allocates a single shared buffer for KV cache that all backends can access

These are set via llama_model_params:

llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = 30;      // Offload 30 layers to GPU
model_params.offload_kqv = true;       // Keep KV on GPU
model_params.unified_kv = true;        // Single buffer for all devices

Memory Management for Hybrid Models

Hybrid architectures containing both attention and recurrent layers require specialized memory handling through the llama_memory_hybrid class.

Combining KV and Recurrent Caches

The llama_memory_hybrid constructor in src/llama-memory-hybrid.cpp creates both cache types:

llama_memory_hybrid::llama_memory_hybrid(
    const llama_model & model,
    // Attention cache parameters
    ggml_type type_k,
    ggml_type type_v,
    bool offload_kqv,
    uint32_t kv_size,
    uint32_t kv_pad,
    uint32_t swa,
    llama_swa_type swa_type,
    // Recurrent cache parameters  
    ggml_type type_r,
    ggml_type type_w,
    uint32_t recurrent_size,
    // Common parameters
    uint32_t max_seqs,
    bool can_shift,
    bool can_mix
) {
    // Create attention KV cache
    cache_attn = new llama_kv_cache(...);
    
    // Create recurrent cache for Mamba/RWKV layers
    cache_recr = new llama_memory_recurrent(...);
}

Source: src/llama-memory-hybrid.cpp#L11

This unified interface allows the inference graph to treat hybrid models consistently, routing attention operations to the KV cache and recurrent operations to the recurrent cache.

Multi-GPU Pipeline Parallelism

For models too large for a single GPU, llama.cpp automatically enables pipeline parallelism across multiple devices.

Automatic Pipeline Configuration

The context constructor in src/llama-context.cpp determines whether to enable pipeline parallelism based on available hardware:

// Inside llama_context constructor
if (model.n_devices() > 1 && 
    model.n_gpu_layers() > model.hparams.n_layer &&
    cparams.split_mode == LLAMA_SPLIT_MODE_LAYER &&
    cparams.offload_kqv &&
    model.tensor_overrides.empty()) {
    
    cparams.pipeline_parallel = true;
}

Source: src/llama-context.cpp#L311-L317

When enabled, the ggml_backend_sched distributes graph nodes across backends, allowing layer N to execute on GPU 1 while layer N-1 still runs on GPU 0, overlapping computation with data transfer.

Practical Implementation Examples

Loading a Hybrid Model with CPU+GPU Offload

#include "llama.h"

int main() {
    // Configure model parameters for hybrid inference
    llama_model_params model_params = llama_model_default_params();
    model_params.n_gpu_layers = 30;          // Offload first 30 layers to GPU
    model_params.offload_kqv = true;         // Keep KV cache on GPU
    model_params.unified_kv = true;          // Single buffer for all devices

    // Load the model - automatically detects hybrid architecture
    llama_model * model = llama_load_model_from_file("granite-hybrid.gguf", 
                                                    model_params);
    if (!model) {
        fprintf(stderr, "Failed to load model\n");
        return 1;
    }

    // Create context - enables pipeline parallelism if multiple GPUs detected
    llama_context_params ctx_params = llama_context_default_params();
    llama_context * ctx = llama_new_context_with_model(model, ctx_params);

    // Inference proceeds normally - memory routing handled automatically
    const char * prompt = "Explain hybrid CPU GPU inference.";
    std::vector<llama_token> tokens = llama_tokenize(ctx, prompt, true);
    
    // Encode prompt
    llama_encode(ctx, tokens.data(), tokens.size());

    // Generate tokens
    for (int i = 0; i < 128; ++i) {
        llama_decode(ctx, nullptr, 0);
        int id = llama_sample_token_greedy(ctx);
        printf("%s", llama_token_to_piece(ctx, id).c_str());
    }

    // Cleanup
    llama_free_context(ctx);
    llama_free_model(model);
    
    return 0;
}

Inspecting Memory Distribution

To verify that layers and caches are distributed correctly between CPU and GPU:

// Get memory breakdown by buffer type
auto breakdown = llama_memory_breakdown(ctx);
for (auto & [buft, size] : breakdown) {
    const char * name = ggml_backend_buft_name(buft);
    printf("%-20s : %8.2f MiB\n", name, size / 1024.0 / 1024.0);
}

Source: src/llama-memory-hybrid.cpp#L71-L77

Summary

• Hybrid architecture detection happens automatically via llm_arch_is_hybrid() in src/llama-arch.cpp, identifying models with mixed attention and recurrent layers.

• Layer offloading is controlled by n_gpu_layers in llama_model_params, implemented in src/llama-model.cpp, allowing you to place a suffix of layers on the GPU while the prefix runs on the CPU.

• Hybrid memory management uses llama_memory_hybrid (defined in src/llama-memory-hybrid.cpp) to combine standard KV caches for attention layers with recurrent caches for Mamba/RWKV-style layers, presenting a unified interface.

• Pipeline parallelism activates automatically in src/llama-context.cpp when multiple GPUs are present and the model exceeds single-device capacity, overlapping computation across devices via ggml_backend_sched.

• Optimization flags offload_kqv and unified_kv minimize CPU-GPU data transfers by keeping the KV cache in GPU memory and using a single shared buffer across all backends.

Frequently Asked Questions

How do I determine the optimal number of GPU layers for my hardware?

Start by calculating the memory footprint per layer using the model's parameters and your GPU's VRAM. In src/llama-model.cpp, the loader respects the n_gpu_layers value you provide, placing that many layers on the GPU buffer type. For a 70B model on a 24GB GPU, typically 30-40 layers fit comfortably. Monitor memory usage via llama_memory_breakdown() to verify distribution.

What is the difference between offload_kqv and unified_kv?

offload_kqv (controlled via llama_model_params.offload_kqv) moves the Key-Query-Value matrices to GPU memory, reducing per-token transfer overhead. unified_kv (via unified_kv parameter) creates a single buffer for the KV cache that all backends share, eliminating copies between CPU and GPU during context switching. Both are implemented in src/llama-model.cpp during memory creation.

Does llama.cpp automatically use multiple GPUs for large models?

Yes, when the system detects multiple GPU devices and the model has more layers than fit on a single device, src/llama-context.cpp automatically enables pipeline parallelism (lines 311-317). The ggml_backend_sched distributes graph nodes across available backends, overlapping computation so that layer N runs on GPU 1 while layer N-1 still processes on GPU 0. No manual configuration is required beyond ensuring n_gpu_layers covers all model layers.

How does llama.cpp handle models with both attention and recurrent layers?

For hybrid architectures like Falcon-H1 or Qwen-3-Next, llama.cpp uses the llama_memory_hybrid class defined in src/llama-memory-hybrid.cpp. This class creates both a standard llama_kv_cache for attention layers and a llama_memory_recurrent cache for Mamba/RWKV-style layers, routing operations through a unified interface. The constructor accepts filter lambdas to determine which layers use which cache type, allowing flexible handling of mixed architectures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}