How to Use LoRA Adapters with GGUF Models in llama.cpp

LoRA adapters in llama.cpp require converting PEFT checkpoints to GGUF format using convert_lora_to_gguf.py, loading them with llama_adapter_lora_init() before context creation, and applying them via CLI flags, HTTP API, or the llama_set_adapters_lora() C function.

The llama.cpp inference engine supports Low-Rank Adaptation (LoRA) fine-tuning through self-contained GGUF adapter files. This allows you to dynamically apply, scale, or switch between multiple adapters without merging weights into the base model. According to the ggml-org/llama.cpp source code, the workflow spans three distinct stages: conversion, loading, and runtime application.

Converting PEFT LoRA Checkpoints to GGUF

Before inference, Hugging Face PEFT checkpoints must be converted into the GGUF container format. The project provides convert_lora_to_gguf.py for this purpose.

The script parses adapter_config.json and reads the weight file (adapter_model.safetensors or .bin) to generate a single-file GGUF adapter. It validates that the base model architecture matches the adapter configuration before writing the output.

python3 convert_lora_to_gguf.py \
    --lora-path /path/to/hf_adapter_dir \
    --outfile my_adapter.gguf \
    --outtype f16

Supported output types include f16, f32, bf16, q8_0, and auto. The conversion step is mandatory because the C++ loader only understands GGUF tensors. This process preserves memory-mapping compatibility, allowing adapters to be loaded via mmap without duplication in RAM.

Loading LoRA Adapters in the C API

Adapters must be loaded before any llama_context is created. In include/llama.h, the function llama_adapter_lora_init() handles this initialization.

// 1. Load the base model (must remain alive while adapters exist)
struct llama_model * model = llama_load_model_from_file("model.gguf", ...);

// 2. Load adapters before creating context
struct llama_adapter_lora * adapter0 = llama_adapter_lora_init(model, "my_adapter.gguf");
struct llama_adapter_lora * adapter1 = llama_adapter_lora_init(model, "my_second_adapter.gguf");

// 3. Create context (adapters are now available for this model)
struct llama_context_params ctx_params = llama_context_default_params();
struct llama_context * ctx = llama_new_context_with_model(model, ctx_params);

The pointer returned by llama_adapter_lora_init() remains valid for the lifetime of the llama_model. Attempting to load adapters after calling llama_new_context_with_model() will fail. The adapter files remain separate from the base model GGUF; they are never merged on disk.

Applying Adapters at Runtime

Once loaded, adapters can be applied globally or per-request using multiple interfaces.

Command-Line Interface (CLI)

The llama-cli binary supports the --lora and --lora-scaled flags documented in tools/completion/README.md. Multiple adapters are applied in the order specified.

./llama-cli -m model.gguf \
    --lora my_adapter.gguf \
    --lora-scaled other_adapter.gguf:0.7 \
    -p "Explain quantum entanglement."

The --lora flag applies an adapter with a scale of 1.0, while --lora-scaled accepts a file:scale syntax for custom weighting.

HTTP Server API

The server binary supports hot-swapping adapters via request parameters. Start the server with the --lora flag to preload adapters, optionally using --lora-init-without-apply to load adapters without immediately activating them.

./server \
    -m model.gguf \
    --lora adapter_a.gguf \
    --lora adapter_b.gguf \
    --lora-init-without-apply

Subsequent generations can specify active adapters and scales via JSON:

import requests

payload = {
    "model": "model.gguf",
    "prompt": "Write a haiku about AI.",
    "lora": [
        {"id": 0, "scale": 0.8},  # adapter_a.gguf

        {"id": 1, "scale": 1.0}   # adapter_b.gguf

    ]
}
resp = requests.post("http://localhost:8080/v1/generate", json=payload)

C API for Dynamic Switching

After context creation, use llama_set_adapters_lora() to change the active adapter set. This function updates the context only when the new list differs from the current configuration.

struct llama_adapter_lora * adapters[] = { adapter0, adapter1 };
float scales[] = { 0.5f, 1.2f };
int ret = llama_set_adapters_lora(ctx, adapters, 2, scales);

This allows per-request customization without reloading the base model or destroying the context.

Important Implementation Details

Several constraints govern LoRA usage in llama.cpp:

• Memory mapping: Adapters remain separate GGUF files and utilize the same memory-mapping infrastructure as base models.
• Metadata access: The API provides llama_adapter_meta_* functions in include/llama.h for querying adapter metadata such as author or training data.
• Loading order: Adapters must be initialized via llama_adapter_lora_init() strictly before llama_new_context_with_model().
• Preloading: The server flag --lora-init-without-apply enables fast hot-swapping by deferring application until the first relevant request.

Summary

• Convert Hugging Face LoRA checkpoints to GGUF using convert_lora_to_gguf.py to ensure compatibility with the C++ loader.
• Load adapters before context creation using llama_adapter_lora_init(), keeping the base model alive for the adapter's lifetime.
• Apply adapters globally via CLI flags (--lora, --lora-scaled), per-request via HTTP API JSON parameters, or programmatically via llama_set_adapters_lora().
• Scale individual adapters using floating-point coefficients to control merging strength at runtime.

Frequently Asked Questions

Can I merge LoRA adapters into the base GGUF model permanently?

Yes, though this bypasses the dynamic adapter system. The tools/export-lora/export-lora.cpp utility can merge LoRA weights into a base model GGUF file, creating a single static model. However, this eliminates the ability to hot-swap or scale adapters at runtime.

Why must adapters be loaded before context creation?

The llama_adapter_lora_init() function in include/llama.h allocates internal graph structures that must be present when the compute context is initialized. Once llama_new_context_with_model() executes, the tensor graph is fixed; subsequent adapter loading would require reallocating GPU buffers and recomputing tensor layouts, which the API prohibits for performance and memory safety reasons.

What happens if I provide multiple adapters with conflicting layers?

Adapters are applied in the order specified. If two adapters modify the same layer, the later adapter in the sequence overwrites the modifications of the earlier one for that specific layer. Scaling factors apply individually to each adapter's contribution before the merge operation.

Does llama.cpp support QLoRA or quantization-aware LoRA training?

The conversion script supports quantizing adapter weights to q8_0 or other formats during GGUF export, but the library does not perform quantization-aware training. You should train LoRA adapters in full precision (or BF16) using standard PEFT libraries, then quantize the resulting checkpoint to GGUF for inference in llama.cpp.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}