How to Convert Fine-Tuned Models to GGUF for Ollama and llama.cpp

To convert fine-tuned models to GGUF format, merge your LoRA adapter with the base model, build the llama.cpp quantization tools with CMake, and run the conversion script to generate quantized GGUF files compatible with Ollama and llama.cpp.

The huggingface/skills repository provides a production-ready workflow for transforming Hugging Face LoRA adapters into optimized GGUF artifacts. This guide explains how to use the convert_to_gguf.py script to convert fine-tuned models to GGUF format and deploy them locally.

What Is GGUF and Why Convert Fine-Tuned Models?

GGUF (GPT-Generated Unified Format) is a binary format designed for efficient inference with local LLM engines. When you fine-tune a model using LoRA (Low-Rank Adaptation), you typically produce small adapter weights rather than a full model checkpoint. To use these adapters with Ollama, llama.cpp, or LM Studio, you must merge the adapter with the base model and convert the result to GGUF format. This process reduces memory footprint through quantization while maintaining inference speed.

Prerequisites for Converting Fine-Tuned Models to GGUF

System Dependencies

Before running the conversion, ensure your system has git, gcc, and cmake installed. The convert_to_gguf.py script explicitly installs build tools before cloning the llama.cpp repository because the llama-quantize binary requires compilation with CMake.

Environment Configuration

Set the following environment variables to configure the conversion pipeline:

export ADAPTER_MODEL="username/my-finetuned-model"
export BASE_MODEL="Qwen/Qwen2.5-0.5B"
export OUTPUT_REPO="username/my-model-gguf"
export HF_USERNAME="username"

Step-by-Step Guide: Convert Fine-Tuned Models to GGUF

Step 1: Load and Merge the LoRA Adapter

The script loads the base model specified in BASE_MODEL and the LoRA adapter from ADAPTER_MODEL using the peft library. It merges the adapter weights into the base model to create a single, standalone checkpoint. This merged model is temporarily saved to disk before conversion.

Step 2: Install Build Tools and Clone llama.cpp

Unlike simpler workflows, this implementation installs system dependencies (gcc, cmake) first, then clones the llama.cpp repository. This sequencing is critical because the quantisation tools must be built from source.

# The script handles this automatically, but the equivalent manual steps are:

sudo apt-get update && sudo apt-get install -y git gcc cmake
git clone https://github.com/ggerganov/llama.cpp.git

Step 3: Build llama-quantize with CMake

The script uses CMake (the recommended build system) to compile the llama-quantize binary. This tool is essential for converting FP16 GGUF files into quantized formats like Q4_K_M and Q8_0.

cd llama.cpp && cmake -B build && cmake --build build --config Release

Step 4: Convert to FP16 GGUF Format

Using the convert_hf_to_gguf.py script from the llama.cpp repository, the merged Hugging Face checkpoint is converted to a full-precision FP16 GGUF file. This serves as the source for all quantized variants.

Step 5: Quantize to Multiple GGUF Formats

The script invokes the compiled llama-quantize tool to produce several quantization levels:

• Q4_K_M: 4-bit quantization with medium complexity (balanced size/quality)
• Q5_K_M: 5-bit quantization for higher quality
• Q8_0: 8-bit quantization for maximum accuracy with larger file size
• F16: Full FP16 precision (no quantization)

Step 6: Upload to Hugging Face Hub

Finally, the script generates a comprehensive README.md documenting the model provenance, quantization options, and usage examples. All GGUF variants and the README are uploaded to the repository specified in OUTPUT_REPO.

Running the Conversion as a Hugging Face Job

For large models or automated pipelines, submit the conversion as a Hugging Face Job instead of running locally:

from huggingface_hub import HfApi

hf_api = HfApi()
hf_api.create_repo(repo_id="username/my-model-gguf", repo_type="model", exist_ok=True)

job = {
    "script": open("skills/hugging-face-model-trainer/scripts/convert_to_gguf.py").read(),
    "flavor": "a10g-large",
    "timeout": "45m",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
    "env": {
        "ADAPTER_MODEL": "username/my-finetuned-model",
        "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
        "OUTPUT_REPO": "username/my-model-gguf",
        "HF_USERNAME": "username",
    },
}

# Submit via your preferred job runner

Verify that both ADAPTER_MODEL and BASE_MODEL exist on the Hub before submission using hf_api.model_info().

Using GGUF Models with Ollama and llama.cpp

Load GGUF in Ollama

After converting fine-tuned models to GGUF format, deploy them locally with Ollama:

# Download the quantized model

huggingface-cli download username/my-model-gguf model-q4_k_m.gguf

# Create a Modelfile

echo "FROM ./model-q4_k_m.gguf" > Modelfile

# Build and run

ollama create my-gguf-model -f Modelfile
ollama run my-gguf-model

Run Inference with llama.cpp

Use the compiled llama-cli binary for direct inference:

# CPU inference

./llama.cpp/build/bin/llama-cli -m model-q4_k_m.gguf -p "Explain quantization"

# GPU-accelerated inference (if CUDA available)

./llama.cpp/build/bin/llama-cli -m model-q4_k_m.gguf -ngl 32 -p "Explain quantization"

Import into LM Studio

1. Download the desired .gguf file from your Hugging Face repository.
2. Open LM Studio and navigate to Import Model.
3. Select the GGUF file to load your fine-tuned model for local chat sessions.

Summary

• GGUF conversion requires merging LoRA adapters with base models before quantization.
• The convert_to_gguf.py script in huggingface/skills automates the entire pipeline: dependency installation, CMake builds, FP16 conversion, and multi-format quantization.
• CMake is the recommended build system for compiling llama-quantize, ensuring reliable quantization across platforms.
• The script generates Q4_K_M, Q5_K_M, Q8_0, and F16 variants to balance speed, size, and accuracy.
• Converted models work immediately with Ollama, llama.cpp, and LM Studio without additional configuration.

Frequently Asked Questions

What is the difference between GGUF and Safetensors formats?

Safetensors is a safe, fast serialization format used by Hugging Face Transformers to store model weights during training and inference. GGUF is a binary format specifically designed for local inference engines like llama.cpp and Ollama, optimized for memory-mapped file access and supporting various quantization schemes. You must convert from Safetensors (or PyTorch binaries) to GGUF to run models in local inference tools.

Why must I install build tools before cloning llama.cpp?

The llama-quantize binary required for creating quantized GGUF files must be compiled from C++ source code using CMake and GCC. The convert_to_gguf.py script installs these system dependencies before cloning the repository to ensure the subsequent CMake build succeeds. Attempting to build without these tools results in compilation errors when generating the quantization binaries.

Which quantization format should I choose for Ollama?

For most use cases with Ollama, Q4_K_M (4-bit medium complexity) offers the best balance between model size and output quality, reducing VRAM requirements while maintaining coherent generation. If you have ample GPU memory and require higher fidelity for complex reasoning tasks, use Q8_0 (8-bit). The F16 format provides full precision but requires significantly more storage and RAM, making it suitable only for high-end workstations.

Can I convert models without using the Hugging Face Hub?

Yes, while the convert_to_gguf.py script is optimized for Hugging Face Hub workflows, you can adapt it for local-only conversion by modifying the model loading paths. Load your local base model and adapter using AutoModelForCausalLM.from_pretrained() and PeftModel.from_pretrained() with local directory paths, then proceed with the local CMake build and quantization steps. However, you will need to manually handle the GGUF file management and README generation that the script normally automates for Hub uploads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}