# How to Convert Fine-Tuned Models to GGUF for Ollama and llama.cpp

> Convert fine-tuned models to GGUF for Ollama and llama.cpp. Merge LoRA, build llama.cpp tools, and run the script to generate quantized GGUF files for efficient local AI deployment.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: how-to-guide
- Published: 2026-03-08

---

**To convert fine-tuned models to GGUF format, merge your LoRA adapter with the base model, build the llama.cpp quantization tools with CMake, and run the conversion script to generate quantized GGUF files compatible with Ollama and llama.cpp.**

The `huggingface/skills` repository provides a production-ready workflow for transforming Hugging Face LoRA adapters into optimized GGUF artifacts. This guide explains how to use the [`convert_to_gguf.py`](https://github.com/huggingface/skills/blob/main/convert_to_gguf.py) script to convert fine-tuned models to GGUF format and deploy them locally.

## What Is GGUF and Why Convert Fine-Tuned Models?

**GGUF** (GPT-Generated Unified Format) is a binary format designed for efficient inference with local LLM engines. When you fine-tune a model using LoRA (Low-Rank Adaptation), you typically produce small adapter weights rather than a full model checkpoint. To use these adapters with **Ollama**, **llama.cpp**, or **LM Studio**, you must merge the adapter with the base model and convert the result to GGUF format. This process reduces memory footprint through quantization while maintaining inference speed.

## Prerequisites for Converting Fine-Tuned Models to GGUF

### System Dependencies

Before running the conversion, ensure your system has `git`, `gcc`, and `cmake` installed. The [`convert_to_gguf.py`](https://github.com/huggingface/skills/blob/main/convert_to_gguf.py) script explicitly installs build tools **before** cloning the [`llama.cpp`](https://github.com/huggingface/skills/blob/main/llama.cpp) repository because the `llama-quantize` binary requires compilation with CMake.

### Environment Configuration

Set the following environment variables to configure the conversion pipeline:

```bash
export ADAPTER_MODEL="username/my-finetuned-model"
export BASE_MODEL="Qwen/Qwen2.5-0.5B"
export OUTPUT_REPO="username/my-model-gguf"
export HF_USERNAME="username"

```

## Step-by-Step Guide: Convert Fine-Tuned Models to GGUF

### Step 1: Load and Merge the LoRA Adapter

The script loads the base model specified in `BASE_MODEL` and the LoRA adapter from `ADAPTER_MODEL` using the `peft` library. It merges the adapter weights into the base model to create a single, standalone checkpoint. This merged model is temporarily saved to disk before conversion.

### Step 2: Install Build Tools and Clone llama.cpp

Unlike simpler workflows, this implementation installs system dependencies (`gcc`, `cmake`) first, then clones the [`llama.cpp`](https://github.com/huggingface/skills/blob/main/llama.cpp) repository. This sequencing is critical because the quantisation tools must be built from source.

```bash

# The script handles this automatically, but the equivalent manual steps are:

sudo apt-get update && sudo apt-get install -y git gcc cmake
git clone https://github.com/ggerganov/llama.cpp.git

```

### Step 3: Build llama-quantize with CMake

The script uses CMake (the recommended build system) to compile the `llama-quantize` binary. This tool is essential for converting FP16 GGUF files into quantized formats like `Q4_K_M` and `Q8_0`.

```bash
cd llama.cpp && cmake -B build && cmake --build build --config Release

```

### Step 4: Convert to FP16 GGUF Format

Using the [`convert_hf_to_gguf.py`](https://github.com/huggingface/skills/blob/main/convert_hf_to_gguf.py) script from the [`llama.cpp`](https://github.com/huggingface/skills/blob/main/llama.cpp) repository, the merged Hugging Face checkpoint is converted to a full-precision FP16 GGUF file. This serves as the source for all quantized variants.

### Step 5: Quantize to Multiple GGUF Formats

The script invokes the compiled `llama-quantize` tool to produce several quantization levels:

- **`Q4_K_M`**: 4-bit quantization with medium complexity (balanced size/quality)
- **`Q5_K_M`**: 5-bit quantization for higher quality
- **`Q8_0`**: 8-bit quantization for maximum accuracy with larger file size
- **`F16`**: Full FP16 precision (no quantization)

### Step 6: Upload to Hugging Face Hub

Finally, the script generates a comprehensive [`README.md`](https://github.com/huggingface/skills/blob/main/README.md) documenting the model provenance, quantization options, and usage examples. All GGUF variants and the README are uploaded to the repository specified in `OUTPUT_REPO`.

## Running the Conversion as a Hugging Face Job

For large models or automated pipelines, submit the conversion as a Hugging Face Job instead of running locally:

```python
from huggingface_hub import HfApi

hf_api = HfApi()
hf_api.create_repo(repo_id="username/my-model-gguf", repo_type="model", exist_ok=True)

job = {
    "script": open("skills/hugging-face-model-trainer/scripts/convert_to_gguf.py").read(),
    "flavor": "a10g-large",
    "timeout": "45m",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
    "env": {
        "ADAPTER_MODEL": "username/my-finetuned-model",
        "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
        "OUTPUT_REPO": "username/my-model-gguf",
        "HF_USERNAME": "username",
    },
}

# Submit via your preferred job runner

```

Verify that both `ADAPTER_MODEL` and `BASE_MODEL` exist on the Hub before submission using `hf_api.model_info()`.

## Using GGUF Models with Ollama and llama.cpp

### Load GGUF in Ollama

After converting fine-tuned models to GGUF format, deploy them locally with Ollama:

```bash

# Download the quantized model

huggingface-cli download username/my-model-gguf model-q4_k_m.gguf

# Create a Modelfile

echo "FROM ./model-q4_k_m.gguf" > Modelfile

# Build and run

ollama create my-gguf-model -f Modelfile
ollama run my-gguf-model

```

### Run Inference with llama.cpp

Use the compiled `llama-cli` binary for direct inference:

```bash

# CPU inference

./llama.cpp/build/bin/llama-cli -m model-q4_k_m.gguf -p "Explain quantization"

# GPU-accelerated inference (if CUDA available)

./llama.cpp/build/bin/llama-cli -m model-q4_k_m.gguf -ngl 32 -p "Explain quantization"

```

### Import into LM Studio

1. Download the desired `.gguf` file from your Hugging Face repository.
2. Open LM Studio and navigate to **Import Model**.
3. Select the GGUF file to load your fine-tuned model for local chat sessions.

## Summary

- **GGUF conversion** requires merging LoRA adapters with base models before quantization.
- The [`convert_to_gguf.py`](https://github.com/huggingface/skills/blob/main/convert_to_gguf.py) script in `huggingface/skills` automates the entire pipeline: dependency installation, CMake builds, FP16 conversion, and multi-format quantization.
- **CMake** is the recommended build system for compiling `llama-quantize`, ensuring reliable quantization across platforms.
- The script generates **Q4_K_M**, **Q5_K_M**, **Q8_0**, and **F16** variants to balance speed, size, and accuracy.
- Converted models work immediately with **Ollama**, **llama.cpp**, and **LM Studio** without additional configuration.

## Frequently Asked Questions

### What is the difference between GGUF and Safetensors formats?

**Safetensors** is a safe, fast serialization format used by Hugging Face Transformers to store model weights during training and inference. **GGUF** is a binary format specifically designed for local inference engines like llama.cpp and Ollama, optimized for memory-mapped file access and supporting various quantization schemes. You must convert from Safetensors (or PyTorch binaries) to GGUF to run models in local inference tools.

### Why must I install build tools before cloning llama.cpp?

The `llama-quantize` binary required for creating quantized GGUF files must be compiled from C++ source code using **CMake** and **GCC**. The [`convert_to_gguf.py`](https://github.com/huggingface/skills/blob/main/convert_to_gguf.py) script installs these system dependencies before cloning the repository to ensure the subsequent CMake build succeeds. Attempting to build without these tools results in compilation errors when generating the quantization binaries.

### Which quantization format should I choose for Ollama?

For most use cases with Ollama, **Q4_K_M** (4-bit medium complexity) offers the best balance between model size and output quality, reducing VRAM requirements while maintaining coherent generation. If you have ample GPU memory and require higher fidelity for complex reasoning tasks, use **Q8_0** (8-bit). The **F16** format provides full precision but requires significantly more storage and RAM, making it suitable only for high-end workstations.

### Can I convert models without using the Hugging Face Hub?

Yes, while the [`convert_to_gguf.py`](https://github.com/huggingface/skills/blob/main/convert_to_gguf.py) script is optimized for Hugging Face Hub workflows, you can adapt it for local-only conversion by modifying the model loading paths. Load your local base model and adapter using `AutoModelForCausalLM.from_pretrained()` and `PeftModel.from_pretrained()` with local directory paths, then proceed with the local CMake build and quantization steps. However, you will need to manually handle the GGUF file management and README generation that the script normally automates for Hub uploads.