# LLMFit Quantization Options: Supported Formats and Dynamic Selection Methods

> Explore LLMFit quantization options including GGUF MLX ONNX AWQ and GPTQ. Discover dynamic selection methods driven by memory budget for efficient model loading.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-20

---

**LLMFit supports GGUF quantizations (Q8_0 through Q2_K), MLX-native formats (8bit/4bit), ONNX variants (Q8_0/Q4_0), and pre-quantized AWQ/GPTQ models, with automatic selection via memory-budget-driven algorithms in [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs).**

LLMFit is a Rust-based CLI and API toolkit for fitting large language models to available hardware. The **quantization options** you can use—and how the system chooses between them—are governed by a sophisticated hierarchy defined in the core model handling code. This article breaks down every supported format and the exact selection logic used when you run a fit operation.

---

## Quantization Hierarchies in LLMFit

The [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) file defines three distinct **quantization hierarchies** for different runtime environments. Each hierarchy orders formats from highest quality to most compressed.

### General GGUF Hierarchy

The default `QUANT_HIERARCHY` applies to most models and covers standard GGUF quantization levels:

```rust
["Q8_0", "Q6_K", "Q5_K_M", "Q4_K_M", "Q3_K_M", "Q2_K"]

```

This array appears at lines 3–6 of [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs) and serves as the fallback for any model not matching specialized runtimes.

### MLX-Native Hierarchy (Apple Silicon)

For Apple's MLX runtime on Apple Silicon devices, LLMFit uses a simplified two-level hierarchy defined in `MLX_QUANT_HIERARCHY` (lines 7–9):

```rust
["mlx-8bit", "mlx-4bit"]

```

These formats leverage Apple's unified memory architecture and neural engine optimizations.

### ONNX Catalog Hierarchy

Models from the ONNX seed catalog use a conservative quantization set (`ONNX_QUANT_HIERARCHY`, lines 10–12):

```rust
["Q8_0", "Q4_0"]

```

This reflects the limited quantization options available in upstream ONNX model repositories.

---

## Core Quantization Utilities

LLMFit implements four key functions in [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs) to evaluate quantization trade-offs. All accept a quantization string (e.g., `"Q4_K_M"` or `"mlx-4bit"`) and return metrics used by the selection algorithm.

| Function | Purpose | Lines |
|----------|---------|-------|
| `quant_bpp` | Returns **bytes per parameter** for size and memory estimates | 14–37 |
| `quant_speed_multiplier` | Provides a **speed factor** where lower quality = faster inference | 40–61 |
| `quant_quality_penalty` | Applies a **quality penalty** for lower-bit quantizations in scoring | 89–115 |
| `quant_bytes_per_param` | Alias metric used by the bandwidth estimator | 64–85 |

These utilities enable precise modeling of the size/speed/quality trade-off for any quantization level.

---

## Automatic Quantization Selection

When you don't specify a quantization manually, LLMFit uses **budget-driven dynamic selection** via `LlmModel::best_quant_for_budget` and its generic variant `best_quant_for_budget_with`.

### Algorithm Implementation

Located at lines 68–86 of [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs), the selection logic works as follows:

1. Iterates through the appropriate hierarchy (General, MLX, or ONNX)
2. Calls `estimate_memory_gb` using `quant_bpp` and KV-cache estimation
3. Returns the first quantization where `estimated_memory ≤ budget_gb`
4. If no quantization fits, optionally halves context length and retries (lines 77–84)

The memory estimate combines **model weight size** (via `quant_bpp`) and **KV cache size** (`estimate_memory_gb_with_kv` → `kv_cache_gb`) for the requested context length.

### Usage Example

```rust
// Find best quantization for 12 GB VRAM budget
let db = llmfit_core::ModelDatabase::default();
let model = db.models.iter()
    .find(|m| m.name == "Meta-Llama-3.1-8B")
    .expect("model not found");

let ctx = 4096;
let budget_gb = 12.0;

if let Some((quant, mem)) = model.best_quant_for_budget(budget_gb, ctx) {
    println!("Selected quant: {quant} (≈ {mem:.2f} GB memory)");
} else {
    println!("No quant fits budget – consider reducing context length");
}

```

---

## Manual Quantization Override

The CLI and TUI layers allow **explicit quantization selection** via the `--quantization` flag. In [`llmfit-tui/src/main.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/main.rs), this is defined as:

```rust
/// Specific GGUF quantization to download (e.g. "Q4_K_M", "Q8_0").
#[clap(long)]
quantization: Option<String>,

```

When provided, the string is validated against available quantizations for the selected model. The validation logic in [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs) (lines 1049–1055) strips suffixes and checks against the appropriate hierarchy.

### CLI Example

```bash

# Force Q8_0 quantization explicitly

llmfit fit --model "Meta-Llama-3.1-8B" --quantization Q8_0 --ctx 8192

```

---

## Pre-Quantized Model Formats

Some model formats cannot be re-quantized on-the-fly. The `ModelFormat` enum in [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs) (lines 21–27) marks these with `is_prequantized`:

- **AWQ** — Activation-aware weight quantization
- **GPTQ** — Post-training quantization for GPT models
- **AutoRound** — Rounding-based quantization scheme

For pre-quantized models, `LlmModel::is_prequantized` (lines 31–33) returns true, and the quantization string from the catalog is used **as-is** without dynamic selection.

```rust
// Check if model format prevents re-quantization
if model.is_prequantized() {
    println!("{} is pre-quantized (format = {:?})", 
             model.name, model.format);
}

```

---

## How Selection Propagates Through the Stack

The same quantization logic operates consistently across all LLMFit interfaces:

1. **Catalog Loading** — `ModelDatabase::new` reads the `quantization` field from [`hf_models.json`](https://github.com/AlexsJones/llmfit/blob/main/hf_models.json) or ONNX metadata
2. **Fit Analysis** — `ModelFit::analyze` calls `best_quant_for_budget` for GPU-first runtimes when no user quantization is supplied
3. **Display Layers** — [`display.rs`](https://github.com/AlexsJones/llmfit/blob/main/display.rs) and [`tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/tui_ui.rs) show both `fit.best_quant` (selected) and `fit.model.quantization` (catalog default)

This unified approach ensures identical behavior whether you use the CLI, TUI, or HTTP API.

---

## Summary

- **GGUF quantizations**: Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K (General hierarchy)
- **MLX formats**: mlx-8bit, mlx-4bit (Apple Silicon only)
- **ONNX variants**: Q8_0, Q4_0 (limited catalog support)
- **Pre-quantized formats**: AWQ, GPTQ, AutoRound (no re-quantization possible)
- **Automatic selection**: Memory-budget algorithm in `best_quant_for_budget` using `quant_bpp` and KV-cache estimation
- **Manual override**: `--quantization` CLI flag with validation in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs)

---

## Frequently Asked Questions

### What quantization format gives the best quality in LLMFit?

**Q8_0** provides the highest quality among dynamically selectable quantizations, using 8-bit weights with no grouping. It appears first in all hierarchies and is selected when memory budget permits. According to the `quant_quality_penalty` function in [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs), Q8_0 incurs zero quality penalty compared to full precision.

### How does LLMFit handle quantization on Apple Silicon Macs?

LLMFit detects MLX-native models and uses the `MLX_QUANT_HIERARCHY` with **mlx-8bit** and **mlx-4bit** formats. These leverage Apple's unified memory and neural engine. The selection algorithm remains identical—only the available format list changes based on runtime detection.

### Can I force a specific quantization even if it exceeds my GPU memory?

The CLI accepts any valid quantization string via `--quantization`, but the **best-quant algorithm will reject it** if `estimate_memory_gb` exceeds your specified budget. You can bypass this by also increasing the budget or reducing context length. Pre-quantized models (AWQ/GPTQ) ignore the algorithm entirely and use their fixed quantization.

### Why does LLMFit sometimes halve my context length automatically?

When no quantization fits the memory budget at the requested context length, `best_quant_for_budget_with` implements a fallback strategy (lines 77–84): it halves the context and retries the hierarchy. This trade-off prioritizes model loading over maximum sequence length when hardware-constrained.