LLMFit Quantization Options: Supported Formats and Dynamic Selection Methods

LLMFit supports GGUF quantizations (Q8_0 through Q2_K), MLX-native formats (8bit/4bit), ONNX variants (Q8_0/Q4_0), and pre-quantized AWQ/GPTQ models, with automatic selection via memory-budget-driven algorithms in models.rs.

LLMFit is a Rust-based CLI and API toolkit for fitting large language models to available hardware. The quantization options you can use—and how the system chooses between them—are governed by a sophisticated hierarchy defined in the core model handling code. This article breaks down every supported format and the exact selection logic used when you run a fit operation.

---

Quantization Hierarchies in LLMFit

The llmfit-core/src/models.rs file defines three distinct quantization hierarchies for different runtime environments. Each hierarchy orders formats from highest quality to most compressed.

General GGUF Hierarchy

The default QUANT_HIERARCHY applies to most models and covers standard GGUF quantization levels:

["Q8_0", "Q6_K", "Q5_K_M", "Q4_K_M", "Q3_K_M", "Q2_K"]

This array appears at lines 3–6 of models.rs and serves as the fallback for any model not matching specialized runtimes.

MLX-Native Hierarchy (Apple Silicon)

For Apple's MLX runtime on Apple Silicon devices, LLMFit uses a simplified two-level hierarchy defined in MLX_QUANT_HIERARCHY (lines 7–9):

["mlx-8bit", "mlx-4bit"]

These formats leverage Apple's unified memory architecture and neural engine optimizations.

ONNX Catalog Hierarchy

Models from the ONNX seed catalog use a conservative quantization set (ONNX_QUANT_HIERARCHY, lines 10–12):

["Q8_0", "Q4_0"]

This reflects the limited quantization options available in upstream ONNX model repositories.

---

Core Quantization Utilities

LLMFit implements four key functions in models.rs to evaluate quantization trade-offs. All accept a quantization string (e.g., "Q4_K_M" or "mlx-4bit") and return metrics used by the selection algorithm.

Function	Purpose	Lines
quant_bpp	Returns bytes per parameter for size and memory estimates	14–37
quant_speed_multiplier	Provides a speed factor where lower quality = faster inference	40–61
quant_quality_penalty	Applies a quality penalty for lower-bit quantizations in scoring	89–115
quant_bytes_per_param	Alias metric used by the bandwidth estimator	64–85

These utilities enable precise modeling of the size/speed/quality trade-off for any quantization level.

---

Automatic Quantization Selection

When you don't specify a quantization manually, LLMFit uses budget-driven dynamic selection via LlmModel::best_quant_for_budget and its generic variant best_quant_for_budget_with.

Algorithm Implementation

Located at lines 68–86 of models.rs, the selection logic works as follows:

1. Iterates through the appropriate hierarchy (General, MLX, or ONNX)
2. Calls estimate_memory_gb using quant_bpp and KV-cache estimation
3. Returns the first quantization where estimated_memory ≤ budget_gb
4. If no quantization fits, optionally halves context length and retries (lines 77–84)

The memory estimate combines model weight size (via quant_bpp) and KV cache size (estimate_memory_gb_with_kv → kv_cache_gb) for the requested context length.

Usage Example

// Find best quantization for 12 GB VRAM budget
let db = llmfit_core::ModelDatabase::default();
let model = db.models.iter()
    .find(|m| m.name == "Meta-Llama-3.1-8B")
    .expect("model not found");

let ctx = 4096;
let budget_gb = 12.0;

if let Some((quant, mem)) = model.best_quant_for_budget(budget_gb, ctx) {
    println!("Selected quant: {quant} (≈ {mem:.2f} GB memory)");
} else {
    println!("No quant fits budget – consider reducing context length");
}

---

Manual Quantization Override

The CLI and TUI layers allow explicit quantization selection via the --quantization flag. In llmfit-tui/src/main.rs, this is defined as:

/// Specific GGUF quantization to download (e.g. "Q4_K_M", "Q8_0").
#[clap(long)]
quantization: Option<String>,

When provided, the string is validated against available quantizations for the selected model. The validation logic in llmfit-core/src/providers.rs (lines 1049–1055) strips suffixes and checks against the appropriate hierarchy.

CLI Example

# Force Q8_0 quantization explicitly

llmfit fit --model "Meta-Llama-3.1-8B" --quantization Q8_0 --ctx 8192

---

Pre-Quantized Model Formats

Some model formats cannot be re-quantized on-the-fly. The ModelFormat enum in models.rs (lines 21–27) marks these with is_prequantized:

• AWQ — Activation-aware weight quantization
• GPTQ — Post-training quantization for GPT models
• AutoRound — Rounding-based quantization scheme

For pre-quantized models, LlmModel::is_prequantized (lines 31–33) returns true, and the quantization string from the catalog is used as-is without dynamic selection.

// Check if model format prevents re-quantization
if model.is_prequantized() {
    println!("{} is pre-quantized (format = {:?})", 
             model.name, model.format);
}

---

How Selection Propagates Through the Stack

The same quantization logic operates consistently across all LLMFit interfaces:

1. Catalog Loading — ModelDatabase::new reads the quantization field from hf_models.json or ONNX metadata
2. Fit Analysis — ModelFit::analyze calls best_quant_for_budget for GPU-first runtimes when no user quantization is supplied
3. Display Layers — display.rs and tui_ui.rs show both fit.best_quant (selected) and fit.model.quantization (catalog default)

This unified approach ensures identical behavior whether you use the CLI, TUI, or HTTP API.

---

Summary

• GGUF quantizations: Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K (General hierarchy)
• MLX formats: mlx-8bit, mlx-4bit (Apple Silicon only)
• ONNX variants: Q8_0, Q4_0 (limited catalog support)
• Pre-quantized formats: AWQ, GPTQ, AutoRound (no re-quantization possible)
• Automatic selection: Memory-budget algorithm in best_quant_for_budget using quant_bpp and KV-cache estimation
• Manual override: --quantization CLI flag with validation in providers.rs

---

Frequently Asked Questions

What quantization format gives the best quality in LLMFit?

Q8_0 provides the highest quality among dynamically selectable quantizations, using 8-bit weights with no grouping. It appears first in all hierarchies and is selected when memory budget permits. According to the quant_quality_penalty function in models.rs, Q8_0 incurs zero quality penalty compared to full precision.

How does LLMFit handle quantization on Apple Silicon Macs?

LLMFit detects MLX-native models and uses the MLX_QUANT_HIERARCHY with mlx-8bit and mlx-4bit formats. These leverage Apple's unified memory and neural engine. The selection algorithm remains identical—only the available format list changes based on runtime detection.

Can I force a specific quantization even if it exceeds my GPU memory?

The CLI accepts any valid quantization string via --quantization, but the best-quant algorithm will reject it if estimate_memory_gb exceeds your specified budget. You can bypass this by also increasing the budget or reducing context length. Pre-quantized models (AWQ/GPTQ) ignore the algorithm entirely and use their fixed quantization.

Why does LLMFit sometimes halve my context length automatically?

When no quantization fits the memory budget at the requested context length, best_quant_for_budget_with implements a fallback strategy (lines 77–84): it halves the context and retries the hierarchy. This trade-off prioritizes model loading over maximum sequence length when hardware-constrained.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}