# How llmfit Performs Dynamic Quantization for LLMs

> Discover how llmfit achieves dynamic quantization for LLMs. llmfit automatically selects optimal quantization levels to fit hardware memory, maximizing quality within your budget.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-19

---

**llmfit automatically selects the optimal quantization level that fits within available hardware memory by scanning predefined hierarchies of quantization formats and selecting the highest-quality option that satisfies the memory budget.**

llmfit is a Rust-based CLI tool and library that automates the process of fitting large language models (LLMs) to specific hardware constraints. Its **dynamic quantization for LLMs** feature eliminates manual trial-and-error by programmatically selecting the best quantization format based on real-time memory analysis of the target system.

## Understanding Quantization Hierarchies

The foundation of llmfit's dynamic quantization system rests on ordered lists of quantization formats, ranked from highest quality to most compact. These hierarchies are defined in [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) and vary by target runtime.

The general-purpose hierarchy spans from `Q8_0` (highest fidelity) down to `Q2_K` (most compressed). For specialized runtimes, llmfit maintains distinct hierarchies:

- **`QUANT_HIERARCHY`** (lines 3‑6): The default sequence for general inference runtimes (e.g., `Q8_0 → Q6_K → … → Q2_K`)
- **`MLX_QUANT_HIERARCHY`** (lines 7‑9): Optimized for Apple Silicon via the MLX framework
- **`ONNX_QUANT_HIERARCHY`** (lines 10‑12): Tailored for ONNX Runtime deployments

These constants enable the system to prioritize quality while respecting hardware constraints, ensuring that the selection algorithm always prefers the highest-fidelity quant that will actually run on the target device.

## Memory Budget Calculation

Before selecting a quantization format, llmfit calculates the available memory budget for the inference operation. During the analysis phase in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs), the system computes `mem_available` based on the selected run mode and detected hardware specifications.

This memory budget serves as the constraint parameter for the dynamic selection algorithm. The calculation accounts for the specific inference runtime (Llama.cpp, MLX, ONNX, etc.) and the current system state, providing a realistic upper bound for model weights and activation memory.

## The Dynamic Selection Algorithm

The core logic resides in `ModelFit::analyze` within [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) (lines 33‑57). When processing a model that is **not** pre-quantized (i.e., not AWQ, GPTQ, or AutoRound), the system executes the following decision tree:

```rust
let (best_quant, _best_quant_mem) = if model.is_prequantized() {
    (model.quantization.as_str(), mem_required)
} else {
    let budget = mem_available;
    // pick hierarchy based on model format / runtime
    let hierarchy = if model.format == models::ModelFormat::Onnx {
        models::ONNX_QUANT_HIERARCHY
    } else if runtime == InferenceRuntime::Mlx {
        models::MLX_QUANT_HIERARCHY
    } else {
        models::QUANT_HIERARCHY
    };
    // try to find a quant that fits the budget
    model
        .best_quant_for_budget_with(budget, estimation_ctx, hierarchy)
        .or_else(|| {
            // fallback to the default hierarchy for MLX when needed
            if runtime == InferenceRuntime::Mlx {
                model.best_quant_for_budget(budget, estimation_ctx)
            } else {
                None
            }
        })
        .unwrap_or((model.quantization.as_str(), mem_required))
};

```

### Runtime-Specific Hierarchy Selection

The algorithm first determines which quantization hierarchy to use based on the inference context. ONNX models receive the ONNX-specific hierarchy, MLX runtimes on Apple Silicon use the MLX hierarchy, and all other configurations default to the general `QUANT_HIERARCHY`. This ensures that quantization choices align with the numerical stability and kernel optimizations available on each platform.

### Pre-Quantized Model Handling

If `model.is_prequantized()` returns `true`, the system skips dynamic selection entirely. Pre-quantized models (AWQ, GPTQ, AutoRound) have fixed quantization schemes that cannot be altered without re-quantization, so llmfit respects the existing format and calculates memory requirements accordingly.

## Budget-Aware Search Implementation

The actual search logic lives in `best_quant_for_budget_with`, implemented in [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) (lines 68‑85). This helper method iterates through the selected hierarchy, calculating memory requirements for each candidate format until it finds one that fits:

```rust
for &q in hierarchy {
    let mem = self.estimate_memory_gb(q, ctx);
    if mem <= budget_gb { return Some((q, mem)); }
}
// optionally try half-context
...

```

The function calls `estimate_memory_gb` for each quantization level in the hierarchy. It returns the first format where the estimated memory is less than or equal to the available budget. If no standard quantization fits, the system optionally attempts to halve the context length and retry the search—an escape hatch for fitting very large models into constrained VRAM.

## Result Propagation and User Feedback

Once selected, the optimal quantization string is stored in `ModelFit.best_quant` and used for downstream calculations including memory estimates, inference speed projections, and compatibility scoring. 

If the dynamically selected quantization differs from the model's default recommendation, llmfit generates a user-facing notification (lines 58‑66 in [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs)):

```rust
if best_quant != model.quantization {
    notes.push(format!(…));
}

```

This results in CLI output such as:

```

Best Quant: Q4_K_M (default Q8_0)

```

## CLI and Programmatic Usage

To observe dynamic quantization in action via the command line:

```bash

# Show fit for the 7B Llama model on the current machine

llmfit fit --model llama-2-7b

```

For Rust developers integrating llmfit as a library:

```rust
use llmfit_core::models::LlmModel;
use llmfit_core::fit::CalcConfig;

// Load a model (e.g., from the embedded catalog)
let model: LlmModel = llmfit_core::models::load("llama-2-7b");

// Create a dummy system spec (replace with real detection)
let system = llmfit_core::hardware::SystemSpecs::mock();

// Perform analysis – this runs the dynamic quant selection internally
let fit = llmfit_core::analysis::analyze_model(
    &model,
    &system,
    CalcConfig::default(),
    None, // use default context cap
);
println!("Chosen quant: {}", fit.best_quant);

```

You can also query the best quantization for a specific memory budget directly:

```rust
use llmfit_core::models::LlmModel;

let model = llmfit_core::models::load("phi-2");

// Suppose we have 8 GiB of VRAM available
let budget_gb = 8.0;
let best = model.best_quant_for_budget(budget_gb, 8192);
println!("Best quant for 8 GiB: {:?}", best);
// Output: Some(("Q4_K_M", 7.5))

```

## Summary

- **Dynamic quantization** in llmfit automatically negotiates between quality and hardware constraints by selecting the highest-fidelity quantization format that fits within available memory.
- The system maintains **three distinct hierarchies** (`QUANT_HIERARCHY`, `MLX_QUANT_HIERARCHY`, `ONNX_QUANT_HIERARCHY`) in [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) to optimize for different runtimes.
- The selection algorithm in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) skips pre-quantized models (AWQ, GPTQ, AutoRound) and only runs for full-precision checkpoints.
- **Budget-aware search** iterates through quality-ranked formats, using `estimate_memory_gb` to find the first viable candidate, with optional context-length reduction as a fallback.
- Results are stored in `ModelFit.best_quant` and surfaced to users through CLI and programmatic interfaces.

## Frequently Asked Questions

### What is dynamic quantization in llmfit?

Dynamic quantization is llmfit's automated process for selecting the optimal GGUF quantization level (e.g., Q8_0, Q4_K_M, Q2_K) based on real-time hardware analysis. Instead of requiring users to manually specify a quantization format, the system calculates available VRAM, tests formats against a memory budget using `best_quant_for_budget_with`, and selects the highest-quality option that will fit.

### How does llmfit handle pre-quantized models?

Pre-quantized models using formats like AWQ, GPTQ, or AutoRound bypass the dynamic selection algorithm entirely. In [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs), the `is_prequantized()` check returns `true` for these models, causing the system to use their fixed quantization scheme rather than searching the hierarchy. This preserves the specialized compression and accuracy characteristics of these formats.

### What quantization hierarchies does llmfit support?

The system supports three hierarchies defined in [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs): a general-purpose hierarchy for Llama.cpp-based inference (`QUANT_HIERARCHY`), an Apple Silicon-optimized hierarchy for MLX (`MLX_QUANT_HIERARCHY`), and an ONNX Runtime-specific hierarchy (`ONNX_QUANT_HIERARCHY`). Each orders formats from highest quality (Q8_0) to most compact (Q2_K) according to the numerical requirements of the respective runtime.

### Can I override the automatic quantization selection?

While llmfit's `analyze_model` function automatically selects the best quantization, you can directly query specific options using the `best_quant_for_budget` or `best_quant_for_budget_with` methods on `LlmModel`. These allow you to specify exact memory budgets and even custom hierarchies programmatically, though the CLI `fit` command automatically handles this selection based on detected hardware specifications.