How llmfit Performs Dynamic Quantization for LLMs

llmfit automatically selects the optimal quantization level that fits within available hardware memory by scanning predefined hierarchies of quantization formats and selecting the highest-quality option that satisfies the memory budget.

llmfit is a Rust-based CLI tool and library that automates the process of fitting large language models (LLMs) to specific hardware constraints. Its dynamic quantization for LLMs feature eliminates manual trial-and-error by programmatically selecting the best quantization format based on real-time memory analysis of the target system.

Understanding Quantization Hierarchies

The foundation of llmfit's dynamic quantization system rests on ordered lists of quantization formats, ranked from highest quality to most compact. These hierarchies are defined in llmfit-core/src/models.rs and vary by target runtime.

The general-purpose hierarchy spans from Q8_0 (highest fidelity) down to Q2_K (most compressed). For specialized runtimes, llmfit maintains distinct hierarchies:

• QUANT_HIERARCHY (lines 3‑6): The default sequence for general inference runtimes (e.g., Q8_0 → Q6_K → … → Q2_K)
• MLX_QUANT_HIERARCHY (lines 7‑9): Optimized for Apple Silicon via the MLX framework
• ONNX_QUANT_HIERARCHY (lines 10‑12): Tailored for ONNX Runtime deployments

These constants enable the system to prioritize quality while respecting hardware constraints, ensuring that the selection algorithm always prefers the highest-fidelity quant that will actually run on the target device.

Memory Budget Calculation

Before selecting a quantization format, llmfit calculates the available memory budget for the inference operation. During the analysis phase in llmfit-core/src/fit.rs, the system computes mem_available based on the selected run mode and detected hardware specifications.

This memory budget serves as the constraint parameter for the dynamic selection algorithm. The calculation accounts for the specific inference runtime (Llama.cpp, MLX, ONNX, etc.) and the current system state, providing a realistic upper bound for model weights and activation memory.

The Dynamic Selection Algorithm

The core logic resides in ModelFit::analyze within llmfit-core/src/fit.rs (lines 33‑57). When processing a model that is not pre-quantized (i.e., not AWQ, GPTQ, or AutoRound), the system executes the following decision tree:

let (best_quant, _best_quant_mem) = if model.is_prequantized() {
    (model.quantization.as_str(), mem_required)
} else {
    let budget = mem_available;
    // pick hierarchy based on model format / runtime
    let hierarchy = if model.format == models::ModelFormat::Onnx {
        models::ONNX_QUANT_HIERARCHY
    } else if runtime == InferenceRuntime::Mlx {
        models::MLX_QUANT_HIERARCHY
    } else {
        models::QUANT_HIERARCHY
    };
    // try to find a quant that fits the budget
    model
        .best_quant_for_budget_with(budget, estimation_ctx, hierarchy)
        .or_else(|| {
            // fallback to the default hierarchy for MLX when needed
            if runtime == InferenceRuntime::Mlx {
                model.best_quant_for_budget(budget, estimation_ctx)
            } else {
                None
            }
        })
        .unwrap_or((model.quantization.as_str(), mem_required))
};

Runtime-Specific Hierarchy Selection

The algorithm first determines which quantization hierarchy to use based on the inference context. ONNX models receive the ONNX-specific hierarchy, MLX runtimes on Apple Silicon use the MLX hierarchy, and all other configurations default to the general QUANT_HIERARCHY. This ensures that quantization choices align with the numerical stability and kernel optimizations available on each platform.

Pre-Quantized Model Handling

If model.is_prequantized() returns true, the system skips dynamic selection entirely. Pre-quantized models (AWQ, GPTQ, AutoRound) have fixed quantization schemes that cannot be altered without re-quantization, so llmfit respects the existing format and calculates memory requirements accordingly.

Budget-Aware Search Implementation

The actual search logic lives in best_quant_for_budget_with, implemented in llmfit-core/src/models.rs (lines 68‑85). This helper method iterates through the selected hierarchy, calculating memory requirements for each candidate format until it finds one that fits:

for &q in hierarchy {
    let mem = self.estimate_memory_gb(q, ctx);
    if mem <= budget_gb { return Some((q, mem)); }
}
// optionally try half-context
...

The function calls estimate_memory_gb for each quantization level in the hierarchy. It returns the first format where the estimated memory is less than or equal to the available budget. If no standard quantization fits, the system optionally attempts to halve the context length and retry the search—an escape hatch for fitting very large models into constrained VRAM.

Result Propagation and User Feedback

Once selected, the optimal quantization string is stored in ModelFit.best_quant and used for downstream calculations including memory estimates, inference speed projections, and compatibility scoring.

If the dynamically selected quantization differs from the model's default recommendation, llmfit generates a user-facing notification (lines 58‑66 in fit.rs):

if best_quant != model.quantization {
    notes.push(format!(…));
}

This results in CLI output such as:

Best Quant: Q4_K_M (default Q8_0)

CLI and Programmatic Usage

To observe dynamic quantization in action via the command line:

# Show fit for the 7B Llama model on the current machine

llmfit fit --model llama-2-7b

For Rust developers integrating llmfit as a library:

use llmfit_core::models::LlmModel;
use llmfit_core::fit::CalcConfig;

// Load a model (e.g., from the embedded catalog)
let model: LlmModel = llmfit_core::models::load("llama-2-7b");

// Create a dummy system spec (replace with real detection)
let system = llmfit_core::hardware::SystemSpecs::mock();

// Perform analysis – this runs the dynamic quant selection internally
let fit = llmfit_core::analysis::analyze_model(
    &model,
    &system,
    CalcConfig::default(),
    None, // use default context cap
);
println!("Chosen quant: {}", fit.best_quant);

You can also query the best quantization for a specific memory budget directly:

use llmfit_core::models::LlmModel;

let model = llmfit_core::models::load("phi-2");

// Suppose we have 8 GiB of VRAM available
let budget_gb = 8.0;
let best = model.best_quant_for_budget(budget_gb, 8192);
println!("Best quant for 8 GiB: {:?}", best);
// Output: Some(("Q4_K_M", 7.5))

Summary

• Dynamic quantization in llmfit automatically negotiates between quality and hardware constraints by selecting the highest-fidelity quantization format that fits within available memory.
• The system maintains three distinct hierarchies (QUANT_HIERARCHY, MLX_QUANT_HIERARCHY, ONNX_QUANT_HIERARCHY) in llmfit-core/src/models.rs to optimize for different runtimes.
• The selection algorithm in llmfit-core/src/fit.rs skips pre-quantized models (AWQ, GPTQ, AutoRound) and only runs for full-precision checkpoints.
• Budget-aware search iterates through quality-ranked formats, using estimate_memory_gb to find the first viable candidate, with optional context-length reduction as a fallback.
• Results are stored in ModelFit.best_quant and surfaced to users through CLI and programmatic interfaces.

Frequently Asked Questions

What is dynamic quantization in llmfit?

Dynamic quantization is llmfit's automated process for selecting the optimal GGUF quantization level (e.g., Q8_0, Q4_K_M, Q2_K) based on real-time hardware analysis. Instead of requiring users to manually specify a quantization format, the system calculates available VRAM, tests formats against a memory budget using best_quant_for_budget_with, and selects the highest-quality option that will fit.

How does llmfit handle pre-quantized models?

Pre-quantized models using formats like AWQ, GPTQ, or AutoRound bypass the dynamic selection algorithm entirely. In llmfit-core/src/fit.rs, the is_prequantized() check returns true for these models, causing the system to use their fixed quantization scheme rather than searching the hierarchy. This preserves the specialized compression and accuracy characteristics of these formats.

What quantization hierarchies does llmfit support?

The system supports three hierarchies defined in llmfit-core/src/models.rs: a general-purpose hierarchy for Llama.cpp-based inference (QUANT_HIERARCHY), an Apple Silicon-optimized hierarchy for MLX (MLX_QUANT_HIERARCHY), and an ONNX Runtime-specific hierarchy (ONNX_QUANT_HIERARCHY). Each orders formats from highest quality (Q8_0) to most compact (Q2_K) according to the numerical requirements of the respective runtime.

Can I override the automatic quantization selection?

While llmfit's analyze_model function automatically selects the best quantization, you can directly query specific options using the best_quant_for_budget or best_quant_for_budget_with methods on LlmModel. These allow you to specify exact memory budgets and even custom hierarchies programmatically, though the CLI fit command automatically handles this selection based on detected hardware specifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}