How the llmfit Dynamic Quantization Selection Algorithm Works

The llmfit dynamic quantization selection algorithm uses a greedy two-step process to select the highest-quality quantization format that fits within a specified memory budget, automatically falling back to reduced context lengths when necessary.

The llmfit tool by AlexsJones automates large language model deployment by dynamically choosing optimal quantization formats at runtime. The llmfit dynamic quantization selection algorithm ensures users receive maximum model fidelity without exceeding available GPU VRAM or system RAM constraints. Implemented in Rust, this logic operates transparently within llmfit-core/src/models.rs to balance quality against hardware limitations.

Core Algorithm Mechanics

The selection logic resides in the LlmModel struct and centers on two primary methods: best_quant_for_budget and best_quant_for_budget_with. These functions implement a quality-first greedy search that prioritizes model fidelity while respecting memory constraints.

Quantization Hierarchy Definition

At the heart of the algorithm lies the QUANT_HIERARCHY constant defined in llmfit-core/src/models.rs. This array establishes a quality-ordered preference list from highest to lowest fidelity:

• Q8_0 (highest quality, largest footprint)
• Q6_K
• Q5_K_M
• Q4_K_M
• Progressively lower bit-depth formats

The algorithm always attempts to select the leftmost (highest quality) format before considering more aggressive compression options.

Memory-Budget Validation

For each candidate quantization in the hierarchy, the algorithm calls estimate_memory_gb to calculate total memory consumption. This function derives the footprint from the model's parameter count multiplied by the per-parameter byte size (quant_bpp) for the specific quantization, plus the KV-cache requirements. The first quantization whose estimated usage falls within the user-specified budget (in GB) is immediately selected and returned.

Context Length Fallback

If no quantization format fits the initial budget, the algorithm implements an automatic fallback mechanism. It halves the requested context length—provided the result remains at least 1024 tokens—and repeats the hierarchy scan from the top. This trade-off allows larger models to load by sacrificing context window size rather than forcing a load failure.

Extensibility via Custom Hierarchies

The best_quant_for_budget_with method provides flexibility for alternative backends. Callers can supply custom hierarchies—such as an MLX-specific list for Apple Silicon—while reusing the same greedy selection logic and memory validation. This design decouples the selection algorithm from specific quantization schemes, enabling experimental or backend-specific formats without modifying core logic.

Memory Estimation Implementation

Accurate selection depends on precise memory modeling implemented in llmfit-core/src/models.rs. The estimate_memory_gb function calculates consumption from three primary factors:

• Model weights: Total parameters multiplied by quant_bpp (bytes per parameter for the target quantization)
• KV cache: Size computed via kv_cache_gb based on context length and KV quantization settings
• System overhead: Fixed buffers and temporary allocations required for inference

Practical Usage Examples

The following example demonstrates dynamic selection for a 12 GB VRAM budget with a 4096-token context:

use llmfit_core::models::LlmModel;

// Model entry loaded from the embedded catalog
let model: LlmModel = ...;
let budget_gb = 12.0;
let ctx_len = 4096;

// Dynamically select the best quantization
if let Some((quant, mem)) = model.best_quant_for_budget(budget_gb, ctx_len) {
    println!("Selected quantization: {quant} (≈ {mem:.2} GB)");
} else {
    println!("No quantization fits – consider reducing context length or increasing budget.");
}

To use a custom hierarchy (e.g., MLX-specific formats):

let mlx_hierarchy = &["mlx-8bit", "mlx-4bit"]; // Custom hierarchy slice
if let Some((quant, mem)) = model.best_quant_for_budget_with(budget_gb, ctx_len, mlx_hierarchy) {
    println!("MLX-compatible quant: {quant} ({mem:.2} GB)");
}

Summary

• The algorithm prioritizes quality by scanning QUANT_HIERARCHY from highest to lowest fidelity in llmfit-core/src/models.rs.
• LlmModel::best_quant_for_budget selects the first format where estimate_memory_gb returns a value within the specified budget.
• If no fit exists, the system automatically halves the context length (minimum 1024 tokens) and retries the scan.
• Memory calculations incorporate parameter count, quant_bpp (bytes per parameter), and KV-cache requirements via kv_cache_gb.
• Custom quantization ladders are supported via best_quant_for_budget_with for backend-specific implementations.

Frequently Asked Questions

What happens if no quantization fits even after reducing context length?

If the algorithm exhausts all options in the hierarchy and the context length cannot be reduced below 1024 tokens, best_quant_for_budget returns None. This indicates the model cannot run within the specified hardware constraints, requiring either a larger memory budget or a smaller base model.

How does llmfit calculate memory usage for different quantizations?

The estimate_memory_gb function multiplies the model's total parameter count by the quantization-specific quant_bpp (bytes per parameter) and adds the KV-cache size computed by kv_cache_gb. This accounts for the requested context length and selected KV quantization format, providing an accurate total memory footprint.

Can I use this algorithm with custom quantization formats?

Yes. The best_quant_for_budget_with method accepts any string slice hierarchy, allowing you to define custom quantization ladders (such as MLX-specific formats) while retaining the same greedy selection logic and memory-checking implementation.

Why does the algorithm prefer higher-quality quantizations first?

The QUANT_HIERARCHY is ordered by fidelity to maximize model accuracy and coherence. By scanning from Q8_0 downward, llmfit ensures users receive the best possible quality their hardware can support, only falling back to aggressive compression when necessary to satisfy memory constraints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}