# How WhichLLM Accounts for Quantization When Ranking Models: A Technical Deep Dive

> Discover how WhichLLM ranks models by accounting for quantization. Learn about variant discovery, weight estimation, and quality penalty application for precise scoring.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: deep-dive
- Published: 2026-06-09

---

**WhichLLM incorporates quantization into its ranking pipeline through three distinct stages: variant discovery and filtering, quantization type resolution with weight estimation, and quality penalty application that adjusts final scores based on precision loss.**

This open-source LLM selection engine evaluates large language models by weighing benchmark performance against practical deployment constraints. Understanding **how WhichLLM accounts for quantization** is essential for users who need to balance model capability with hardware limitations, as the tool automatically penalizes lower-precision formats to reflect their inherent quality trade-offs.

## The Three-Stage Quantization Pipeline

The codebase implements quantization-aware ranking through a structured pipeline defined in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) and [`src/whichllm/engine/quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/quantization.py). Each stage progressively narrows candidate models and adjusts their scores to reflect the real-world impact of quantized weights.

### Stage 1: Variant Discovery and Filtering

The process begins in `_iter_candidate_variants` (lines 46-74 of [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py)), where WhichLLM detects the quantization format of each candidate model. The system distinguishes between **GGUF formats** (native to llama.cpp) and legacy formats such as **AWQ**, **GPTQ**, and **BNB-4bit**.

During this phase, the engine filters candidates based on the user-specified `--quant` flag. Extreme ultra-low-bit formats like **Q1_0** or **Q2_K** are automatically excluded unless explicitly requested, preventing users from accidentally selecting models with catastrophic quality degradation.

### Stage 2: Quantization Type Resolution and Weight Estimation

Once variants are identified, `_effective_quant_type` (line 53 of [`quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/quantization.py)) determines the precise quantization type for each model-variant pair. For GGUF files, this extracts the quant type directly from the variant metadata; for non-GGUF repositories, it falls back to `infer_non_gguf_quant_type(model.id)` (lines 54-58).

Simultaneously, `_estimate_weight_bytes` (line 60 of [`quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/quantization.py)) calculates the on-disk weight size when physical GGUF files are absent. This estimation feeds directly into VRAM compatibility checks, ensuring the hardware layer can validate whether a quantized model will actually fit within available GPU memory.

### Stage 3: Quality Penalty and Final Scoring

The critical adjustment occurs in `_quant_quality_penalty` (line 70 of [`quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/quantization.py)), which applies empirically-derived quality penalties before final ranking. The function consults the global `QUANT_QUALITY_PENALTY` constant (defined in [`constants.py`](https://github.com/Andyyyy64/whichllm/blob/main/constants.py)) for known GGUF formats, then falls back to `_NON_GGUF_QUALITY_PENALTY` (lines 33-41) for legacy formats.

These penalties directly modify the scoring formula in `_compute_quality_score` (lines 13-15 of [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py)). The code multiplies the combined benchmark and size score by `(1 - quant_penalty)`, meaning a 5% penalty on a 90-point base score reduces the contribution to 85.5 points before speed or popularity bonuses are applied.

## How Quantization Detection Works

For non-GGUF repositories, WhichLLM employs pattern-based inference through `_NON_GGUF_PATTERNS` at the top of [`quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/quantization.py) (lines 11-19). This regex-based system analyzes the repository ID to detect quantization schemes:

- **AWQ** and **GPTQ** variants incur a 5% quality penalty
- **BNB_4BIT**, **INT8**, **FP8**, and **BF16** receive format-specific penalties
- Unmatched repositories default to **FP16** with zero penalty

This detection mechanism ensures that models quantized through different toolchains receive appropriate score adjustments even when GGUF metadata is unavailable.

## Synthetic Variant Generation

When official model repositories lack GGUF files entirely, WhichLLM prevents size-score inflation by synthesizing realistic variants. The `_synthesize_variants_for_official_repo` function (lines 9-25 of [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py)) creates artificial **Q4_K_M** and **Q8_0** variants.

This synthesis allows the ranking algorithm to evaluate size/quality trade-offs against realistic quantized baselines rather than comparing against the inflated FP16 uncompressed size. Without this step, unquantized models would receive artificially high size-efficiency scores, distorting the final rankings.

## Impact on Final Scoring

The quantization penalty integrates into the quality score through a multiplicative factor. Lower-precision formats face specific penalties:

- **AWQ** and **GPTQ**: 5% quality reduction
- **Q1_0** (extreme quantization): 5% quality reduction via `_NON_GGUF_QUALITY_PENALTY`
- **Standard GGUF variants**: Penalties defined in `QUANT_QUALITY_PENALTY` table

This approach quantifies the fidelity loss inherent in weight compression, ensuring that rankings reflect not just raw benchmark performance but the practical output quality users can expect from quantized deployments.

## Code Examples

Detecting quantization types and penalties programmatically:

```python
from whichllm.engine.quantization import (
    effective_quant_type,
    quant_quality_penalty,
)

# Given a ModelInfo `model` and an optional GGUFVariant `variant`

qt = effective_quant_type(model, variant)               # e.g. "Q4_K_M"

penalty = quant_quality_penalty(model, variant)        # e.g. 0.05 (5% quality loss)

print(f"Model {model.id} uses {qt} quantization → quality penalty {penalty:.0%}")

```

Running rankings with quantization filters:

```python
from whichllm.engine.ranker import rank_models

# Rank models for a user's GPU, forcing only Q4_K_M quantization

results = rank_models(
    models=all_models,
    hardware=user_hardware,
    quant_filter="q4_k_m",          # Only variants with quant_type.upper() == "Q4_K_M"

    top_n=5,
)

for r in results:
    print(f"{r.model.name} ({r.variant.quant_type if r.variant else 'FP16'}): "
          f"Score {r.quality_score:.1f}, Penalty {r.variant.quant_type if r.variant else 'N/A'}")

```

## Summary

- WhichLLM evaluates quantization through three pipeline stages: variant filtering in `_iter_candidate_variants`, type resolution via `_effective_quant_type`, and penalty application through `_quant_quality_penalty`.
- The system supports both GGUF formats and legacy quantized formats (AWQ, GPTQ, BNB-4bit) through regex-based pattern matching in `_NON_GGUF_PATTERNS`.
- Synthetic variants (Q4_K_M and Q8_0) are generated for official repositories lacking GGUF files to ensure accurate size/quality comparisons.
- Quality penalties are multiplicative factors applied in `_compute_quality_score`, with AWQ and GPTQ typically incurring a 5% reduction.
- All quantization metadata flows through [`src/whichllm/models/types.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/types.py) data classes, ensuring consistent handling across the ranking engine.

## Frequently Asked Questions

### How does WhichLLM handle models without GGUF files?

When a repository lacks GGUF variants, WhichLLM invokes `_synthesize_variants_for_official_repo` in [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py) (lines 9-25) to create artificial Q4_K_M and Q8_0 variants. This prevents the ranking algorithm from using the uncompressed FP16 size as a baseline, ensuring realistic size-efficiency calculations for official model releases that haven't been converted to GGUF format yet.

### What quantization formats receive the highest quality penalties?

According to the `_NON_GGUF_QUALITY_PENALTY` table in [`quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/quantization.py) (lines 33-41), extreme formats like **Q1_0** incur significant penalties comparable to legacy formats like AWQ and GPTQ (typically 5%). Standard 4-bit GGUF variants such as Q4_K_M have minimal or zero penalties defined in `QUANT_QUALITY_PENALTY`, reflecting their superior quality-to-size efficiency in empirical benchmarks.

### Can users filter rankings by specific quantization types?

Yes, the `--quant` flag filters candidates during the `_iter_candidate_variants` phase in [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py) (lines 46-74). Users can specify exact formats like `q4_k_m` or `q8_0`, and the engine will exclude non-matching variants. The system also automatically drops ultra-low-bit formats (Q1_0, Q2_K) unless explicitly requested, protecting users from accidentally selecting heavily degraded models.

### Where does the quality penalty actually affect the score calculation?

The penalty applies in `_compute_quality_score` within [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py) (lines 13-15), where the code calculates `(1 - quant_penalty) * (benchmark_score + size_score)`. This multiplication occurs before speed, popularity, or lineage bonuses are added, ensuring that quantization affects the core quality metric while preserving the relative impact of other ranking factors.