How Does whichllm Generate Synthetic GGUF Variants for LLMs? A Code-Level Breakdown

whichllm generates synthetic GGUF variants by estimating file sizes from parameter counts and bytes-per-weight constants when a model repository only provides Safetensors files, allowing the ranking engine to score models that lack real GGUF conversions.

The open-source tool whichllm solves the problem of missing GGUF conversions by simulating them. When a model repository ships only Safetensors weights, the engine synthesizes plausible GGUF variants so they can still be ranked for size, speed, and quality. This article explains exactly how the Andyyyy64/whichllm ranking pipeline generates these synthetic estimates directly from the source code.

Detecting Missing GGUF Files in the Ranker

In src/whichllm/engine/ranker.py, the method _iter_candidate_variants inspects model.gguf_variants to decide whether to use real or synthetic data. If that list is empty, the engine falls back to synthetic generation at lines 150–156. This fallback ensures that Safetensors-only models are not silently discarded during ranking.

Eligibility Filters for Synthetic Generation

Official Organization Check

The helper _synthesize_variants_for_official_repo first verifies that the model owner is trusted. It splits model.id and checks the namespace against _OFFICIAL_ORGS, defined in src/whichllm/constants.py. If the namespace is not on the allow-list, synthetic generation is skipped entirely.

Pre-Quantized Repository Exclusion

The same function also excludes repositories that already advertise pre-quantized formats such as -AWQ or -GPTQ. A regex _PREQUANTIZED_REPO_RE filters these out to avoid creating redundant synthetic estimates, as seen in lines 109–125 of ranker.py.

Choosing Quantization Levels and Estimating Sizes

Hard-Coded Synthetic Quant Types

whichllm uses a fixed tuple _SYNTHETIC_QUANTS = ("Q3_K_M", "Q4_K_M", "Q5_K_M", "Q6_K", "Q8_0") to represent the quantization profiles the community typically publishes shortly after release. This tuple is defined near lines 102–103 in src/whichllm/engine/ranker.py.

Parameter-to-Size Calculation

For each quant type, the approximate file size is computed as model.parameter_count * QUANT_BYTES_PER_WEIGHT[quant]. The mapping QUANT_BYTES_PER_WEIGHT lives in src/whichllm/constants.py and defaults to 0.5625 bytes per weight when a specific quant is not mapped. This calculation occurs at lines 135–141 of the ranker.

Emitting and Scoring Synthetic Variants

The GGUFVariant Dataclass

Each synthetic result is wrapped in a GGUFVariant object defined in src/whichllm/models/types.py. The dataclass stores filename, quant_type, and file_size_bytes.

Integration into the Ranking Pipeline

After generation, synthetic variants flow through the same compatibility and performance scoring as real GGUF files. The engine marks them with a note via logic in src/whichllm/engine/performance.py—specifically _looks_synthetic_gguf—so users know the values are estimates rather than measured binaries.

Code Walkthrough: _synthesize_variants_for_official_repo

Here is the core logic from src/whichllm/engine/ranker.py that produces synthetic variants:

def _synthesize_variants_for_official_repo(model: ModelInfo,
                                            quant_filter_upper: str | None) -> list[GGUFVariant]:
    # ① Only official orgs & not already pre-quantized

    if model.id.split("/", 1)[0] not in _OFFICIAL_ORGS:
        return []
    if _PREQUANTIZED_REPO_RE.search(model.id):
        return []

    variants = []
    for quant in _SYNTHETIC_QUANTS:
        if quant_filter_upper and quant != quant_filter_upper:
            continue
        # ② Approximate size = params × bytes-per-weight

        bpw = QUANT_BYTES_PER_WEIGHT.get(quant, 0.5625)
        variants.append(
            GGUFVariant(
                filename=f"{model.name}.{quant}.gguf",
                quant_type=quant,
                file_size_bytes=int(model.parameter_count * bpw),
            )
        )
    return variants

The caller, _iter_candidate_variants, substitutes this list whenever a model has no real GGUF files, ensuring the ranking pipeline never drops an otherwise runnable candidate.

Practical Usage Examples

Ranking a Model Without GGUF Files

When you pass a ModelInfo with an empty gguf_variants list to rank_models, whichllm automatically injects synthetic estimates:

from whichllm.engine.ranker import rank_models
from whichllm.models.types import ModelInfo
from whichllm.hardware.types import HardwareInfo

model = ModelInfo(
    id="Qwen/Qwen3.6-27B",
    family_id="qwen3-6b-gguf",
    name="Qwen3.6-27B",
    parameter_count=27_000_000_000,
    gguf_variants=[],
)

hw = HardwareInfo(os="linux", gpus=[...])
ranked = rank_models([model], hw, top_n=5)

# ranked[0].gguf_variant is a synthetic GGUFVariant, e.g.:

# GGUFVariant(filename='Qwen3.6-27B.Q4_K_M.gguf', quant_type='Q4_K_M', ...)

Generating Synthetic Variants Directly

You can also call the internal helper directly for testing or debugging:

from whichllm.engine.ranker import _synthesize_variants_for_official_repo
from whichllm.models.types import ModelInfo

model = ModelInfo(
    id="meta-llama/Llama-2-7B",
    family_id="llama-2-7b",
    name="Llama-2-7B",
    parameter_count=7_000_000_000,
)

synthetic = _synthesize_variants_for_official_repo(model, None)

# Returns list[GGUFVariant] for Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, and Q8_0

Key Files in the Synthetic GGUF Pipeline

• src/whichllm/engine/ranker.py — Hosts _synthesize_variants_for_official_repo and the _iter_candidate_variants fallback logic.
• src/whichllm/models/types.py — Defines the GGUFVariant dataclass.
• src/whichllm/engine/performance.py — Adds synthetic notes and adjusts speed uncertainty via _looks_synthetic_gguf.
• src/whichllm/engine/types.py — Supplies CompatibilityResult for ranking flow integration.
• src/whichllm/constants.py — Stores QUANT_BYTES_PER_WEIGHT, _OFFICIAL_ORGS, and _SYNTHETIC_QUANTS.

Summary

• whichllm creates synthetic GGUF variants when a repository only provides Safetensors files.
• Eligibility is restricted to official organizations and excludes pre-quantized repositories.
• File sizes are estimated using parameter_count * QUANT_BYTES_PER_WEIGHT with a default of 0.5625 bytes per weight.
• The fixed quant set ("Q3_K_M", "Q4_K_M", "Q5_K_M", "Q6_K", "Q8_0") covers common community conversions.
• Synthetic variants are wrapped in GGUFVariant objects and scored identically to real GGUF files, but flagged as estimates.

Frequently Asked Questions

What is a synthetic GGUF variant in whichllm?

A synthetic GGUF variant is a placeholder object that simulates a quantized GGUF file for a model that does not publish one. It is generated by the ranking engine so that size and compatibility constraints can still be evaluated as if a real conversion existed.

Which quantization types does whichllm synthesize?

The engine synthesizes five standard types defined in _SYNTHETIC_QUANTS: Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, and Q8_0. These reflect the most common post-release GGUF quantization profiles published by the community.

How accurate are the synthetic GGUF size estimates?

Accuracy depends on the QUANT_BYTES_PER_WEIGHT constant for each quant type, with a fallback default of 0.5625 bytes per weight. The estimate is deterministic and fast, but it remains an approximation until an actual GGUF conversion is published and measured.

Why does whichllm only generate synthetic variants for official repositories?

The helper _synthesize_variants_for_official_repo gates synthetic creation on the _OFFICIAL_ORGS list to reduce the risk of estimating invalid or misleading variants from unofficial forks. This policy keeps the ranking results focused on trusted model publishers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}