How whichllm Groups and Deduplicates LLM Model Variants: A Deep Dive into the Source Code

whichllm groups and deduplicates LLM model variants by normalizing model identifiers into stable family keys, grouping first by base_model references and then by normalized names, and finally collapsing each group into a single ModelFamily object where benchmarks are merged and duplicates are hidden from CLI output.

The whichllm open-source project, as implemented in Andyyyy64/whichllm, tackles registry noise by ensuring that quantized files, instruction-tuned checkpoints, and re-uploaded prefixes of the same underlying LLM appear as one logical entry. To group and deduplicate LLM model variants, it runs a deterministic multi-pass pipeline implemented in src/whichllm/models/grouper.py with data structures defined in src/whichllm/models/types.py. The result is a concise list of ModelFamily instances that downstream code, including the CLI in src/whichllm/cli.py, renders for end users.

How Model Normalization Creates a Stable Family Key

At the heart of deduplication is the _normalize_name function inside src/whichllm/models/grouper.py. This method transforms a raw model ID into a deterministic key so that any two identifiers mapping to the same normalized name are treated as one family.

The function performs several cleaning steps:

1. Strips prefixes – Organization slugs like org/… and common meta-prefixes such as qwen_, meta-llama_, or google_ are removed.
2. Removes stacked suffixes – Tokens that denote quantization formats, chat/instruction variants, or date stamps are stripped in up to three passes to handle stacked suffixes such as -gguf-q4 or -instruct-2401.
3. Collapses versions and sizes – Patterns like mistral-small-3.2-24b are rewritten to mistral-small-24b, and minor version numbers such as 3.5 are collapsed to 3.
4. Returns a stable key – The final string, for example qwen3-30b-a3b or meta-llama-3.1-8b, becomes the canonical identifier used for comparison.

Because the key is deterministic, two otherwise distinct model entries that differ only by quantizer tag or upload prefix will converge to the same family signature.

The Two-Pass Grouping Strategy

The group_models function in src/whichllm/models/grouper.py organizes records through two sequential grouping passes. This design ensures that explicit parent-child relationships are honored before falling back to name-based clustering.

Pass 1: Group by base_model Field

Every ModelInfo object may carry a base_model field that points to the upstream checkpoint it was fine-tuned from. In this phase:

• All models with a non-empty base_model are collected into base_model_groups keyed by the lower-cased base_model value.
• Models that lack a base_model reference are placed into an ungrouped bucket for the next pass.

This captures direct derivation relationships, such as a GGUF re-upload declaring the original Transformers checkpoint as its parent.

Pass 2: Group by Normalised Name

After the first pass, the remaining ungrouped models are normalized:

• Each entry in ungrouped is run through _normalize_name and placed into name_groups.
• The existing base_model_groups are also merged into a normalized view called merged_base, so groups that share the same normalized key are combined.
• If a normalized name appears in both merged_base and name_groups, the two lists are merged into a single group.

This guarantees that a model is never counted twice, even when it could be matched both by explicit base_model lineage and by identifier similarity.

Family Construction and Deduplication Logic

For every resulting group—whether it originated from merged_base or name_groups—the algorithm constructs a final ModelFamily as defined in src/whichllm/models/types.py.

Base Model Selection

The algorithm chooses one representative from the group using the following priority:

1. Referenced upstream checkpoint – Prefer a model that is explicitly referenced by another member’s base_model field, since it is the true parent.
2. Clean, non-derived model – If no upstream exists, prefer a model that has no GGUF or quantization suffixes and does not itself declare a base_model.
3. Fallback – If neither condition is met, pick any member.
4. Tie-breaker – When multiple candidates satisfy the same rule, the model with the highest downloads count wins.

The chosen model’s normalized name becomes the family_id, and the same family_id is written back to every member via model.family_id = family_id.

Variant and Benchmark Aggregation

Once the base is selected:

• All other members are stored as variants inside the ModelFamily.
• Benchmark scores are merged across the entire family, keeping the highest value per metric in the best_benchmark dictionary.

The resulting ModelFamily carries:

• family_id – the stable key derived from the base model.
• display_name – the human-readable name of the selected base.
• base_model – the full ModelInfo of the chosen representative.
• variants – the remaining ModelInfo objects belonging to the same family.
• best_benchmark – the best metric scores found across all variants.

Downstream code can now deduplicate the variants and present a single concise view to the user.

Code Example: Grouping Models Programmatically

You can reproduce the grouping behavior directly by importing group_models and passing a list of ModelInfo objects:

from whichllm.models.grouper import group_models
from whichllm.models.types import ModelInfo

# A few mock ModelInfo objects simulating registry variants

models = [
    ModelInfo(
        id="meta-llama/Meta-Llama-3.1-8B",
        family_id="",
        name="Meta Llama 3.1 8B",
        parameter_count=8_000_000_000,
        downloads=120_000,
        base_model=None,
    ),
    ModelInfo(
        id="meta-llama/Meta-Llama-3.1-8B-gguf",
        family_id="",
        name="Meta Llama 3.1 8B (GGUF)",
        parameter_count=8_000_000_000,
        downloads=30_000,
        gguf_variants=[],
        base_model="meta-llama/Meta-Llama-3.1-8B",
    ),
    ModelInfo(
        id="someorg/Meta-Llama-3.1-8B-Instruct",
        family_id="",
        name="Meta Llama 3.1 8B Instruct",
        parameter_count=8_000_000_000,
        downloads=15_000,
        base_model=None,
    ),
]

families = group_models(models)

for fam in families:
    print(f"Family: {fam.family_id}")
    print(f"  Base:   {fam.base_model.name} ({fam.base_model.id})")
    print(f"  Variants:")
    for v in fam.variants:
        print(f"    - {v.name} ({v.id})")

Running this script demonstrates how the GGUF re-upload and the Instruct fork are folded under one family anchored by the original checkpoint.

How Deduplication Appears in the CLI

When you run whichllm list, the tool fetches model metadata, calls group_models, and renders only the display_name of each ModelFamily. The individual variants remain accessible in the data layer but are hidden from the default view.

$ whichllm list
Meta Llama 3.1 8B          ← family (base + GGUF + Instruct variants)
Qwen 2.5 7B                ← another family
...

Because every member shares the same family_id, the CLI can collapse duplicates without losing information about the representative model or the aggregated benchmarks.

Summary

• Normalization first – _normalize_name in src/whichllm/models/grouper.py strips prefixes, removes stacked suffixes, and collapses version tokens to produce a stable family key.
• Two-pass grouping – Models are first bucketed by explicit base_model references, then remaining entries are clustered by normalized name and merged safely.
• Smart base selection – The algorithm prefers the true upstream checkpoint, falls back to a clean non-quantized model, and breaks ties by download count.
• Unified representation – Each family receives a shared family_id, a list of variants, and a best_benchmark score map, enabling downstream consumers like src/whichllm/cli.py to deduplicate output cleanly.

Frequently Asked Questions

What file contains the core deduplication logic in whichllm?

The core logic lives in src/whichllm/models/grouper.py. This file implements both the _normalize_name helper and the group_models entry point that orchestrates the two-pass clustering and family construction pipeline.

How does whichllm choose the base model for a family?

It applies a priority hierarchy: the preferred candidate is the upstream checkpoint referenced by another member’s base_model field. If none exists, it selects a model without quantization suffixes and without its own base_model, breaking ties by the highest downloads count.

Can models from different organizations be grouped into the same family?

Yes. Because _normalize_name strips organization prefixes and common meta-prefixes before comparison, two models uploaded by different users can map to the same normalized key and therefore be treated as variants of a single family.

Does whichllm merge benchmark scores across variants?

Yes. During family construction, the algorithm scans every member in the group and keeps the highest observed value for each metric. These aggregated scores are stored in the best_benchmark dictionary on the final ModelFamily object.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}