# How whichllm Groups and Deduplicates LLM Model Variants: A Deep Dive into the Source Code

> Discover how whichllm groups and deduplicates LLM model variants. Explore source code to understand identifier normalization, grouping by base models, and merging benchmarks for a cleaner CLI experience.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: deep-dive
- Published: 2026-06-09

---

**whichllm groups and deduplicates LLM model variants by normalizing model identifiers into stable family keys, grouping first by `base_model` references and then by normalized names, and finally collapsing each group into a single `ModelFamily` object where benchmarks are merged and duplicates are hidden from CLI output.**

The **whichllm** open-source project, as implemented in `Andyyyy64/whichllm`, tackles registry noise by ensuring that quantized files, instruction-tuned checkpoints, and re-uploaded prefixes of the same underlying LLM appear as one logical entry. To **group and deduplicate LLM model variants**, it runs a deterministic multi-pass pipeline implemented in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) with data structures defined in [`src/whichllm/models/types.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/types.py). The result is a concise list of **ModelFamily** instances that downstream code, including the CLI in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py), renders for end users.

## How Model Normalization Creates a Stable Family Key

At the heart of **deduplication** is the `_normalize_name` function inside [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py). This method transforms a raw model ID into a **deterministic key** so that any two identifiers mapping to the same normalized name are treated as one family.

The function performs several cleaning steps:

1. **Strips prefixes** – Organization slugs like `org/…` and common meta-prefixes such as `qwen_`, `meta-llama_`, or `google_` are removed.
2. **Removes stacked suffixes** – Tokens that denote quantization formats, chat/instruction variants, or date stamps are stripped in up to three passes to handle stacked suffixes such as `-gguf-q4` or `-instruct-2401`.
3. **Collapses versions and sizes** – Patterns like `mistral-small-3.2-24b` are rewritten to `mistral-small-24b`, and minor version numbers such as `3.5` are collapsed to `3`.
4. **Returns a stable key** – The final string, for example `qwen3-30b-a3b` or `meta-llama-3.1-8b`, becomes the canonical identifier used for comparison.

Because the key is **deterministic**, two otherwise distinct model entries that differ only by quantizer tag or upload prefix will converge to the same family signature.

## The Two-Pass Grouping Strategy

The `group_models` function in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) organizes records through two sequential **grouping passes**. This design ensures that explicit parent-child relationships are honored before falling back to name-based clustering.

### Pass 1: Group by `base_model` Field

Every **ModelInfo** object may carry a `base_model` field that points to the upstream checkpoint it was fine-tuned from. In this phase:

- All models with a non-empty `base_model` are collected into `base_model_groups` keyed by the lower-cased `base_model` value.
- Models that lack a `base_model` reference are placed into an **ungrouped** bucket for the next pass.

This captures direct **derivation relationships**, such as a GGUF re-upload declaring the original Transformers checkpoint as its parent.

### Pass 2: Group by Normalised Name

After the first pass, the remaining ungrouped models are normalized:

- Each entry in `ungrouped` is run through `_normalize_name` and placed into **name_groups**.
- The existing `base_model_groups` are also merged into a normalized view called **merged_base**, so groups that share the same normalized key are combined.
- If a normalized name appears in both `merged_base` and `name_groups`, the two lists are merged into a single group.

This guarantees that a model is never counted twice, even when it could be matched both by explicit `base_model` **lineage** and by identifier similarity.

## Family Construction and Deduplication Logic

For every resulting group—whether it originated from `merged_base` or `name_groups`—the algorithm constructs a final **ModelFamily** as defined in [`src/whichllm/models/types.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/types.py).

### Base Model Selection

The algorithm chooses one **representative** from the group using the following priority:

1. **Referenced upstream checkpoint** – Prefer a model that is explicitly referenced by another member’s `base_model` field, since it is the true parent.
2. **Clean, non-derived model** – If no upstream exists, prefer a model that has no GGUF or quantization suffixes and does not itself declare a `base_model`.
3. **Fallback** – If neither condition is met, pick any member.
4. **Tie-breaker** – When multiple candidates satisfy the same rule, the model with the highest `downloads` count wins.

The chosen model’s normalized name becomes the **family_id**, and the same `family_id` is written back to every member via `model.family_id = family_id`.

### Variant and Benchmark Aggregation

Once the base is selected:

- All other members are stored as **variants** inside the `ModelFamily`.
- **Benchmark scores** are merged across the entire family, keeping the highest value per metric in the `best_benchmark` dictionary.

The resulting `ModelFamily` carries:

- `family_id` – the stable key derived from the base model.
- `display_name` – the human-readable name of the selected base.
- `base_model` – the full `ModelInfo` of the chosen representative.
- `variants` – the remaining `ModelInfo` objects belonging to the same family.
- `best_benchmark` – the best metric scores found across all variants.

Downstream code can now deduplicate the variants and present a single concise view to the user.

## Code Example: Grouping Models Programmatically

You can reproduce the **grouping behavior** directly by importing `group_models` and passing a list of `ModelInfo` objects:

```python
from whichllm.models.grouper import group_models
from whichllm.models.types import ModelInfo

# A few mock ModelInfo objects simulating registry variants

models = [
    ModelInfo(
        id="meta-llama/Meta-Llama-3.1-8B",
        family_id="",
        name="Meta Llama 3.1 8B",
        parameter_count=8_000_000_000,
        downloads=120_000,
        base_model=None,
    ),
    ModelInfo(
        id="meta-llama/Meta-Llama-3.1-8B-gguf",
        family_id="",
        name="Meta Llama 3.1 8B (GGUF)",
        parameter_count=8_000_000_000,
        downloads=30_000,
        gguf_variants=[],
        base_model="meta-llama/Meta-Llama-3.1-8B",
    ),
    ModelInfo(
        id="someorg/Meta-Llama-3.1-8B-Instruct",
        family_id="",
        name="Meta Llama 3.1 8B Instruct",
        parameter_count=8_000_000_000,
        downloads=15_000,
        base_model=None,
    ),
]

families = group_models(models)

for fam in families:
    print(f"Family: {fam.family_id}")
    print(f"  Base:   {fam.base_model.name} ({fam.base_model.id})")
    print(f"  Variants:")
    for v in fam.variants:
        print(f"    - {v.name} ({v.id})")

```

Running this script demonstrates how the GGUF re-upload and the Instruct fork are folded under one family anchored by the original checkpoint.

## How Deduplication Appears in the CLI

When you run `whichllm list`, the tool fetches model metadata, calls `group_models`, and renders only the **display_name** of each `ModelFamily`. The individual variants remain accessible in the data layer but are hidden from the default view.

```bash
$ whichllm list
Meta Llama 3.1 8B          ← family (base + GGUF + Instruct variants)
Qwen 2.5 7B                ← another family
...

```

Because every member shares the same `family_id`, the CLI can collapse duplicates without losing information about the representative model or the aggregated benchmarks.

## Summary

- **Normalization first** – `_normalize_name` in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) strips prefixes, removes stacked suffixes, and collapses version tokens to produce a stable family key.
- **Two-pass grouping** – Models are first bucketed by explicit `base_model` references, then remaining entries are clustered by normalized name and merged safely.
- **Smart base selection** – The algorithm prefers the true upstream checkpoint, falls back to a clean non-quantized model, and breaks ties by download count.
- **Unified representation** – Each family receives a shared `family_id`, a list of `variants`, and a `best_benchmark` score map, enabling downstream consumers like [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) to deduplicate output cleanly.

## Frequently Asked Questions

### What file contains the core deduplication logic in whichllm?

The core logic lives in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py). This file implements both the `_normalize_name` helper and the `group_models` entry point that orchestrates the two-pass clustering and family construction pipeline.

### How does whichllm choose the base model for a family?

It applies a priority hierarchy: the preferred candidate is the upstream checkpoint referenced by another member’s `base_model` field. If none exists, it selects a model without quantization suffixes and without its own `base_model`, breaking ties by the highest `downloads` count.

### Can models from different organizations be grouped into the same family?

Yes. Because `_normalize_name` strips organization prefixes and common meta-prefixes before comparison, two models uploaded by different users can map to the same normalized key and therefore be treated as variants of a single family.

### Does whichllm merge benchmark scores across variants?

Yes. During family construction, the algorithm scans every member in the group and keeps the highest observed value for each metric. These aggregated scores are stored in the `best_benchmark` dictionary on the final `ModelFamily` object.