# How Does WhichLLM Group Similar LLM Models? A Two-Pass Clustering Approach

> Discover how WhichLLM groups similar LLM models with its innovative two-pass clustering approach. It leverages Hugging Face metadata and name normalization for accurate model family identification.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: internals
- Published: 2026-06-10

---

**WhichLLM groups similar LLM models using a two-pass clustering algorithm that first checks Hugging Face `base_model` metadata, then falls back to aggressive name normalization to collapse quantized variants and forks into coherent families.**

WhichLLM is an open-source tool that helps developers navigate the fragmented landscape of Hugging Face model repositories by organizing thousands of fine-tuned and quantized variants into logical families. Understanding how the tool clusters these repositories is essential for interpreting its ranking results and recommendations. The grouping logic lives in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) and implements a deterministic pipeline that prioritizes authoritative metadata before applying heuristic name matching.

## The Two-Pass Grouping Strategy

The `group_models()` function in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) processes model metadata through two distinct passes to maximize accuracy while remaining resilient to missing metadata.

### Pass 1: Base-Model Metadata Grouping

The grouper first attempts to leverage explicit metadata from Hugging Face model cards. If a model's card data contains a `base_model` field, the repository is immediately grouped by that value using a case-insensitive comparison【source】. This approach captures the ground-truth lineage for models that properly declare their architectural ancestry, such as fine-tuned variants that specify `meta-llama/Llama-2-7b-hf` as their foundation.

This metadata-driven approach handles cases where repository names diverge significantly from their base architecture (e.g., community fine-tunes with creative naming conventions). The implementation at lines 64-71 in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) extracts this field and uses it as the primary grouping key before falling back to name-based heuristics.

### Pass 2: Name Normalization Grouping

Models lacking `base_model` metadata—or those that couldn't be grouped in the first pass—are processed through a sophisticated normalization pipeline that strips repository names to their architectural core. This logic spans lines 13-55 in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) and executes several transformations:

**Organization and common prefix removal** – The normalizer strips organization prefixes like `bartowski/` and common series prefixes such as `qwen_` or `meta-llama_` to prevent the same architecture from being split across different namespaces.

**Quantization and variant suffix stripping** – A comprehensive block of regular expressions (lines 20-34) removes suffixes indicating quantization formats (`-gguf`, `-gptq`), instruction tuning (`-chat`, `-instruct`), dates (`-2507`), bit-widths (`-4bit`, `-8bit`), and other technical variants. The algorithm performs up to three passes (lines 35-41) to catch stacked suffixes like `model-v2-4bit-gguf`.

**Version and size token handling** – The normalizer collapses version numbers appearing before size tokens (e.g., `mistral-small-3.2-24b` retains only the size dimension), and strips minor version indicators from series names while preserving size suffixes (e.g., `qwen3.5-27b` becomes `qwen3-27b`)【source】.

## Merging Groups and Deriving Family IDs

After the two passes complete, the grouper reconciles potentially overlapping groups. Base-model groups whose normalized keys match name-based groups are merged into single families (lines 81-91). This prevents duplication when a model declares a `base_model` but its normalized name also matches other repositories.

For each final group, WhichLLM derives a **family id** by selecting the base model with the highest download count among prioritized candidates (lines 102-128). The normalized name of this chosen base model becomes the canonical identifier for the entire family. This ensures that popular, well-known variants (like the original Meta Llama checkpoints) define the family identity rather than obscure forks.

## Aggregation and Ranking

Once families are established, the grouper assigns the derived `family_id` to all member models and aggregates benchmark scores across the family (lines 132-138). This aggregation allows WhichLLM to rank entire model families rather than individual quantized variants, presenting users with the best-performing configuration while maintaining awareness of available alternatives.

As documented in [`docs/how-it-works.md`](https://github.com/Andyyyy64/whichllm/blob/main/docs/how-it-works.md), step 5 of the request flow explicitly handles "Group related model repos into families" to prepare for downstream ranking operations.

## Practical Implementation

To observe the grouping behavior programmatically, you can invoke the `group_models` function directly:

```python
from whichllm.models.grouper import group_models
from whichllm.models.types import ModelInfo

# Pretend we already fetched a list of ModelInfo objects:

models = [...]   # List[ModelInfo]

families = group_models(models)

for fam in families:
    print(f"Family: {fam.display_name} ({fam.family_id})")
    print("  Base model:", fam.base_model.id)
    print("  Variants:")
    for v in fam.variants:
        print(f"    - {v.id}")

```

This code processes the `ModelInfo` dataclasses defined in [`src/whichllm/models/types.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/types.py) and returns `ModelFamily` objects containing the aggregated metadata. The fetcher module ([`src/whichllm/models/fetcher.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/fetcher.py)) retrieves the necessary metadata including `base_model` fields, while the ranker ([`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py)) consumes these families for final recommendation scoring.

## Summary

- **Metadata-first approach**: WhichLLM prioritizes explicit `base_model` fields from Hugging Face cards when available.
- **Aggressive normalization**: When metadata is absent, the tool strips organizations, quantizations (GGUF, GPTQ), instruction tags, and version numbers to reveal architectural roots.
- **Multi-pass cleaning**: The normalizer runs three iterations to handle stacked suffixes and complex naming conventions.
- **Smart deduplication**: Groups from both methods are merged and deduplicated before finalizing family assignments.
- **Download-based canonicalization**: The most popular variant in a group defines the family identity and display name.

## Frequently Asked Questions

### What happens if a model has both a base_model field and a normalized name that matches other repositories?

If a model declares a `base_model` in its metadata but its normalized repository name also matches other models, WhichLLM merges these groups into a single family. According to lines 81-91 in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py), the system reconciles base-model groups with name-based groups by comparing their normalized keys, preventing duplicate families for the same underlying architecture.

### How does WhichLLM handle different quantization formats like GGUF and GPTQ?

The normalization logic treats quantization suffixes as noise to be stripped. Lines 20-34 in [`src/whichllm/models/grouper.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/grouper.py) explicitly remove suffixes like `-gguf`, `-gptq`, `-4bit`, and `-8bit` through multiple regex passes. This ensures that `llama-2-7b-gguf` and `llama-2-7b-gptq` collapse into the same family despite differing file formats.

### Why does the family ID come from the model with the highest download count?

WhichLLM selects the base model with the highest download count (lines 102-128) to serve as the canonical representative because popularity generally correlates with community trust and validation. This heuristic ensures that well-known official releases define the family identity rather than obscure forks or experimental variants that might happen to have similar names.

### Can the grouping algorithm handle nested or stacked versioning in model names?

Yes, the normalizer performs up to three passes (lines 35-41) to handle stacked suffixes, and includes specific logic for version numbers preceding size tokens (lines 42-47) and series versions with size suffixes (lines 48-55). For example, `qwen3.5-27b` normalizes to `qwen3-27b`, preserving the size dimension while standardizing the series version.