How Does WhichLLM Group Similar LLM Models? A Two-Pass Clustering Approach

WhichLLM groups similar LLM models using a two-pass clustering algorithm that first checks Hugging Face base_model metadata, then falls back to aggressive name normalization to collapse quantized variants and forks into coherent families.

WhichLLM is an open-source tool that helps developers navigate the fragmented landscape of Hugging Face model repositories by organizing thousands of fine-tuned and quantized variants into logical families. Understanding how the tool clusters these repositories is essential for interpreting its ranking results and recommendations. The grouping logic lives in src/whichllm/models/grouper.py and implements a deterministic pipeline that prioritizes authoritative metadata before applying heuristic name matching.

The Two-Pass Grouping Strategy

The group_models() function in src/whichllm/models/grouper.py processes model metadata through two distinct passes to maximize accuracy while remaining resilient to missing metadata.

Pass 1: Base-Model Metadata Grouping

The grouper first attempts to leverage explicit metadata from Hugging Face model cards. If a model's card data contains a base_model field, the repository is immediately grouped by that value using a case-insensitive comparison【source】. This approach captures the ground-truth lineage for models that properly declare their architectural ancestry, such as fine-tuned variants that specify meta-llama/Llama-2-7b-hf as their foundation.

This metadata-driven approach handles cases where repository names diverge significantly from their base architecture (e.g., community fine-tunes with creative naming conventions). The implementation at lines 64-71 in src/whichllm/models/grouper.py extracts this field and uses it as the primary grouping key before falling back to name-based heuristics.

Pass 2: Name Normalization Grouping

Models lacking base_model metadata—or those that couldn't be grouped in the first pass—are processed through a sophisticated normalization pipeline that strips repository names to their architectural core. This logic spans lines 13-55 in src/whichllm/models/grouper.py and executes several transformations:

Organization and common prefix removal – The normalizer strips organization prefixes like bartowski/ and common series prefixes such as qwen_ or meta-llama_ to prevent the same architecture from being split across different namespaces.

Quantization and variant suffix stripping – A comprehensive block of regular expressions (lines 20-34) removes suffixes indicating quantization formats (-gguf, -gptq), instruction tuning (-chat, -instruct), dates (-2507), bit-widths (-4bit, -8bit), and other technical variants. The algorithm performs up to three passes (lines 35-41) to catch stacked suffixes like model-v2-4bit-gguf.

Version and size token handling – The normalizer collapses version numbers appearing before size tokens (e.g., mistral-small-3.2-24b retains only the size dimension), and strips minor version indicators from series names while preserving size suffixes (e.g., qwen3.5-27b becomes qwen3-27b)【source】.

Merging Groups and Deriving Family IDs

After the two passes complete, the grouper reconciles potentially overlapping groups. Base-model groups whose normalized keys match name-based groups are merged into single families (lines 81-91). This prevents duplication when a model declares a base_model but its normalized name also matches other repositories.

For each final group, WhichLLM derives a family id by selecting the base model with the highest download count among prioritized candidates (lines 102-128). The normalized name of this chosen base model becomes the canonical identifier for the entire family. This ensures that popular, well-known variants (like the original Meta Llama checkpoints) define the family identity rather than obscure forks.

Aggregation and Ranking

Once families are established, the grouper assigns the derived family_id to all member models and aggregates benchmark scores across the family (lines 132-138). This aggregation allows WhichLLM to rank entire model families rather than individual quantized variants, presenting users with the best-performing configuration while maintaining awareness of available alternatives.

As documented in docs/how-it-works.md, step 5 of the request flow explicitly handles "Group related model repos into families" to prepare for downstream ranking operations.

Practical Implementation

To observe the grouping behavior programmatically, you can invoke the group_models function directly:

from whichllm.models.grouper import group_models
from whichllm.models.types import ModelInfo

# Pretend we already fetched a list of ModelInfo objects:

models = [...]   # List[ModelInfo]

families = group_models(models)

for fam in families:
    print(f"Family: {fam.display_name} ({fam.family_id})")
    print("  Base model:", fam.base_model.id)
    print("  Variants:")
    for v in fam.variants:
        print(f"    - {v.id}")

This code processes the ModelInfo dataclasses defined in src/whichllm/models/types.py and returns ModelFamily objects containing the aggregated metadata. The fetcher module (src/whichllm/models/fetcher.py) retrieves the necessary metadata including base_model fields, while the ranker (src/whichllm/engine/ranker.py) consumes these families for final recommendation scoring.

Summary

• Metadata-first approach: WhichLLM prioritizes explicit base_model fields from Hugging Face cards when available.
• Aggressive normalization: When metadata is absent, the tool strips organizations, quantizations (GGUF, GPTQ), instruction tags, and version numbers to reveal architectural roots.
• Multi-pass cleaning: The normalizer runs three iterations to handle stacked suffixes and complex naming conventions.
• Smart deduplication: Groups from both methods are merged and deduplicated before finalizing family assignments.
• Download-based canonicalization: The most popular variant in a group defines the family identity and display name.

Frequently Asked Questions

What happens if a model has both a base_model field and a normalized name that matches other repositories?

If a model declares a base_model in its metadata but its normalized repository name also matches other models, WhichLLM merges these groups into a single family. According to lines 81-91 in src/whichllm/models/grouper.py, the system reconciles base-model groups with name-based groups by comparing their normalized keys, preventing duplicate families for the same underlying architecture.

How does WhichLLM handle different quantization formats like GGUF and GPTQ?

The normalization logic treats quantization suffixes as noise to be stripped. Lines 20-34 in src/whichllm/models/grouper.py explicitly remove suffixes like -gguf, -gptq, -4bit, and -8bit through multiple regex passes. This ensures that llama-2-7b-gguf and llama-2-7b-gptq collapse into the same family despite differing file formats.

Why does the family ID come from the model with the highest download count?

WhichLLM selects the base model with the highest download count (lines 102-128) to serve as the canonical representative because popularity generally correlates with community trust and validation. This heuristic ensures that well-known official releases define the family identity rather than obscure forks or experimental variants that might happen to have similar names.

Can the grouping algorithm handle nested or stacked versioning in model names?

Yes, the normalizer performs up to three passes (lines 35-41) to handle stacked suffixes, and includes specific logic for version numbers preceding size tokens (lines 42-47) and series versions with size suffixes (lines 48-55). For example, qwen3.5-27b normalizes to qwen3-27b, preserving the size dimension while standardizing the series version.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}