What Parameters Does WhichLLM Extract from HuggingFace Models?

WhichLLM extracts 15+ distinct metadata parameters from every HuggingFace model, including technical specs, licensing, popularity metrics, quantization variants, and benchmark scores, storing them in a structured ModelInfo dataclass.

The open-source WhichLLM repository performs deep metadata extraction from the HuggingFace Hub to power its LLM ranking and compatibility engine. Each model fetched through the fetch_models API undergoes parsing in src/whichllm/models/fetcher.py, where specialized extraction functions resolve technical specifications, MoE configurations, and evaluation data into a strongly-typed ModelInfo object defined in src/whichllm/models/types.py.

Core Identification Parameters

WhichLLM establishes model identity through three primary identifiers extracted in the _parse_model function (lines 44–45, 60 of fetcher.py):

• Model ID (id): The canonical repository identifier (e.g., meta-llama/Llama-3.2-8B)
• Family ID (family_id): Initially matches the model ID, later grouped by the grouper engine
• Name (name): A short display name derived from the repository path

These identifiers populate the ModelInfo dataclass (lines 14–30 of types.py), which serves as the central schema for all extracted data.

Technical Architecture Parameters

Parameter Count Resolution

The most complex extraction involves determining model size through _extract_param_count (lines 50–61), which follows a prioritized cascade:

1. Authoritative overrides (hard-coded corrections for known discrepancies)
2. Safetensors metadata (from safetensors index files)
3. GGUF metadata (for quantized variants)
4. Config estimation (calculating from architecture config)
5. Name-hint parsing (extracting size from model ID strings)
6. Hard-coded table (fallback lookup for known models)

Results are stored in ModelInfo.parameter_count.

Active Parameter Counts for MoE Models

For Mixture-of-Experts (MoE) architectures, WhichLLM extracts the active parameter count—the number of parameters actually used per token—via _resolve_moe_active_params (lines 92–104). This logic checks curated tables (_KNOWN_MOE_ACTIVE_PARAMS) or parses the model ID (_extract_active_size_hint_from_id) to distinguish between total and active parameters stored in ModelInfo.parameter_count_active.

Architecture and MoE Detection

The _extract_architecture function (lines 16–38) normalizes the model architecture by inspecting config["architectures"] or falling back to config["model_type"], producing standardized strings like llama, qwen2, or mixtral.

MoE status is determined in _parse_model (lines 92–94) by checking config fields or known MoE active-parameter lookups, populating the boolean is_moe field.

Context Length Extraction

Maximum sequence length is resolved through a priority chain:

• Primary source: config.max_position_embeddings or config.max_sequence_length (lines 45–48)
• Fallback: gguf metadata context_length field (lines 49–50)

The final value is stored in ModelInfo.context_length.

Distribution and Popularity Metrics

WhichLLM captures engagement metrics directly from the HuggingFace API response:

• Downloads (downloads): Total download count from data.get("downloads") (line 68)
• Likes (likes): Community endorsement count from data.get("likes") (line 69)
• Published At (published_at): Timestamp extraction via _extract_published_at (lines 34–42), preferring createdAt with fallback to lastModified
• License (license): SPDX-style license string from cardData.license (line 66)

GGUF Variant Analysis

For quantized deployments, WhichLLM performs deep inspection of GGUF files through a loop over siblings (lines 108–138). Each *.gguf file generates a GGUFVariant entry containing:

• Filename: The original file name
• Quantization type: Extracted via _extract_quant_type (lines 15–31) parsing filename patterns for identifiers like Q4_K_M or FP16
• File size: Reported size or estimates from _estimate_gguf_size using QUANT_BYTES_PER_WEIGHT constants from constants.py

Evaluation and Lineage Data

• Benchmark Scores: The _extract_hf_eval_score function (lines 83–104) filters evaluation results for "general-chat" tasks, computing median scores across benchmarks like MMLU and GSM8K, stored under the hf_eval key in ModelInfo.benchmark_scores
• Base Model: Extracted from cardData.base_model (lines 49–55) to identify finetuned variants and their parent architectures

The Extraction Pipeline Architecture

The complete parameter extraction follows this orchestrated flow in src/whichllm/models/fetcher.py:

1. API Request Construction: fetch_models calls https://huggingface.co/api/models with expand[] parameters for config, safetensors, gguf, cardData, siblings, and evalResults (lines 84–96)

2. Model Parsing: _parse_model validates the ID and dispatches to specialized extractors

3. Parameter Resolution: The cascade in _extract_param_count ensures accurate sizing even for metadata-sparse models

4. MoE Handling: Conditional logic detects expert architectures and resolves active parameter counts

5. Quantization Detection: Filename pattern matching identifies quantization schemes

6. Architecture Normalization: Standardization of architecture strings for consistent categorization

7. Benchmark Aggregation: Median calculation across available evaluation results

8. Dataclass Assembly: All fields populate the ModelInfo instance for JSON serialization and caching

Code Examples

The following example demonstrates fetching models and accessing extracted parameters:

import asyncio
from whichllm.models.fetcher import fetch_models

async def demo():
    models = await fetch_models(limit=10, include_vision=False)
    for m in models:
        print(f"🧩 {m.id}")
        print(f"  Params: {m.parameter_count:,} ({m.parameter_count_active or 'dense'})")
        print(f"  Arch:  {m.architecture}")
        print(f"  MoE?  {'yes' if m.is_moe else 'no'}")
        print(f"  Context length: {m.context_length or 'unknown'}")
        print(f"  License: {m.license or 'unspecified'}")
        print(f"  Downloads: {m.downloads:,}")
        print(f"  HF Eval score: {m.benchmark_scores.get('hf_eval', 'n/a')}")
        for gguf in m.gguf_variants:
            print(f"    GGUF: {gguf.filename} – {gguf.quant_type} – {gguf.file_size_bytes:,} B")
        print()

asyncio.run(demo())

To inspect publication timelines for specific frontier models:

import asyncio
from whichllm.models.fetcher import fetch_model_published_at

async def get_dates():
    dates = await fetch_model_published_at(
        ["meta-llama/Llama-4-Scout-17B-16E-Instruct",
         "deepseek-ai/DeepSeek-V4-Pro"]
    )
    for model_id, ts in dates.items():
        print(f"{model_id} → published/modified at {ts}")

asyncio.run(get_dates())

Summary

• WhichLLM extracts 15+ distinct parameters from each HuggingFace model, spanning technical specs, licensing, and performance metrics
• Parameter counts are resolved through a robust six-tier cascade (authoritative → safetensors → GGUF → config → name-hint → table) in src/whichllm/models/fetcher.py
• MoE models receive special handling to distinguish between total and active parameter counts
• GGUF variants are fully cataloged with quantization types and file sizes for deployment planning
• All extracted data is stored in the ModelInfo dataclass defined in src/whichllm/models/types.py

Frequently Asked Questions

How does WhichLLM calculate parameter counts for models missing explicit metadata?

WhichLLM employs a prioritized resolution cascade in _extract_param_count (lines 50–61). If safetensors metadata is unavailable, it falls back to GGUF headers, then estimates from the config file, parses the model ID for size hints, or consults a hard-coded lookup table for known models. This ensures every model receives a sensible parameter count estimate even when upstream metadata is incomplete.

What distinguishes active parameter count from total parameter count in MoE models?

For Mixture-of-Experts architectures, the total parameter count represents all weights in the model, while the active parameter count reflects only the parameters used per token (typically the activated expert pathways). WhichLLM extracts this via _resolve_moe_active_params using curated tables or ID parsing, storing the result in ModelInfo.parameter_count_active to give users accurate VRAM and latency estimates.

How are quantization types detected for GGUF variants?

Quantization detection occurs in _extract_quant_type (lines 15–31) through filename pattern matching. The function parses GGUF filenames for standard identifiers like Q4_K_M, FP16, or Q5_0, returning a normalized quantization string. When file size metadata is missing, WhichLLM estimates it using byte-per-weight constants from constants.py to ensure complete variant listings.

Where does WhichLLM store the extracted HuggingFace model parameters?

All extracted parameters populate the ModelInfo dataclass defined in src/whichllm/models/types.py (lines 14–30). This structured type includes fields for core identifiers (id, name), technical specs (parameter_count, architecture, context_length), popularity metrics (downloads, likes), and nested structures for GGUFVariant and benchmark scores. The fetcher serializes these instances to JSON for caching and downstream ranking algorithms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}