How WhichLLM's Ranking Engine Scores Models: The Complete Algorithm Guide

WhichLLM's ranking engine calculates a weighted quality score from seven distinct signals—benchmark evidence, inference speed, popularity, evidence source, generation bonus, and derivative penalties—clamping the final result between 0 and 100 to surface optimal LLM variants for your hardware.

The open-source repository Andyyyy64/whichllm implements a sophisticated multi-factor scoring system to rank Large Language Models (LLMs) based on hardware compatibility and performance evidence. At the heart of this system lies the ranking engine in src/whichllm/engine/ranker.py, which aggregates benchmark data, speed estimates, and metadata to compute comparable quality scores across diverse model families.

The Seven Components of Quality Scores

The _compute_quality_score function in src/whichllm/engine/ranker.py (lines ~596-623) aggregates seven distinct signals into a final 0-100 score. Each component addresses a specific dimension of model suitability.

Benchmark Quality Core (quality_core)

The foundation of every score comes from benchmark evidence—direct benchmark results, self-reported evaluations, or estimated scores derived from similar models. This baseline is calculated around lines ~640-660 of ranker.py during the call to _compute_quality_score. Models with rigorous third-party benchmarks receive higher baseline values than those relying on estimation.

Inference Speed Score (speed_score)

Token throughput directly impacts usability. The engine calls estimate_tok_per_sec followed by estimate_speed_uncertainty (lines ~441-470) to generate a confidence-adjusted speed range. Faster models receive a higher contribution to their final score, with the system favoring variants that maximize tokens-per-second on the detected hardware.

Popularity Score (pop_score)

Community adoption signals model reliability. The engine derives popularity from download and like counts, using the most-downloaded variant in a family as the baseline. The score scales against family-wide maximums captured in the family_max_downloads and family_max_likes maps (lines ~444-452). Highly downloaded models indicate battle-tested stability.

Evidence Source Bonus (source_bonus)

Not all benchmark data carries equal weight. The source of evidence determines a confidence multiplier calculated inside _compute_quality_score (lines ~602-610):

• Direct benchmark → Highest bonus
• Self-reported → Moderate bonus
• Inferred/estimated → Lower bonus

Generation Bonus (gen_bonus)

Newer architectural generations receive preferential treatment. The generation-specific boost (lines ~596-607) operates inversely to benchmark confidence: if a model lacks benchmark data, the generation bonus amplifies to compensate; if direct benchmarks exist, the bonus dampens to avoid over-weighting novelty over proven performance.

Derivative Penalty

Forks and modified variants face scrutiny. The _derivative_name_penalty function (lines ~608-614) inspects model names for keywords indicating derivative works—such as "abliterated," "uncensored," or "heretic" forks—and applies a penalty to prioritize independent research over specialized modifications.

Score Clamping

The final sum undergoes normalization. The return statement of _compute_quality_score (lines ~616-623) clamps the aggregated score to the interval [0...100], ensuring comparability across all evaluated models regardless of their individual signal strengths.

The Eight-Step Ranking Workflow

The public rank_models() function orchestrates the complete evaluation pipeline. Located at line ~626 in src/whichllm/engine/ranker.py, this function executes a deterministic filtering and scoring process:

1. Pre-processing – Build family-wide statistics (maximum downloads/likes, dominant parameter counts) to deduplicate families and detect outlier forks.

2. Filtering – Exclude models matching exclusion criteria, incompatible task profiles, or falling below the min_params_b threshold.

3. Variant Handling – Iterate over compatible GGUF quantization variants using _iter_candidate_variants to evaluate quantization-specific performance.

4. Compatibility Check – Invoke check_compatibility to verify the hardware can load and run the specific model variant.

5. Speed Estimation – Compute tok_per_sec and uncertainty ranges via estimate_speed_uncertainty to predict real-world inference performance.

6. Benchmark Lookup – Pull evidence from the supplied benchmark_scores dictionary or self-reported scores using lookup_benchmark_evidence, applying source-specific confidence weighting.

7. Quality Score – Execute _compute_quality_score with all gathered signals (speed, popularity, benchmark, generation, derivative status).

8. Best-Variant Selection – Retain the highest-scoring variant per family, then return the top N results sorted by quality_score.

Implementation Example

The following example demonstrates how to invoke the ranking engine with hardware detection and custom benchmark data:

from whichllm.engine.ranker import rank_models
from whichllm.models.fetcher import fetch_models
from whichllm.hardware.windows import detect_hardware

# 1. Load candidate models from the HuggingFace Hub

models = fetch_models(["meta-llama/Meta-Llama-3-8B", "mistralai/Mistral-7B-v0.1"])

# 2. Detect local GPU/CPU hardware description

hardware = detect_hardware()

# 3. Load benchmark scores (model_id → score mapping)

benchmark_scores = {
    "meta-llama/Meta-Llama-3-8B": 78.4,
    "mistralai/Mistral-7B-v0.1": 75.1,
}

# 4. Rank models for the top-5 results

top_models = rank_models(
    models,
    hardware,
    top_n=5,
    benchmark_scores=benchmark_scores,
    task_profile="general",
)

for result in top_models:
    print(
        f"{result.model.id:30}  score={result.quality_score:5.1f}  "
        f"speed={result.estimated_tok_per_sec:6.0f} tok/s  "
        f"benchmark={result.benchmark_status}"
    )

Typical output:

meta-llama/Meta-Llama-3-8B       score=92.3  speed=  7200 tok/s  benchmark=direct
mistralai/Mistral-7B-v0.1        score=89.7  speed=  6900 tok/s  benchmark=estimated

Key Architecture Files

The scoring system spans multiple modules according to the WhichLLM source code:

• src/whichllm/engine/ranker.py – Core ranking logic, quality-score computation, and family deduplication.
• src/whichllm/engine/compatibility.py – Hardware feasibility determination for model variants.
• src/whichllm/engine/vram.py – VRAM calculations and token-per-second estimation helpers.
• src/whichllm/models/fetcher.py – Metadata retrieval (downloads, likes, family IDs, benchmarks).
• docs/scoring.md – Human-readable scoring philosophy documentation.
• tests/test_ranker.py – Unit tests verifying scoring mathematics and edge cases.

Summary

• WhichLLM's ranking engine resides in src/whichllm/engine/ranker.py and calculates scores via the rank_models() function.
• The system aggregates seven distinct signals: benchmark quality, speed estimates, popularity, evidence source, generation bonus, derivative penalties, and clamping.
• Benchmark evidence is weighted by source reliability, with direct benchmarks receiving the highest confidence.
• Derivative variants (uncensored, abliterated forks) incur penalties through _derivative_name_penalty.
• Final scores clamp to 0-100 to maintain comparability across heterogeneous model families.
• The engine evaluates GGUF quantization variants individually, selecting the optimal variant per family based on hardware compatibility.

Frequently Asked Questions

How does WhichLLM handle models without direct benchmark scores?

When direct benchmarks are unavailable, the engine falls back to self-reported evaluations or estimates derived from similar models. According to the source code in ranker.py, the lookup_benchmark_evidence function sources data from multiple evidence tiers, and the generation bonus (gen_bonus) amplifies to compensate for lower confidence in estimated scores.

What triggers the derivative penalty and which models does it affect?

The _derivative_name_penalty function (lines ~608-614) scans model names for keywords indicating non-standard forks. Variants containing terms like "abliterated," "uncensored," or "heretic" receive reduced scores because they represent modified versions of base research rather than independent architectural developments.

How does the ranking engine account for different hardware configurations?

Hardware detection occurs through detect_hardware() and check_compatibility before scoring begins. The estimate_tok_per_sec function calculates hardware-specific inference speeds, while the compatibility layer in src/whichllm/engine/compatibility.py filters out variants exceeding available VRAM or incompatible compute capabilities.

Where is the minimum quality threshold configured?

The rank_models() function accepts several filtering parameters that act as thresholds: min_params_b sets minimum parameter counts, min_speed enforces minimum token-per-second requirements, and evidence_filter can restrict results to models with direct benchmark evidence only. These parameters are processed during the initial filtering stage before quality scores are calculated.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}