# How WhichLLM's Ranking Engine Scores Models: The Complete Algorithm Guide

> Discover how WhichLLM's ranking engine scores models using seven signals including benchmarks speed and popularity. Understand the complete algorithm to find optimal LLM variants for your hardware.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: deep-dive
- Published: 2026-06-10

---

**WhichLLM's ranking engine calculates a weighted quality score from seven distinct signals—benchmark evidence, inference speed, popularity, evidence source, generation bonus, and derivative penalties—clamping the final result between 0 and 100 to surface optimal LLM variants for your hardware.**

The open-source repository **Andyyyy64/whichllm** implements a sophisticated multi-factor scoring system to rank Large Language Models (LLMs) based on hardware compatibility and performance evidence. At the heart of this system lies the ranking engine in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py), which aggregates benchmark data, speed estimates, and metadata to compute comparable quality scores across diverse model families.

## The Seven Components of Quality Scores

The `_compute_quality_score` function in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) (lines ~596-623) aggregates seven distinct signals into a final 0-100 score. Each component addresses a specific dimension of model suitability.

### Benchmark Quality Core (`quality_core`)

The foundation of every score comes from **benchmark evidence**—direct benchmark results, self-reported evaluations, or estimated scores derived from similar models. This baseline is calculated around lines ~640-660 of [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py) during the call to `_compute_quality_score`. Models with rigorous third-party benchmarks receive higher baseline values than those relying on estimation.

### Inference Speed Score (`speed_score`)

Token throughput directly impacts usability. The engine calls `estimate_tok_per_sec` followed by `estimate_speed_uncertainty` (lines ~441-470) to generate a confidence-adjusted speed range. **Faster models receive a higher contribution** to their final score, with the system favoring variants that maximize tokens-per-second on the detected hardware.

### Popularity Score (`pop_score`)

Community adoption signals model reliability. The engine derives popularity from download and like counts, using the most-downloaded variant in a family as the baseline. The score scales against family-wide maximums captured in the `family_max_downloads` and `family_max_likes` maps (lines ~444-452). Highly downloaded models indicate battle-tested stability.

### Evidence Source Bonus (`source_bonus`)

Not all benchmark data carries equal weight. The source of evidence determines a confidence multiplier calculated inside `_compute_quality_score` (lines ~602-610):

- **Direct benchmark** → Highest bonus
- **Self-reported** → Moderate bonus  
- **Inferred/estimated** → Lower bonus

### Generation Bonus (`gen_bonus`)

Newer architectural generations receive preferential treatment. The generation-specific boost (lines ~596-607) operates inversely to benchmark confidence: if a model lacks benchmark data, the generation bonus amplifies to compensate; if direct benchmarks exist, the bonus dampens to avoid over-weighting novelty over proven performance.

### Derivative Penalty

Forks and modified variants face scrutiny. The `_derivative_name_penalty` function (lines ~608-614) inspects model names for keywords indicating derivative works—such as "abliterated," "uncensored," or "heretic" forks—and applies a penalty to prioritize independent research over specialized modifications.

### Score Clamping

The final sum undergoes normalization. The return statement of `_compute_quality_score` (lines ~616-623) clamps the aggregated score to the interval **[0...100]**, ensuring comparability across all evaluated models regardless of their individual signal strengths.

## The Eight-Step Ranking Workflow

The public `rank_models()` function orchestrates the complete evaluation pipeline. Located at line ~626 in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py), this function executes a deterministic filtering and scoring process:

1. **Pre-processing** – Build family-wide statistics (maximum downloads/likes, dominant parameter counts) to deduplicate families and detect outlier forks.

2. **Filtering** – Exclude models matching exclusion criteria, incompatible task profiles, or falling below the `min_params_b` threshold.

3. **Variant Handling** – Iterate over compatible GGUF quantization variants using `_iter_candidate_variants` to evaluate quantization-specific performance.

4. **Compatibility Check** – Invoke `check_compatibility` to verify the hardware can load and run the specific model variant.

5. **Speed Estimation** – Compute `tok_per_sec` and uncertainty ranges via `estimate_speed_uncertainty` to predict real-world inference performance.

6. **Benchmark Lookup** – Pull evidence from the supplied `benchmark_scores` dictionary or self-reported scores using `lookup_benchmark_evidence`, applying source-specific confidence weighting.

7. **Quality Score** – Execute `_compute_quality_score` with all gathered signals (speed, popularity, benchmark, generation, derivative status).

8. **Best-Variant Selection** – Retain the highest-scoring variant per family, then return the top `N` results sorted by `quality_score`.

## Implementation Example

The following example demonstrates how to invoke the ranking engine with hardware detection and custom benchmark data:

```python
from whichllm.engine.ranker import rank_models
from whichllm.models.fetcher import fetch_models
from whichllm.hardware.windows import detect_hardware

# 1. Load candidate models from the HuggingFace Hub

models = fetch_models(["meta-llama/Meta-Llama-3-8B", "mistralai/Mistral-7B-v0.1"])

# 2. Detect local GPU/CPU hardware description

hardware = detect_hardware()

# 3. Load benchmark scores (model_id → score mapping)

benchmark_scores = {
    "meta-llama/Meta-Llama-3-8B": 78.4,
    "mistralai/Mistral-7B-v0.1": 75.1,
}

# 4. Rank models for the top-5 results

top_models = rank_models(
    models,
    hardware,
    top_n=5,
    benchmark_scores=benchmark_scores,
    task_profile="general",
)

for result in top_models:
    print(
        f"{result.model.id:30}  score={result.quality_score:5.1f}  "
        f"speed={result.estimated_tok_per_sec:6.0f} tok/s  "
        f"benchmark={result.benchmark_status}"
    )

```

Typical output:

```

meta-llama/Meta-Llama-3-8B       score=92.3  speed=  7200 tok/s  benchmark=direct
mistralai/Mistral-7B-v0.1        score=89.7  speed=  6900 tok/s  benchmark=estimated

```

## Key Architecture Files

The scoring system spans multiple modules according to the WhichLLM source code:

- **[`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py)** – Core ranking logic, quality-score computation, and family deduplication.
- **[`src/whichllm/engine/compatibility.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/compatibility.py)** – Hardware feasibility determination for model variants.
- **[`src/whichllm/engine/vram.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/vram.py)** – VRAM calculations and token-per-second estimation helpers.
- **[`src/whichllm/models/fetcher.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/fetcher.py)** – Metadata retrieval (downloads, likes, family IDs, benchmarks).
- **[`docs/scoring.md`](https://github.com/Andyyyy64/whichllm/blob/main/docs/scoring.md)** – Human-readable scoring philosophy documentation.
- **[`tests/test_ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/tests/test_ranker.py)** – Unit tests verifying scoring mathematics and edge cases.

## Summary

- WhichLLM's ranking engine resides in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) and calculates scores via the `rank_models()` function.
- The system aggregates **seven distinct signals**: benchmark quality, speed estimates, popularity, evidence source, generation bonus, derivative penalties, and clamping.
- Benchmark evidence is weighted by source reliability, with direct benchmarks receiving the highest confidence.
- Derivative variants (uncensored, abliterated forks) incur penalties through `_derivative_name_penalty`.
- Final scores clamp to **0-100** to maintain comparability across heterogeneous model families.
- The engine evaluates **GGUF quantization variants** individually, selecting the optimal variant per family based on hardware compatibility.

## Frequently Asked Questions

### How does WhichLLM handle models without direct benchmark scores?

When direct benchmarks are unavailable, the engine falls back to self-reported evaluations or estimates derived from similar models. According to the source code in [`ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/ranker.py), the `lookup_benchmark_evidence` function sources data from multiple evidence tiers, and the generation bonus (`gen_bonus`) amplifies to compensate for lower confidence in estimated scores.

### What triggers the derivative penalty and which models does it affect?

The `_derivative_name_penalty` function (lines ~608-614) scans model names for keywords indicating non-standard forks. Variants containing terms like "abliterated," "uncensored," or "heretic" receive reduced scores because they represent modified versions of base research rather than independent architectural developments.

### How does the ranking engine account for different hardware configurations?

Hardware detection occurs through `detect_hardware()` and `check_compatibility` before scoring begins. The `estimate_tok_per_sec` function calculates hardware-specific inference speeds, while the compatibility layer in [`src/whichllm/engine/compatibility.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/compatibility.py) filters out variants exceeding available VRAM or incompatible compute capabilities.

### Where is the minimum quality threshold configured?

The `rank_models()` function accepts several filtering parameters that act as thresholds: `min_params_b` sets minimum parameter counts, `min_speed` enforces minimum token-per-second requirements, and `evidence_filter` can restrict results to models with direct benchmark evidence only. These parameters are processed during the initial filtering stage before quality scores are calculated.