# How Inference Speed Affects whichllm Model Rankings

> Discover how inference speed impacts whichllm model rankings. Learn how speed scores and filters affect LLM performance evaluations in the whichllm repository.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: performance
- Published: 2026-06-09

---

**Inference speed directly modifies whichllm's quality scores through a signed speed term that penalizes slow models below hardware-specific thresholds and rewards fast ones up to a +8 point bonus, while hard floor filters automatically exclude unusably slow candidates from the final rankings.**

`whichllm` is an open-source Python tool that ranks large language models for local inference based on your detected hardware. While model quality remains the primary ranking signal, inference speed acts as a decisive secondary factor that can raise, lower, or completely disqualify a candidate from the top-N list. Understanding how the speed score is computed inside the engine is essential for interpreting the final results.

## How the Speed Score Is Calculated in `whichllm`

The ranking engine in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) injects inference speed into the scoring formula through the `_compute_quality_score` logic. This function compares a model's estimated `tok_per_sec` against a **required baseline** that depends on the hardware fit type determined by the compatibility layer.

The required baseline thresholds are:

- **Full GPU offload**: 8.0 tokens per second
- **Partial offload**: 4.0 tokens per second
- **CPU-only**: 1.5 tokens per second

The speed score calculation, implemented in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) around lines 31-37, follows this logic:

```python
required_speed = (
    8.0 if fit_type == "full_gpu"
    else (4.0 if fit_type == "partial_offload" else 1.5)
)
if tok_per_sec > 0:
    if tok_per_sec < required_speed:
        speed_score = -8.0 * (1 - (tok_per_sec / required_speed))
    else:
        speed_score = min(8.0, math.log2(tok_per_sec / required_speed + 1.0) * 3.2)

```

The resulting `speed_score` is then summed with the core quality rating, popularity weight, source bonus, and generation bonus to produce the final quality value used for ranking.

### Penalties for Models Below the Speed Threshold

When a model's estimated speed falls short of the required baseline for its fit type, the engine assigns a **negative speed score**. The penalty scales linearly with the distance from the threshold, causing severely slow models to sink in the rankings.

### Rewards for Models Above the Speed Threshold

Models that exceed the required baseline receive a **positive speed score** computed with a logarithmic formula. The bonus is capped at **+8.0 points**, which prevents extreme throughput from completely overriding quality considerations.

## Speed Confidence and Uncertainty Estimates

Speed estimates are not treated as absolute values. The `estimate_speed_uncertainty` function in [`src/whichllm/engine/performance.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/performance.py) evaluates the reliability of the `tok_per_sec` prediction and returns a confidence level of `high`, `medium`, or `low`, along with an estimated range and explanatory notes.

This metadata is attached to the `CompatibilityResult` object and surfaced to the user through [`src/whichllm/output/display.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/output/display.py). When speed confidence is low, the CLI emits a caution message such as "Speed caution: Low-confidence speed estimates in top ranks," alerting users that the ordering may be less stable.

## Hard Speed Floors That Exclude Unusable Models

After all candidates are scored, the `rank_models` function applies two **speed-floor filters** to prune models that are too slow to be practical. According to the source code in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py), the logic behaves as follows:

- If at least one model achieves an estimated speed of **5.0 tok/s or higher**, every model below **1.5 tok/s** is dropped from the results.
- If no model meets the 5.0 tok/s bar, the list is left untouched because the detected hardware cannot run any model responsively.

These filters guarantee that the final table contains only practically usable models.

## From Hardware Metrics to Rankings: The Performance Pipeline

The `tok_per_sec` value consumed by the ranker originates in `estimate_tok_per_sec` inside [`src/whichllm/engine/performance.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/performance.py). This estimator synthesizes memory bandwidth, quantization efficiency, backend-specific factors, and Mixture-of-Experts (MoE) read-ratio logic to derive the speed figure.

Key hardware-specific adjustments include:

- **Partial-offload penalty**: Discrete GPUs lose **45%** efficiency when partial offload is required, while Apple Silicon or shared-memory APUs receive a milder **15%** penalty.
- **MoE read-ratio floor**: MoE models benefit from a bandwidth-scaling floor that prevents unfair penalization on low-bandwidth systems.

All of these elements feed the single `tok_per_sec` number that the ranker uses, making inference speed a first-order signal for both ranking order and eligibility.

## Inspecting Speed Data and Setting Custom Filters

You can inspect the speed-related fields on every ranked result to understand how inference speed influenced the ordering. The following example runs the full ranking pipeline and prints the speed metadata attached to each `CompatibilityResult`:

```python
from whichllm.engine.ranker import rank_models
from whichllm.hardware.types import detect_hardware

hardware = detect_hardware()
results = rank_models(models, hardware, top_n=5)

for r in results:
    print(f"Model: {r.model.id}")
    print(f"  Quality score: {r.quality_score:.1f}")
    print(f"  Estimated speed: {r.estimated_tok_per_sec:.1f} tok/s")
    print(f"  Speed confidence: {r.speed_confidence}")
    print(f"  Speed range: {r.speed_range_tok_per_sec}")

```

If you want to enforce a stricter speed requirement, pass the `min_speed` argument to `rank_models`. Any variant whose `tok_per_sec` falls below your custom threshold is skipped, as implemented in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py):

```python
fast_results = rank_models(
    models,
    hardware,
    top_n=10,
    min_speed=3.0,
    require_direct_top=False,
)

```

## Summary

- `whichllm` adds a signed **speed score** to the core quality score in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py), penalizing slow models and rewarding fast ones up to a +8 cap.
- Required speed baselines are **8.0 t/s** for full GPU offload, **4.0 t/s** for partial offload, and **1.5 t/s** for CPU-only.
- **Hard floor filters** automatically drop models slower than 1.5 t/s whenever at least one candidate reaches 5.0 t/s.
- The `estimate_speed_uncertainty` function in [`src/whichllm/engine/performance.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/performance.py) provides confidence metadata that is displayed to warn users about low-reliability estimates.
- Hardware-specific adjustments for partial offloading and MoE architectures feed into the final `tok_per_sec` value consumed by the ranker.

## Frequently Asked Questions

### How does whichllm penalize slow models during ranking?

Models with an estimated speed below the hardware fit-type baseline receive a negative speed score calculated as `-8.0 * (1 - (tok_per_sec / required_speed))`. This linear penalty reduces the total quality score and causes slower models to drop in the rankings.

### What are the default speed thresholds in whichllm?

The `_compute_quality_score` logic uses three required baselines: **8.0 tok/s** for `full_gpu`, **4.0 tok/s** for `partial_offload`, and **1.5 tok/s** for CPU-only execution. These thresholds determine whether a model earns a bonus or incurs a penalty.

### Can users set a custom minimum inference speed filter?

Yes. The `rank_models` function accepts an optional `min_speed` parameter. When provided, the engine excludes any model whose estimated `tok_per_sec` is below the user-defined value before returning the final rankings.

### How does whichllm handle uncertainty in speed estimates?

The `estimate_speed_uncertainty` function in [`src/whichllm/engine/performance.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/performance.py) assigns a confidence level of `high`, `medium`, or `low` to each speed estimate. If top-ranked results have low-confidence speed data, the CLI displays a caution message through [`src/whichllm/output/display.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/output/display.py) to warn the user.