How Inference Speed Affects whichllm Model Rankings

Inference speed directly modifies whichllm's quality scores through a signed speed term that penalizes slow models below hardware-specific thresholds and rewards fast ones up to a +8 point bonus, while hard floor filters automatically exclude unusably slow candidates from the final rankings.

whichllm is an open-source Python tool that ranks large language models for local inference based on your detected hardware. While model quality remains the primary ranking signal, inference speed acts as a decisive secondary factor that can raise, lower, or completely disqualify a candidate from the top-N list. Understanding how the speed score is computed inside the engine is essential for interpreting the final results.

How the Speed Score Is Calculated in whichllm

The ranking engine in src/whichllm/engine/ranker.py injects inference speed into the scoring formula through the _compute_quality_score logic. This function compares a model's estimated tok_per_sec against a required baseline that depends on the hardware fit type determined by the compatibility layer.

The required baseline thresholds are:

• Full GPU offload: 8.0 tokens per second
• Partial offload: 4.0 tokens per second
• CPU-only: 1.5 tokens per second

The speed score calculation, implemented in src/whichllm/engine/ranker.py around lines 31-37, follows this logic:

required_speed = (
    8.0 if fit_type == "full_gpu"
    else (4.0 if fit_type == "partial_offload" else 1.5)
)
if tok_per_sec > 0:
    if tok_per_sec < required_speed:
        speed_score = -8.0 * (1 - (tok_per_sec / required_speed))
    else:
        speed_score = min(8.0, math.log2(tok_per_sec / required_speed + 1.0) * 3.2)

The resulting speed_score is then summed with the core quality rating, popularity weight, source bonus, and generation bonus to produce the final quality value used for ranking.

Penalties for Models Below the Speed Threshold

When a model's estimated speed falls short of the required baseline for its fit type, the engine assigns a negative speed score. The penalty scales linearly with the distance from the threshold, causing severely slow models to sink in the rankings.

Rewards for Models Above the Speed Threshold

Models that exceed the required baseline receive a positive speed score computed with a logarithmic formula. The bonus is capped at +8.0 points, which prevents extreme throughput from completely overriding quality considerations.

Speed Confidence and Uncertainty Estimates

Speed estimates are not treated as absolute values. The estimate_speed_uncertainty function in src/whichllm/engine/performance.py evaluates the reliability of the tok_per_sec prediction and returns a confidence level of high, medium, or low, along with an estimated range and explanatory notes.

This metadata is attached to the CompatibilityResult object and surfaced to the user through src/whichllm/output/display.py. When speed confidence is low, the CLI emits a caution message such as "Speed caution: Low-confidence speed estimates in top ranks," alerting users that the ordering may be less stable.

Hard Speed Floors That Exclude Unusable Models

After all candidates are scored, the rank_models function applies two speed-floor filters to prune models that are too slow to be practical. According to the source code in src/whichllm/engine/ranker.py, the logic behaves as follows:

• If at least one model achieves an estimated speed of 5.0 tok/s or higher, every model below 1.5 tok/s is dropped from the results.
• If no model meets the 5.0 tok/s bar, the list is left untouched because the detected hardware cannot run any model responsively.

These filters guarantee that the final table contains only practically usable models.

From Hardware Metrics to Rankings: The Performance Pipeline

The tok_per_sec value consumed by the ranker originates in estimate_tok_per_sec inside src/whichllm/engine/performance.py. This estimator synthesizes memory bandwidth, quantization efficiency, backend-specific factors, and Mixture-of-Experts (MoE) read-ratio logic to derive the speed figure.

Key hardware-specific adjustments include:

• Partial-offload penalty: Discrete GPUs lose 45% efficiency when partial offload is required, while Apple Silicon or shared-memory APUs receive a milder 15% penalty.
• MoE read-ratio floor: MoE models benefit from a bandwidth-scaling floor that prevents unfair penalization on low-bandwidth systems.

All of these elements feed the single tok_per_sec number that the ranker uses, making inference speed a first-order signal for both ranking order and eligibility.

Inspecting Speed Data and Setting Custom Filters

You can inspect the speed-related fields on every ranked result to understand how inference speed influenced the ordering. The following example runs the full ranking pipeline and prints the speed metadata attached to each CompatibilityResult:

from whichllm.engine.ranker import rank_models
from whichllm.hardware.types import detect_hardware

hardware = detect_hardware()
results = rank_models(models, hardware, top_n=5)

for r in results:
    print(f"Model: {r.model.id}")
    print(f"  Quality score: {r.quality_score:.1f}")
    print(f"  Estimated speed: {r.estimated_tok_per_sec:.1f} tok/s")
    print(f"  Speed confidence: {r.speed_confidence}")
    print(f"  Speed range: {r.speed_range_tok_per_sec}")

If you want to enforce a stricter speed requirement, pass the min_speed argument to rank_models. Any variant whose tok_per_sec falls below your custom threshold is skipped, as implemented in src/whichllm/engine/ranker.py:

fast_results = rank_models(
    models,
    hardware,
    top_n=10,
    min_speed=3.0,
    require_direct_top=False,
)

Summary

• whichllm adds a signed speed score to the core quality score in src/whichllm/engine/ranker.py, penalizing slow models and rewarding fast ones up to a +8 cap.
• Required speed baselines are 8.0 t/s for full GPU offload, 4.0 t/s for partial offload, and 1.5 t/s for CPU-only.
• Hard floor filters automatically drop models slower than 1.5 t/s whenever at least one candidate reaches 5.0 t/s.
• The estimate_speed_uncertainty function in src/whichllm/engine/performance.py provides confidence metadata that is displayed to warn users about low-reliability estimates.
• Hardware-specific adjustments for partial offloading and MoE architectures feed into the final tok_per_sec value consumed by the ranker.

Frequently Asked Questions

How does whichllm penalize slow models during ranking?

Models with an estimated speed below the hardware fit-type baseline receive a negative speed score calculated as -8.0 * (1 - (tok_per_sec / required_speed)). This linear penalty reduces the total quality score and causes slower models to drop in the rankings.

What are the default speed thresholds in whichllm?

The _compute_quality_score logic uses three required baselines: 8.0 tok/s for full_gpu, 4.0 tok/s for partial_offload, and 1.5 tok/s for CPU-only execution. These thresholds determine whether a model earns a bonus or incurs a penalty.

Can users set a custom minimum inference speed filter?

Yes. The rank_models function accepts an optional min_speed parameter. When provided, the engine excludes any model whose estimated tok_per_sec is below the user-defined value before returning the final rankings.

How does whichllm handle uncertainty in speed estimates?

The estimate_speed_uncertainty function in src/whichllm/engine/performance.py assigns a confidence level of high, medium, or low to each speed estimate. If top-ranked results have low-confidence speed data, the CLI displays a caution message through src/whichllm/output/display.py to warn the user.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}