How to Set a Minimum Speed Requirement for LLM Selection in whichllm

Use the --min-speed flag in the whichllm CLI, or pass the min_speed argument to rank_models() in Python, to exclude model variants whose estimated throughput falls below your required tokens-per-second floor.

whichllm is an open-source LLM recommendation engine that ranks candidate models using hardware-aware benchmarks and performance estimates. If your workload requires a strict minimum speed requirement for LLM selection, the tool lets you define a throughput floor in tokens per second so only fast enough models appear in the results.

How the --min-speed Filter Works

The speed filter is implemented across three layers: CLI parsing, core ranking, and performance estimation.

CLI Entry Point in src/whichllm/cli.py

When you invoke whichllm from the terminal, the optional --min-speed flag is defined by Typer in src/whichllm/cli.py. The parsed value is forwarded directly into the ranking pipeline.

Filtering Logic in src/whichllm/engine/ranker.py

The core ranking routine rank_models inside src/whichllm/engine/ranker.py evaluates every candidate model. For each variant, it calls estimate_tok_per_sec to predict throughput. If you supply a min_speed value, the routine short-circuits any variant that fails to meet the threshold:

tok_per_sec = estimate_tok_per_sec(...)
if min_speed is not None and tok_per_sec < min_speed:
    continue

Only models with an estimated rate greater than or equal to your floor continue through the pipeline and become eligible for final output.

Speed Estimation in src/whichllm/engine/performance.py

The actual throughput prediction is produced by estimate_tok_per_sec, located in src/whichllm/engine/performance.py. This estimator consumes hardware profiles from src/whichllm/hardware/detector.py—which detects your GPU or CPU configuration—to produce a context-aware tokens-per-second value before the filter is applied.

Usage Examples

You can enforce a minimum speed requirement for LLM selection from both the command line and the Python API.

CLI Examples

Show only the top models that can generate at least 30 tok/s:

whichllm --min-speed 30

Combine the speed floor with other filters, such as a parameter minimum and strict evidence requirements:

whichllm --min-speed 45 --min-params 7 --evidence strict

Python API Example

For embedded or scripted workflows, import rank_models along with the hardware and model fetchers, then pass min_speed as a float:

from whichllm.engine.ranker import rank_models
from whichllm.hardware.detector import detect_hardware
from whichllm.models.fetcher import fetch_models

hardware = detect_hardware()
models = fetch_models()

# Enforce a minimum speed requirement for LLM selection of 25 tok/s

results = rank_models(
    models,
    hardware,
    min_speed=25.0,
    top_n=10,
    quant_filter=None,
)
for r in results:
    print(f"{r.model_id}: {r.estimated_tok_per_sec:.1f} tok/s")

In this snippet, any variant estimated below 25 tok/s is silently dropped before the final ranked list is returned.

Summary

• The --min-speed CLI flag and min_speed Python parameter let you set a throughput floor in tokens per second.
• In src/whichllm/engine/ranker.py, the rank_models function skips variants when tok_per_sec < min_speed.
• The estimator estimate_tok_per_sec in src/whichllm/engine/performance.py predicts speed using your hardware profile from src/whichllm/hardware/detector.py.
• This filter composes cleanly with other flags such as --min-params and --evidence.

Frequently Asked Questions

What happens if I do not provide a minimum speed?

If min_speed is omitted, the filter is bypassed entirely. As implemented in src/whichllm/engine/ranker.py, the check if min_speed is not None and tok_per_sec < min_speed only executes when you supply an explicit value, so all ranked models are considered regardless of throughput.

What units does the --min-speed parameter use?

The parameter expects a float representing tokens per second (tok/s). This aligns with the return value of estimate_tok_per_sec in src/whichllm/engine/performance.py, so the comparison is always done in the same unit.

How does whichllm estimate a model's token generation speed?

Speed estimation is handled by estimate_tok_per_sec inside src/whichllm/engine/performance.py, which takes into account the active hardware detected by src/whichllm/hardware/detector.py. The estimate is hardware-aware, meaning a given model may receive different tok/s predictions on different GPUs or CPUs.

Can I combine --min-speed with other ranking filters?

Yes. The rank_models signature accepts multiple constraints simultaneously. In the CLI, flags such as --min-params and --evidence strict can be used alongside --min-speed, and in Python you can pass the corresponding arguments to rank_models in src/whichllm/engine/ranker.py without conflict.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}