How to Filter LLM Rankings by Direct Benchmark Evidence in whichllm

Use the --evidence strict or --direct CLI flags to restrict whichllm rankings to only those models with independently verified benchmark scores from sources like the Open LLM Leaderboard.

The whichllm tool ranks large language models using a weighted combination of hardware compatibility, performance benchmarks, and quantization penalties. When you need to ensure the recommendations rely on trustworthy, independently verified data rather than self-reported metrics, you can filter LLM rankings by direct benchmark evidence in whichllm to eliminate models with inherited or unverified scores.

Understanding Benchmark Evidence Levels

The ranker categorizes benchmark provenance into six distinct source types. The evidence filter determines which of these sources are acceptable when calculating final scores:

Filter Mode	Accepted Sources	Use Case
any (default)	direct, variant, base_model, line_interp, self_reported, none	Maximum coverage, including speculative scores
base	direct, variant, base_model	Moderate trust, excluding interpolated or self-reported data
strict	Only direct	Maximum confidence, requiring independent leaderboard verification

Direct evidence indicates that the score originates from an independent leaderboard such as the Open LLM Leaderboard, making it the most trustworthy tier. When you activate strict filtering, the system discards any model whose benchmarks are self-reported, inherited from a base model, or algorithmically interpolated.

Command-Line Filtering

To apply strict filtering from the terminal, use either the explicit evidence flag or its backward-compatible alias. Both commands produce identical results, displaying only models that have direct benchmark verification.

# Explicit strict mode

whichllm --evidence strict

# Legacy alias for backward compatibility

whichllm --direct

These flags are parsed in src/whichllm/cli.py by the _resolve_evidence_mode function, which validates the input and maps --direct to the "strict" internal value. The resulting mode is then passed to the ranking engine.

Programmatic Filtering with the Python API

When integrating whichllm into a Python application, pass the evidence_filter parameter to the rank_models function. Set it to "strict" to enforce direct evidence requirements.

from whichllm.engine.ranker import rank_models
from whichllm.hardware.detector import detect_hardware
from whichllm.models.fetcher import fetch_models

# Detect local hardware capabilities

hardware = detect_hardware()

# Load model metadata (cached or fresh)

models = fetch_models(include_vision=True)

# Rank with strict evidence filtering

results = rank_models(
    models,
    hardware,
    top_n=10,
    evidence_filter="strict",  # Only allows direct benchmark sources

)

# Verify the filter applied correctly

for r in results:
    print(f"{r.model.id}: {r.benchmark_status} (score: {r.quality_score:.1f})")

In this configuration, every result in the results list will have r.benchmark_status set to "direct", confirming the filter excluded all lesser evidence types.

Internal Implementation Details

The filtering mechanism operates across two primary components in the source code.

CLI Resolution (src/whichllm/cli.py): The _resolve_evidence_mode function handles argument parsing, normalizing both --evidence strict and --direct into the internal "strict" mode before passing it downstream.

Filtering Logic (src/whichllm/engine/ranker.py): The rank_models function receives the evidence_filter parameter and applies it through the _passes_evidence_filter(source, evidence_filter) helper. This function returns True only when a model's benchmark source satisfies the current filter criteria. For strict mode, it explicitly checks that source == "direct", rejecting entries with self_reported, variant, base_model, or line_interp provenance.

Summary

• Strict filtering (--evidence strict or --direct) limits results to models with independently verified benchmark scores.
• The filter is implemented in src/whichllm/engine/ranker.py via the _passes_evidence_filter function.
• Three evidence modes exist: any (default), base, and strict.
• When using the Python API, set evidence_filter="strict" in the rank_models call.
• Direct evidence indicates verification by external leaderboards like the Open LLM Leaderboard.

Frequently Asked Questions

What is the difference between --evidence strict and --direct?

There is no functional difference. The --direct flag is a legacy alias maintained for backward compatibility, while --evidence strict is the explicit, recommended syntax. Both are resolved to the same internal mode in src/whichllm/cli.py.

Which benchmark sources are excluded when using strict filtering?

Strict mode excludes self_reported (manufacturer claims), variant (inherited from fine-tunes), base_model (projected from parent architectures), line_interp (algorithmically interpolated scores), and none (missing data). Only direct leaderboard results are retained.

Can I combine the evidence filter with other ranking parameters?

Yes. The evidence_filter parameter works alongside hardware detection, top_n limits, and vision model flags. You can request the top 10 vision-capable models that fit your GPU while simultaneously requiring strict benchmark evidence.

How do I verify that a specific result passed the evidence filter?

When using the Python API, inspect the benchmark_status or benchmark_source attributes on the result object. If r.benchmark_status equals "direct", the model satisfied the strict filter. In the CLI output, look for the "🟢 direct" badge accompanying the model entry.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}