# How to Filter LLM Rankings by Direct Benchmark Evidence in whichllm

> Filter whichllm rankings by direct benchmark evidence using strict CLI flags This ensures reliable LLM performance data from trusted sources like the Open LLM Leaderboard

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: how-to-guide
- Published: 2026-06-09

---

**Use the `--evidence strict` or `--direct` CLI flags to restrict whichllm rankings to only those models with independently verified benchmark scores from sources like the Open LLM Leaderboard.**

The `whichllm` tool ranks large language models using a weighted combination of hardware compatibility, performance benchmarks, and quantization penalties. When you need to ensure the recommendations rely on trustworthy, independently verified data rather than self-reported metrics, you can filter LLM rankings by direct benchmark evidence in whichllm to eliminate models with inherited or unverified scores.

## Understanding Benchmark Evidence Levels

The ranker categorizes benchmark provenance into six distinct source types. The **evidence filter** determines which of these sources are acceptable when calculating final scores:

| Filter Mode | Accepted Sources | Use Case |
|-------------|------------------|----------|
| `any` (default) | `direct`, `variant`, `base_model`, `line_interp`, `self_reported`, `none` | Maximum coverage, including speculative scores |
| `base` | `direct`, `variant`, `base_model` | Moderate trust, excluding interpolated or self-reported data |
| `strict` | **Only** `direct` | Maximum confidence, requiring independent leaderboard verification |

**Direct evidence** indicates that the score originates from an independent leaderboard such as the Open LLM Leaderboard, making it the most trustworthy tier. When you activate strict filtering, the system discards any model whose benchmarks are self-reported, inherited from a base model, or algorithmically interpolated.

## Command-Line Filtering

To apply strict filtering from the terminal, use either the explicit evidence flag or its backward-compatible alias. Both commands produce identical results, displaying only models that have direct benchmark verification.

```bash

# Explicit strict mode

whichllm --evidence strict

# Legacy alias for backward compatibility

whichllm --direct

```

These flags are parsed in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) by the `_resolve_evidence_mode` function, which validates the input and maps `--direct` to the `"strict"` internal value. The resulting mode is then passed to the ranking engine.

## Programmatic Filtering with the Python API

When integrating whichllm into a Python application, pass the `evidence_filter` parameter to the `rank_models` function. Set it to `"strict"` to enforce direct evidence requirements.

```python
from whichllm.engine.ranker import rank_models
from whichllm.hardware.detector import detect_hardware
from whichllm.models.fetcher import fetch_models

# Detect local hardware capabilities

hardware = detect_hardware()

# Load model metadata (cached or fresh)

models = fetch_models(include_vision=True)

# Rank with strict evidence filtering

results = rank_models(
    models,
    hardware,
    top_n=10,
    evidence_filter="strict",  # Only allows direct benchmark sources

)

# Verify the filter applied correctly

for r in results:
    print(f"{r.model.id}: {r.benchmark_status} (score: {r.quality_score:.1f})")

```

In this configuration, every result in the `results` list will have `r.benchmark_status` set to `"direct"`, confirming the filter excluded all lesser evidence types.

## Internal Implementation Details

The filtering mechanism operates across two primary components in the source code.

**CLI Resolution** ([`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py)):
The `_resolve_evidence_mode` function handles argument parsing, normalizing both `--evidence strict` and `--direct` into the internal `"strict"` mode before passing it downstream.

**Filtering Logic** ([`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py)):
The `rank_models` function receives the `evidence_filter` parameter and applies it through the `_passes_evidence_filter(source, evidence_filter)` helper. This function returns `True` only when a model's benchmark source satisfies the current filter criteria. For strict mode, it explicitly checks that `source == "direct"`, rejecting entries with `self_reported`, `variant`, `base_model`, or `line_interp` provenance.

## Summary

- **Strict filtering** (`--evidence strict` or `--direct`) limits results to models with independently verified benchmark scores.
- The filter is implemented in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) via the `_passes_evidence_filter` function.
- Three evidence modes exist: `any` (default), `base`, and `strict`.
- When using the Python API, set `evidence_filter="strict"` in the `rank_models` call.
- Direct evidence indicates verification by external leaderboards like the Open LLM Leaderboard.

## Frequently Asked Questions

### What is the difference between `--evidence strict` and `--direct`?

There is no functional difference. The `--direct` flag is a legacy alias maintained for backward compatibility, while `--evidence strict` is the explicit, recommended syntax. Both are resolved to the same internal mode in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py).

### Which benchmark sources are excluded when using strict filtering?

Strict mode excludes `self_reported` (manufacturer claims), `variant` (inherited from fine-tunes), `base_model` (projected from parent architectures), `line_interp` (algorithmically interpolated scores), and `none` (missing data). Only `direct` leaderboard results are retained.

### Can I combine the evidence filter with other ranking parameters?

Yes. The `evidence_filter` parameter works alongside hardware detection, `top_n` limits, and vision model flags. You can request the top 10 vision-capable models that fit your GPU while simultaneously requiring strict benchmark evidence.

### How do I verify that a specific result passed the evidence filter?

When using the Python API, inspect the `benchmark_status` or `benchmark_source` attributes on the result object. If `r.benchmark_status` equals `"direct"`, the model satisfied the strict filter. In the CLI output, look for the "🟢 direct" badge accompanying the model entry.