# How to Auto-Pick the Best LLM and Start a Chat with whichllm

> Effortlessly auto pick the best LLM for your hardware and start a chat with whichllm. Detect capabilities and launch optimal local LLMs with a single command.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: how-to-guide
- Published: 2026-06-09

---

**Run `whichllm run` to automatically detect your hardware, rank available models against your system capabilities, and launch an interactive chat session with the optimal locally runnable LLM.**

The `whichllm` open-source tool eliminates the guesswork from running large language models locally by intelligently matching your hardware constraints against the Hugging Face model ecosystem. Instead of manually comparing VRAM requirements and quantization formats, you can rely on the library’s built-in ranking engine to auto-pick the best LLM and start a chat with a single command. This workflow combines hardware introspection, evidence-based scoring, and automatic runtime environment setup to get you chatting within minutes.

## The Three-Stage Auto-Pick Pipeline

When you invoke the auto-pick functionality, `whichllm` executes a deterministic pipeline composed of detection, ranking, and execution phases. Each phase is implemented in dedicated modules under the `src/whichllm/` directory.

### Hardware Detection

The process begins in [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py) with the `detect_hardware()` function. This routine inspects your CPU architecture, available RAM, operating system, and GPU configuration (including CUDA, ROCm, or Metal backends). The function returns a `HardwareInfo` object that encapsulates your machine’s compute profile, which subsequent stages use as a constraint matrix.

You can optionally override detected values using CLI flags:

```bash
whichllm run --gpu "RTX 4090" --vram 24

```

### Model Ranking

The core decision logic resides in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) inside the `rank_models()` function. This algorithm scores every candidate model from the Hugging Face catalog against your hardware profile and returns the top-N results (default 10) as `CompatibilityResult` objects.

The ranking combines multiple evidence sources:

*   **Benchmark evidence** — Leaderboard scores and line-family inference handled in [`src/whichllm/models/benchmark.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/benchmark.py)
*   **Quantization penalties** — Quality degradation curves defined in [`src/whichllm/engine/quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/quantization.py)
*   **Fit-type assessment** — Classification into full-GPU, partial-offload, or CPU-only execution modes from [`src/whichllm/engine/compatibility.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/compatibility.py)
*   **Speed estimation** — Tok/sec predictions based on quantization level and compute type from [`src/whichllm/engine/performance.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/performance.py)
*   **Metadata signals** — Model popularity, generation lineage, and organization trust scores

Each `CompatibilityResult` contains the selected model ID, the optimal GGUF variant (or transformer fallback), and a composite quality score.

### Chat Script Generation and Execution

If you invoke `whichllm run` without specifying a model name, the CLI entry point in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) orchestrates the final stage:

1.  Calls `_resolve_ranked_gguf_for_run()` to pick the highest-scoring entry
2.  Generates a self-contained Python script via `_generate_chat_script()` that uses **llama-cpp-python** for GGUF files or **transformers** for safetensors
3.  Spins up an isolated environment using **uv** with the minimal required dependencies (e.g., `llama-cpp-python` and `huggingface-hub`)
4.  Executes the script and presents an interactive prompt

## Auto-Picking via the Command Line

The simplest way to auto-pick the best LLM and start a chat is the bare command:

```bash
whichllm run

```

This triggers the full pipeline: hardware detection, model ranking, dependency installation, and chat initialization.

### Constraining the Quantization

To prefer a specific quantization format while still auto-picking the model:

```bash
whichllm run --quant Q4_K_M

```

The ranker will prioritize GGUF variants matching this quantization type, falling back to the next best fit if unavailable.

### Simulating Different Hardware

For "what-if" analysis or testing configurations before purchasing hardware:

```bash
whichllm run --gpu "RTX 4090" --vram 24

```

The detector will instantiate a `HardwareInfo` object with your specified overrides rather than querying the actual system.

## Programmatic Auto-Picking in Python

You can embed the auto-pick logic into your own applications by importing the detection and ranking modules directly:

```python
from whichllm.hardware.detector import detect_hardware
from whichllm.engine.ranker import rank_models
from whichllm.models.fetcher import fetch_models, dicts_to_models

# 1️⃣ Detect hardware

hw = detect_hardware()

# 2️⃣ Load model catalog from Hugging Face

models = dicts_to_models(fetch_models(include_vision=False))

# 3️⃣ Rank and select the best candidate

top_result = rank_models(models, hw, top_n=1)[0]

print(f"Best model: {top_result.model.id} ({top_result.gguf_variant.quant_type})")
print(f"Estimated speed: {top_result.estimated_tok_per_sec:.1f} tokens/s")

```

This pattern is useful for building UIs, automated benchmarking pipelines, or custom deployment scripts that need to know the optimal model before allocating resources.

## Core Implementation Files

The auto-pick workflow is distributed across these key source files:

*   **CLI entry point** — [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) (`whichllm run` implementation)
*   **Hardware detection** — [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py) (`detect_hardware()`)
*   **Ranking engine** — [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) (`rank_models()`)
*   **Compatibility logic** — [`src/whichllm/engine/compatibility.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/compatibility.py) (fit-type assessment)
*   **Benchmark data** — [`src/whichllm/models/benchmark.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/benchmark.py) (score lookup and inference)
*   **Quantization handling** — [`src/whichllm/engine/quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/quantization.py) (quality penalties)
*   **Utilities** — [`src/whichllm/utils.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/utils.py) (version and context length helpers)

## Summary

*   **Single-command chat** — `whichllm run` auto-detects hardware, ranks models, and starts an interactive session
*   **Evidence-based ranking** — Combines benchmark scores, quantization quality, fit-type, and speed estimates to select the optimal model
*   **Flexible overrides** — Use `--gpu`, `--vram`, and `--quant` flags to simulate different hardware or enforce quantization preferences
*   **Programmatic API** — Import `detect_hardware()` and `rank_models()` to build custom selection logic into Python applications
*   **Isolated execution** — Automatically manages dependencies via `uv` and generates runnable scripts for `llama-cpp-python` or `transformers`

## Frequently Asked Questions

### How does whichllm determine which LLM is "best" for my hardware?

The `rank_models()` function in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) calculates a composite score for each candidate based on benchmark evidence, quantization penalties, fit-type classification (whether the model fits fully in GPU, requires partial offload, or runs CPU-only), estimated tokens-per-second, and metadata signals like popularity and lineage. The highest-scoring model that fits within your VRAM and RAM constraints is selected as the best option.

### Can I override the automatic hardware detection?

Yes. While `detect_hardware()` in [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py) automatically inspects your system, you can override GPU name and VRAM using the `--gpu` and `--vram` flags. This is useful for testing configurations or running on cloud instances where you want to preview performance before provisioning.

### What happens if I don't have a GPU?

If no GPU is detected, the `HardwareInfo` object reports CPU-only capabilities. The ranker then filters for models that can run entirely in system RAM, typically selecting smaller quantized GGUF variants or compact transformer models. The chat script will use CPU-optimized inference backends automatically.

### Can I use a specific model instead of auto-picking?

Absolutely. If you specify a model ID with `whichllm run <model_id>`, the CLI bypasses the ranking stage and proceeds directly to compatibility checking and chat script generation for that specific model. The auto-pick logic only triggers when you omit the model argument.