How to Auto-Pick the Best LLM and Start a Chat with whichllm

Run whichllm run to automatically detect your hardware, rank available models against your system capabilities, and launch an interactive chat session with the optimal locally runnable LLM.

The whichllm open-source tool eliminates the guesswork from running large language models locally by intelligently matching your hardware constraints against the Hugging Face model ecosystem. Instead of manually comparing VRAM requirements and quantization formats, you can rely on the library’s built-in ranking engine to auto-pick the best LLM and start a chat with a single command. This workflow combines hardware introspection, evidence-based scoring, and automatic runtime environment setup to get you chatting within minutes.

The Three-Stage Auto-Pick Pipeline

When you invoke the auto-pick functionality, whichllm executes a deterministic pipeline composed of detection, ranking, and execution phases. Each phase is implemented in dedicated modules under the src/whichllm/ directory.

Hardware Detection

The process begins in src/whichllm/hardware/detector.py with the detect_hardware() function. This routine inspects your CPU architecture, available RAM, operating system, and GPU configuration (including CUDA, ROCm, or Metal backends). The function returns a HardwareInfo object that encapsulates your machine’s compute profile, which subsequent stages use as a constraint matrix.

You can optionally override detected values using CLI flags:

whichllm run --gpu "RTX 4090" --vram 24

Model Ranking

The core decision logic resides in src/whichllm/engine/ranker.py inside the rank_models() function. This algorithm scores every candidate model from the Hugging Face catalog against your hardware profile and returns the top-N results (default 10) as CompatibilityResult objects.

The ranking combines multiple evidence sources:

• Benchmark evidence — Leaderboard scores and line-family inference handled in src/whichllm/models/benchmark.py
• Quantization penalties — Quality degradation curves defined in src/whichllm/engine/quantization.py
• Fit-type assessment — Classification into full-GPU, partial-offload, or CPU-only execution modes from src/whichllm/engine/compatibility.py
• Speed estimation — Tok/sec predictions based on quantization level and compute type from src/whichllm/engine/performance.py
• Metadata signals — Model popularity, generation lineage, and organization trust scores

Each CompatibilityResult contains the selected model ID, the optimal GGUF variant (or transformer fallback), and a composite quality score.

Chat Script Generation and Execution

If you invoke whichllm run without specifying a model name, the CLI entry point in src/whichllm/cli.py orchestrates the final stage:

1. Calls _resolve_ranked_gguf_for_run() to pick the highest-scoring entry
2. Generates a self-contained Python script via _generate_chat_script() that uses llama-cpp-python for GGUF files or transformers for safetensors
3. Spins up an isolated environment using uv with the minimal required dependencies (e.g., llama-cpp-python and huggingface-hub)
4. Executes the script and presents an interactive prompt

Auto-Picking via the Command Line

The simplest way to auto-pick the best LLM and start a chat is the bare command:

whichllm run

This triggers the full pipeline: hardware detection, model ranking, dependency installation, and chat initialization.

Constraining the Quantization

To prefer a specific quantization format while still auto-picking the model:

whichllm run --quant Q4_K_M

The ranker will prioritize GGUF variants matching this quantization type, falling back to the next best fit if unavailable.

Simulating Different Hardware

For "what-if" analysis or testing configurations before purchasing hardware:

whichllm run --gpu "RTX 4090" --vram 24

The detector will instantiate a HardwareInfo object with your specified overrides rather than querying the actual system.

Programmatic Auto-Picking in Python

You can embed the auto-pick logic into your own applications by importing the detection and ranking modules directly:

from whichllm.hardware.detector import detect_hardware
from whichllm.engine.ranker import rank_models
from whichllm.models.fetcher import fetch_models, dicts_to_models

# 1️⃣ Detect hardware

hw = detect_hardware()

# 2️⃣ Load model catalog from Hugging Face

models = dicts_to_models(fetch_models(include_vision=False))

# 3️⃣ Rank and select the best candidate

top_result = rank_models(models, hw, top_n=1)[0]

print(f"Best model: {top_result.model.id} ({top_result.gguf_variant.quant_type})")
print(f"Estimated speed: {top_result.estimated_tok_per_sec:.1f} tokens/s")

This pattern is useful for building UIs, automated benchmarking pipelines, or custom deployment scripts that need to know the optimal model before allocating resources.

Core Implementation Files

The auto-pick workflow is distributed across these key source files:

• CLI entry point — src/whichllm/cli.py (whichllm run implementation)
• Hardware detection — src/whichllm/hardware/detector.py (detect_hardware())
• Ranking engine — src/whichllm/engine/ranker.py (rank_models())
• Compatibility logic — src/whichllm/engine/compatibility.py (fit-type assessment)
• Benchmark data — src/whichllm/models/benchmark.py (score lookup and inference)
• Quantization handling — src/whichllm/engine/quantization.py (quality penalties)
• Utilities — src/whichllm/utils.py (version and context length helpers)

Summary

• Single-command chat — whichllm run auto-detects hardware, ranks models, and starts an interactive session
• Evidence-based ranking — Combines benchmark scores, quantization quality, fit-type, and speed estimates to select the optimal model
• Flexible overrides — Use --gpu, --vram, and --quant flags to simulate different hardware or enforce quantization preferences
• Programmatic API — Import detect_hardware() and rank_models() to build custom selection logic into Python applications
• Isolated execution — Automatically manages dependencies via uv and generates runnable scripts for llama-cpp-python or transformers

Frequently Asked Questions

How does whichllm determine which LLM is "best" for my hardware?

The rank_models() function in src/whichllm/engine/ranker.py calculates a composite score for each candidate based on benchmark evidence, quantization penalties, fit-type classification (whether the model fits fully in GPU, requires partial offload, or runs CPU-only), estimated tokens-per-second, and metadata signals like popularity and lineage. The highest-scoring model that fits within your VRAM and RAM constraints is selected as the best option.

Can I override the automatic hardware detection?

Yes. While detect_hardware() in src/whichllm/hardware/detector.py automatically inspects your system, you can override GPU name and VRAM using the --gpu and --vram flags. This is useful for testing configurations or running on cloud instances where you want to preview performance before provisioning.

What happens if I don't have a GPU?

If no GPU is detected, the HardwareInfo object reports CPU-only capabilities. The ranker then filters for models that can run entirely in system RAM, typically selecting smaller quantized GGUF variants or compact transformer models. The chat script will use CPU-optimized inference backends automatically.

Can I use a specific model instead of auto-picking?

Absolutely. If you specify a model ID with whichllm run <model_id>, the CLI bypasses the ranking stage and proceeds directly to compatibility checking and chat script generation for that specific model. The auto-pick logic only triggers when you omit the model argument.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}