How to Download and Chat with a Model Using the WhichLLM Run Command

whichllm run is a single-command interface that automatically downloads a compatible LLM, installs isolated runtime dependencies via UV, and launches an interactive chat session without modifying your project files.

The whichllm run command in the Andyyyy64/whichllm repository provides a zero-friction way to download and chat with a model using the WhichLLM run command architecture. It handles everything from hardware-aware model selection to dependency resolution, creating a temporary, self-contained environment for immediate inference.

How the whichllm run Command Works

The command operates through a seven-stage pipeline defined in src/whichllm/cli.py, gluing together model discovery, ranking, and isolated execution.

Step 1: Model Discovery and Selection

First, _load_models() (lines 196-204) reads the cached model list or fetches fresh metadata from HuggingFace. If you provide a specific model name, _search_model() (lines 24-50) locates the exact or best-matching entry. Otherwise, the rank_models function from the ranker engine selects the optimal model for your detected hardware configuration.

Step 2: Variant Resolution and Dependency Mapping

For GGUF-based models, _resolve_ranked_gguf_for_run() (lines 556-573) identifies a runnable file matching your requested quantization or falls back to the best available variant. Next, _resolve_model_deps() (lines 555-572) returns the minimal set of pip packages required—typically llama-cpp-python and huggingface-hub for GGUF files, or transformers and torch for non-GGUF models.

Step 3: Script Generation and Isolated Execution

_generate_chat_script() (lines 774-842) creates a temporary Python script tailored to the model type (GGUF vs. Transformers). The CLI then invokes uv run --no-project (lines 1007-1017) with --with flags for each dependency, ensuring the script runs in an isolated environment without polluting your project's pyproject.toml. The temporary file is automatically deleted after the chat exits.

Step 4: Interactive Chat Session

The generated script loads the model using llama_cpp.Llama for GGUF files or AutoModelForCausalLM for HuggingFace models. It streams responses in real-time—using standard generation for GGUF or a TextIteratorStreamer for Transformers—until you type exit, quit, or q.

Practical Usage Examples

Let the tool auto-select the best model for your hardware:

whichllm run

Specify a model by partial name match:

whichllm run "phi 3 mini gguf"

Force CPU-only execution on machines without GPU acceleration:

whichllm run "llama 3 8b gguf" --cpu-only

Request a specific quantization level (e.g., Q5_K_M):

whichllm run "qwen 2.5 1.5b gguf" --quant Q5_K_M

Key Implementation Files

• src/whichllm/cli.py – Implements the core run command and helper functions including _load_models, _search_model, _resolve_ranked_gguf_for_run, _resolve_model_deps, and _generate_chat_script.
• src/whichllm/engine/ranker.py – Scores models based on hardware compatibility, benchmark evidence, and quantization quality; used when no explicit model is provided.
• src/whichllm/models/types.py – Defines ModelInfo and GGUFVariant data structures manipulated by the CLI.
• src/whichllm/hardware/detector.py – Detects CPU, GPU, RAM, and OS details to inform the ranking process.

Summary

• whichllm run combines model discovery, dependency resolution, and chat execution in one command.
• The workflow uses _load_models and _search_model for selection, or auto-ranks based on hardware if no name is provided.
• GGUF variants are resolved via _resolve_ranked_gguf_for_run with fallback logic for quantization levels.
• Dependencies are injected dynamically using uv run --no-project, ensuring complete isolation from your project environment.
• All core logic resides in src/whichllm/cli.py, with hardware detection in src/whichllm/hardware/detector.py and ranking logic in src/whichllm/engine/ranker.py.

Frequently Asked Questions

What happens if I don't specify a model name?

If you run whichllm run without arguments, the command calls the ranker engine to evaluate your hardware via src/whichllm/hardware/detector.py and selects the highest-scoring model from the cached list automatically.

Does whichllm run modify my project's Python environment?

No. The command uses UV with the --no-project flag to create a temporary, isolated environment for the chat session. Dependencies are installed only for the duration of the run and do not affect your system or project packages.

Can I force CPU-only execution?

Yes. Use the --cpu-only flag to override GPU detection. This ensures the model runs on CPU even if CUDA or Metal backends are available, which is useful for testing or on shared servers.

What quantization formats are supported?

The tool primarily supports GGUF variants. You can request specific quantization levels (like Q4_K_M, Q5_K_M, or Q8_0) using the --quant flag. If the requested quantization is unavailable, the system falls back to the best available GGUF file for that model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}