# How to Download and Chat with a Model Using the WhichLLM Run Command

> Easily download and chat with LLMs using the whichllm run command. This tool handles downloads, dependencies, and launches chat sessions instantly. Get started now!

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: how-to-guide
- Published: 2026-06-10

---

**`whichllm run`** is a single-command interface that automatically downloads a compatible LLM, installs isolated runtime dependencies via UV, and launches an interactive chat session without modifying your project files.

The `whichllm run` command in the **Andyyyy64/whichllm** repository provides a zero-friction way to download and chat with a model using the WhichLLM run command architecture. It handles everything from hardware-aware model selection to dependency resolution, creating a temporary, self-contained environment for immediate inference.

## How the `whichllm run` Command Works

The command operates through a seven-stage pipeline defined in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py), gluing together model discovery, ranking, and isolated execution.

### Step 1: Model Discovery and Selection

First, **`_load_models()`** (lines 196-204) reads the cached model list or fetches fresh metadata from HuggingFace. If you provide a specific model name, **`_search_model()`** (lines 24-50) locates the exact or best-matching entry. Otherwise, the **`rank_models`** function from the ranker engine selects the optimal model for your detected hardware configuration.

### Step 2: Variant Resolution and Dependency Mapping

For GGUF-based models, **`_resolve_ranked_gguf_for_run()`** (lines 556-573) identifies a runnable file matching your requested quantization or falls back to the best available variant. Next, **`_resolve_model_deps()`** (lines 555-572) returns the minimal set of pip packages required—typically `llama-cpp-python` and `huggingface-hub` for GGUF files, or `transformers` and `torch` for non-GGUF models.

### Step 3: Script Generation and Isolated Execution

**`_generate_chat_script()`** (lines 774-842) creates a temporary Python script tailored to the model type (GGUF vs. Transformers). The CLI then invokes **`uv run --no-project`** (lines 1007-1017) with `--with` flags for each dependency, ensuring the script runs in an isolated environment without polluting your project's [`pyproject.toml`](https://github.com/Andyyyy64/whichllm/blob/main/pyproject.toml). The temporary file is automatically deleted after the chat exits.

### Step 4: Interactive Chat Session

The generated script loads the model using **`llama_cpp.Llama`** for GGUF files or **`AutoModelForCausalLM`** for HuggingFace models. It streams responses in real-time—using standard generation for GGUF or a `TextIteratorStreamer` for Transformers—until you type `exit`, `quit`, or `q`.

## Practical Usage Examples

Let the tool auto-select the best model for your hardware:

```bash
whichllm run

```

Specify a model by partial name match:

```bash
whichllm run "phi 3 mini gguf"

```

Force CPU-only execution on machines without GPU acceleration:

```bash
whichllm run "llama 3 8b gguf" --cpu-only

```

Request a specific quantization level (e.g., Q5_K_M):

```bash
whichllm run "qwen 2.5 1.5b gguf" --quant Q5_K_M

```

## Key Implementation Files

- **[`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py)** – Implements the core `run` command and helper functions including `_load_models`, `_search_model`, `_resolve_ranked_gguf_for_run`, `_resolve_model_deps`, and `_generate_chat_script`.
- **[`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py)** – Scores models based on hardware compatibility, benchmark evidence, and quantization quality; used when no explicit model is provided.
- **[`src/whichllm/models/types.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/models/types.py)** – Defines `ModelInfo` and `GGUFVariant` data structures manipulated by the CLI.
- **[`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py)** – Detects CPU, GPU, RAM, and OS details to inform the ranking process.

## Summary

- **`whichllm run`** combines model discovery, dependency resolution, and chat execution in one command.
- The workflow uses **`_load_models`** and **`_search_model`** for selection, or auto-ranks based on hardware if no name is provided.
- GGUF variants are resolved via **`_resolve_ranked_gguf_for_run`** with fallback logic for quantization levels.
- Dependencies are injected dynamically using **`uv run --no-project`**, ensuring complete isolation from your project environment.
- All core logic resides in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py), with hardware detection in [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py) and ranking logic in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py).

## Frequently Asked Questions

### What happens if I don't specify a model name?

If you run `whichllm run` without arguments, the command calls the ranker engine to evaluate your hardware via [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py) and selects the highest-scoring model from the cached list automatically.

### Does `whichllm run` modify my project's Python environment?

No. The command uses **UV** with the `--no-project` flag to create a temporary, isolated environment for the chat session. Dependencies are installed only for the duration of the run and do not affect your system or project packages.

### Can I force CPU-only execution?

Yes. Use the **`--cpu-only`** flag to override GPU detection. This ensures the model runs on CPU even if CUDA or Metal backends are available, which is useful for testing or on shared servers.

### What quantization formats are supported?

The tool primarily supports **GGUF** variants. You can request specific quantization levels (like Q4_K_M, Q5_K_M, or Q8_0) using the **`--quant`** flag. If the requested quantization is unavailable, the system falls back to the best available GGUF file for that model.