# How to Run a Specific LLM Model Using whichllm: A Complete Guide > Learn how to run a specific LLM model with whichllm. Effortlessly download, configure, and execute Hugging Face models in isolated environments with optimal settings. - Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm) - Tags: how-to-guide - Published: 2026-06-09 --- **Use `whichllm run ` to automatically download, configure, and execute any Hugging Face LLM in an isolated environment with optimal quantization and hardware-specific dependencies.** The `whichllm` CLI tool by Andyyyy64 automates the entire workflow of locating, configuring, and executing large language models. When you need to run a specific LLM model using whichllm, the tool handles model discovery from Hugging Face, selects appropriate GGUF quantizations, generates executable Python scripts, and manages dependencies through isolated `uv` environments. ## The CLI Entry Point and Workflow The `run` command in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) (lines 80-104) serves as the single entry point for executing models. This command orchestrates a **cache-first pipeline** that minimizes redundant network calls while ensuring you always get the best available variant for your hardware. The execution flow follows this sequence: 1. **Load model metadata** via `_load_models` (lines 107-119), which checks local cache before fetching fresh data from Hugging Face. 2. **Search or rank** to identify the target model using `_search_model` (lines 24-53) for exact matches or the ranking engine when auto-selecting. 3. **Resolve GGUF variants** through `_resolve_ranked_gguf_for_run` (lines 110-152) to pick the appropriate quantization file. 4. **Determine dependencies** via `_resolve_model_deps` (lines 55-71) to identify exact Python packages required. 5. **Generate and execute** a self-contained chat script using `_generate_chat_script` (lines 174-277) inside an isolated `uv` environment. ## Locating Models with Fuzzy Search You don’t need to remember exact Hugging Face repository IDs. The `_search_model` function implements **fuzzy matching** that accepts full repo IDs, partial names, or suffixes. When you execute: ```bash whichllm run "Qwen1.5-7B" ``` The function searches cached model metadata and returns the best match, handling cases where you omit the organization prefix or use common abbreviated names. This search operates against the local cache populated by `_load_models`, ensuring sub-second response times after the initial fetch. ## GGUF Quantization and Variant Resolution `whichllm` prioritizes **GGUF binaries** for efficient local inference through `llama-cpp-python`. The `_resolve_ranked_gguf_for_run` function translates your model selection into a specific downloadable file. **Key capabilities include:** - **Automatic quantization selection**: If you don’t specify a quant, the tool picks the optimal balance of size and quality (typically Q4_K_M). - **Explicit quantization override**: Use the `--quant` flag to force a specific variant like Q8_0 or Q5_K_M. - **Fallback handling**: If your requested quantization doesn’t exist, the system warns you and falls back to the nearest available match. For models without GGUF variants, `whichllm` automatically generates a Transformers-based script using PyTorch or TensorFlow instead. ## Hardware-Aware Model Selection When you omit the model name entirely, `whichllm` uses **intelligent ranking** to select the best model for your specific hardware configuration: ```bash whichllm run ``` This triggers the hardware detection system in [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py), which analyzes your CPU, GPU VRAM, system RAM, and disk space. The `rank_models` function in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) then scores all available models against these capabilities, selecting the highest-ranked option that will actually fit and run efficiently on your machine. You can also simulate different hardware scenarios using flags like `--cpu-only` to force CPU-only execution, which sets `n_gpu_layers=0` for GGUF models or `device_map="cpu"` for Transformers implementations. ## Script Generation and Isolated Execution The `_generate_chat_script` function creates **self-contained Python scripts** that handle everything from model download to interactive chat loops. For GGUF models, this script: 1. Downloads the specific `.gguf` file using `hf_hub_download` 2. Initializes the `llama_cpp.Llama` class with appropriate context length and GPU layer settings 3. Starts an interactive REPL for conversation For Transformers models, it generates equivalent code using `AutoModelForCausalLM` and `AutoTokenizer`. Crucially, `whichllm` never installs dependencies into your global Python environment. Instead, it constructs a `uv run --with ` command that executes the generated script in a temporary, isolated environment with exact dependency versions. This approach guarantees reproducibility and prevents dependency conflicts. ## Practical Usage Examples ### Run a specific model with default quantization ```bash whichllm run "Qwen/Qwen1.5-7B-Chat" ``` This locates the model, selects the optimal GGUF quantization, and starts an interactive chat session. ### Force a specific quantization level ```bash whichllm run "Meta-Llama-3-8B-Instruct" --quant Q8_0 ``` The `--quant` flag overrides automatic selection, downloading the Q8_0 variant instead of the default Q4_K_M. ### Execute on CPU-only systems ```bash whichllm run "Mistral-7B-Instruct-v0.1" --cpu-only ``` This disables GPU acceleration by setting `n_gpu_layers=0` in the generated GGUF script, ensuring compatibility with systems lacking CUDA or Metal support. ### Auto-select the best model for your hardware ```bash whichllm run ``` When you provide no model argument, the tool detects your hardware, ranks all available models, and selects the top performer that fits within your VRAM and RAM constraints. ### Inspect the generated code before running ```bash whichllm snippet "Gemma-2-9B-it" ``` The `snippet` command prints the exact Python script and `uv` dependencies that would be executed, allowing you to review or customize the code manually. ## Summary To run a specific LLM model using whichllm, you interact with a sophisticated pipeline that handles the complexities of model discovery, quantization selection, and dependency management: - **Use `whichllm run `** to execute any Hugging Face model with automatic GGUF selection and isolated `uv` environments. - **Leverage fuzzy search** so you don’t need exact repository IDs—partial names and suffixes work via `_search_model` in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py). - **Specify `--quant`** to override automatic quantization selection when you need specific quality/size tradeoffs. - **Enable `--cpu-only`** for systems without compatible GPUs, forcing CPU inference paths. - **Omit the model name** entirely to trigger hardware-aware ranking that selects the optimal model for your specific CPU, GPU, and memory configuration. - **Inspect scripts first** with `whichllm snippet` to see exactly what code will execute before downloading multi-gigabyte model files. ## Frequently Asked Questions ### How does whichllm decide which GGUF quantization to use? **If you don’t specify `--quant`, the `_pick_gguf_variant` function selects a default quantization (typically Q4_K_M) that balances quality and file size.** If you request a specific quantization that isn’t available, the system warns you and falls back to the closest available match. This logic resides in the GGUF resolution path of [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) (lines 110-152). ### Can I run whichllm without installing model dependencies globally? **Yes, `whichllm` uses `uv` to create isolated execution environments for every model run.** The CLI generates a temporary Python script and executes it via `uv run --with `, ensuring that packages like `llama-cpp-python` or `transformers` never pollute your global Python installation. This isolation happens automatically in the `run` command block (lines 106-118 of [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py)). ### What happens if I don’t specify a model name? **The tool enters hardware-aware selection mode.** It calls `detect_hardware()` from [`src/whichllm/hardware/detector.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/hardware/detector.py) to analyze your system, then uses `rank_models` from [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) to score all known models against your capabilities. The highest-ranked model that fits within your VRAM and RAM constraints is automatically selected and executed. ### How do I verify what code will run before downloading a model? **Use the `whichllm snippet` command.** This invokes the same `_generate_chat_script` function used by `run`, but instead of executing the code, it prints the complete Python script and required dependencies to your terminal. You can review the exact download URLs, loading logic, and chat loop implementation before committing to multi-gigabyte downloads.