How to Run a Specific LLM Model Using whichllm: A Complete Guide

Use whichllm run <model-name> to automatically download, configure, and execute any Hugging Face LLM in an isolated environment with optimal quantization and hardware-specific dependencies.

The whichllm CLI tool by Andyyyy64 automates the entire workflow of locating, configuring, and executing large language models. When you need to run a specific LLM model using whichllm, the tool handles model discovery from Hugging Face, selects appropriate GGUF quantizations, generates executable Python scripts, and manages dependencies through isolated uv environments.

The CLI Entry Point and Workflow

The run command in src/whichllm/cli.py (lines 80-104) serves as the single entry point for executing models. This command orchestrates a cache-first pipeline that minimizes redundant network calls while ensuring you always get the best available variant for your hardware.

The execution flow follows this sequence:

1. Load model metadata via _load_models (lines 107-119), which checks local cache before fetching fresh data from Hugging Face.
2. Search or rank to identify the target model using _search_model (lines 24-53) for exact matches or the ranking engine when auto-selecting.
3. Resolve GGUF variants through _resolve_ranked_gguf_for_run (lines 110-152) to pick the appropriate quantization file.
4. Determine dependencies via _resolve_model_deps (lines 55-71) to identify exact Python packages required.
5. Generate and execute a self-contained chat script using _generate_chat_script (lines 174-277) inside an isolated uv environment.

Locating Models with Fuzzy Search

You don’t need to remember exact Hugging Face repository IDs. The _search_model function implements fuzzy matching that accepts full repo IDs, partial names, or suffixes.

When you execute:

whichllm run "Qwen1.5-7B"

The function searches cached model metadata and returns the best match, handling cases where you omit the organization prefix or use common abbreviated names. This search operates against the local cache populated by _load_models, ensuring sub-second response times after the initial fetch.

GGUF Quantization and Variant Resolution

whichllm prioritizes GGUF binaries for efficient local inference through llama-cpp-python. The _resolve_ranked_gguf_for_run function translates your model selection into a specific downloadable file.

Key capabilities include:

• Automatic quantization selection: If you don’t specify a quant, the tool picks the optimal balance of size and quality (typically Q4_K_M).
• Explicit quantization override: Use the --quant flag to force a specific variant like Q8_0 or Q5_K_M.
• Fallback handling: If your requested quantization doesn’t exist, the system warns you and falls back to the nearest available match.

For models without GGUF variants, whichllm automatically generates a Transformers-based script using PyTorch or TensorFlow instead.

Hardware-Aware Model Selection

When you omit the model name entirely, whichllm uses intelligent ranking to select the best model for your specific hardware configuration:

whichllm run

This triggers the hardware detection system in src/whichllm/hardware/detector.py, which analyzes your CPU, GPU VRAM, system RAM, and disk space. The rank_models function in src/whichllm/engine/ranker.py then scores all available models against these capabilities, selecting the highest-ranked option that will actually fit and run efficiently on your machine.

You can also simulate different hardware scenarios using flags like --cpu-only to force CPU-only execution, which sets n_gpu_layers=0 for GGUF models or device_map="cpu" for Transformers implementations.

Script Generation and Isolated Execution

The _generate_chat_script function creates self-contained Python scripts that handle everything from model download to interactive chat loops. For GGUF models, this script:

1. Downloads the specific .gguf file using hf_hub_download
2. Initializes the llama_cpp.Llama class with appropriate context length and GPU layer settings
3. Starts an interactive REPL for conversation

For Transformers models, it generates equivalent code using AutoModelForCausalLM and AutoTokenizer.

Crucially, whichllm never installs dependencies into your global Python environment. Instead, it constructs a uv run --with <deps> command that executes the generated script in a temporary, isolated environment with exact dependency versions. This approach guarantees reproducibility and prevents dependency conflicts.

Practical Usage Examples

Run a specific model with default quantization

whichllm run "Qwen/Qwen1.5-7B-Chat"

This locates the model, selects the optimal GGUF quantization, and starts an interactive chat session.

Force a specific quantization level

whichllm run "Meta-Llama-3-8B-Instruct" --quant Q8_0

The --quant flag overrides automatic selection, downloading the Q8_0 variant instead of the default Q4_K_M.

Execute on CPU-only systems

whichllm run "Mistral-7B-Instruct-v0.1" --cpu-only

This disables GPU acceleration by setting n_gpu_layers=0 in the generated GGUF script, ensuring compatibility with systems lacking CUDA or Metal support.

Auto-select the best model for your hardware

whichllm run

When you provide no model argument, the tool detects your hardware, ranks all available models, and selects the top performer that fits within your VRAM and RAM constraints.

Inspect the generated code before running

whichllm snippet "Gemma-2-9B-it"

The snippet command prints the exact Python script and uv dependencies that would be executed, allowing you to review or customize the code manually.

Summary

To run a specific LLM model using whichllm, you interact with a sophisticated pipeline that handles the complexities of model discovery, quantization selection, and dependency management:

• Use whichllm run <model> to execute any Hugging Face model with automatic GGUF selection and isolated uv environments.
• Leverage fuzzy search so you don’t need exact repository IDs—partial names and suffixes work via _search_model in src/whichllm/cli.py.
• Specify --quant to override automatic quantization selection when you need specific quality/size tradeoffs.
• Enable --cpu-only for systems without compatible GPUs, forcing CPU inference paths.
• Omit the model name entirely to trigger hardware-aware ranking that selects the optimal model for your specific CPU, GPU, and memory configuration.
• Inspect scripts first with whichllm snippet to see exactly what code will execute before downloading multi-gigabyte model files.

Frequently Asked Questions

How does whichllm decide which GGUF quantization to use?

If you don’t specify --quant, the _pick_gguf_variant function selects a default quantization (typically Q4_K_M) that balances quality and file size. If you request a specific quantization that isn’t available, the system warns you and falls back to the closest available match. This logic resides in the GGUF resolution path of src/whichllm/cli.py (lines 110-152).

Can I run whichllm without installing model dependencies globally?

Yes, whichllm uses uv to create isolated execution environments for every model run. The CLI generates a temporary Python script and executes it via uv run --with <dependencies>, ensuring that packages like llama-cpp-python or transformers never pollute your global Python installation. This isolation happens automatically in the run command block (lines 106-118 of src/whichllm/cli.py).

What happens if I don’t specify a model name?

The tool enters hardware-aware selection mode. It calls detect_hardware() from src/whichllm/hardware/detector.py to analyze your system, then uses rank_models from src/whichllm/engine/ranker.py to score all known models against your capabilities. The highest-ranked model that fits within your VRAM and RAM constraints is automatically selected and executed.

How do I verify what code will run before downloading a model?

Use the whichllm snippet command. This invokes the same _generate_chat_script function used by run, but instead of executing the code, it prints the complete Python script and required dependencies to your terminal. You can review the exact download URLs, loading logic, and chat loop implementation before committing to multi-gigabyte downloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}