# How to Specify Quantization Type in whichllm: A Complete Guide to the `--quant` Flag

> Learn to specify quantization type in whichllm using the --quant flag. Filter, plan, run models, and generate Python snippets with formats like Q4_K_M and INT8.

- Repository: [andy/whichllm](https://github.com/Andyyyy64/whichllm)
- Tags: how-to-guide
- Published: 2026-06-09

---

**Use the `--quant` or `-q` option with any whichllm command to filter rankings, plan GPU requirements, run models, or generate Python snippets using a specific quantization format such as `Q4_K_M`, `INT8`, or `GPTQ`.**

The `whichllm` CLI helps you discover and run large language models matched to your hardware. When you need to target a specific **quantization scheme**, you can specify quantization type in whichllm commands via a unified flag exposed across all **model-selection subcommands**. As implemented in `Andyyyy64/whichllm`, this option is defined centrally in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) and forwarded to the ranking engine and variant resolver.

## Supported Commands That Accept `--quant`

The **`--quant`** / **`-q`** option is available on every command that deals with model selection. **Typer** parses the argument into `quant: Optional[str]` and passes it through the call chain accordingly.

- **`whichllm` (ranking)** — Defined in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) for the main command. The flag filters ranking results to only **GGUF** variants whose `quant_type` matches the supplied string. The comparison is case-insensitive.

- **`whichllm plan`** — Defined in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) for the `plan` subcommand. Shows the GPU resources required for a model if it were available in the requested quantization.

- **`whichllm run`** — Defined in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) for the `run` subcommand. Chooses a GGUF variant that exactly matches the supplied quant type; if none exists, the command falls back to the best available variant and prints a warning.

- **`whichllm snippet`** — Defined in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) for the `snippet` subcommand. Generates a ready-to-run Python snippet that loads the model using the requested quantization when available.

## How the `--quant` Filter Works Internally

### Parsing and forwarding in cli.py

Typer captures the user-supplied value and stores it in the `quant` variable. Every command forwards this value to the core engine, typically as `quant_filter` in calls to `rank_models`.

In [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py), the ranking invocation looks similar to:

```python
rank_models(..., quant_filter=quant)

```

The parameter is then applied during iteration over candidate GGUF variants.

### Case-insensitive matching in the ranker

Inside the ranking logic in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py), the filter is applied when evaluating each variant:

```python
if quant_filter:
    if v.quant_type.upper() == quant_filter.upper():
        return v

```

Because the comparison uses `.upper()`, inputs like `q4_k_m`, `Q4_K_M`, and `Q4` are all treated equivalently.

### Fallback variant selection

When you request a quant type that does not exist for a given model, `_pick_gguf_variant` in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) handles resolution. If no exact match is found, it emits a yellow warning and selects the next-best variant according to the global preference order stored in `whichllm.constants.QUANT_PREFERENCE_ORDER`.

This ensures the tool remains usable even when a specific quantization build is missing from the model repository.

### Non-GGUF quantization inference

Not all models provide GGUF builds. For these cases, `whichllm` infers the quantization from the repository name using `infer_non_gguf_quant_type` in [`src/whichllm/engine/quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/quantization.py). An `awq`-tagged repository is treated as `AWQ`, allowing the `--quant` filter to work on vanilla transformer models as well.

After a variant is chosen, `_resolve_model_deps` determines the required Python packages—`llama-cpp-python` for GGUF, or `transformers` plus `torch` and optional `autoawq`/`auto-gptq` for **non-GGUF** formats. The generated script from `_generate_chat_script` embeds the quant type into the download URL and loader initialization.

## Practical Examples

### Rank only Q4_K_M models

```bash
whichllm --quant Q4_K_M --top 5

```

This restricts the ranking results to GGUF variants whose `quant_type` equals `Q4_K_M`.

### Estimate GPU usage for an INT8 variant

```bash
whichllm plan "StarCoder" --quant INT8

```

The command displays the estimated VRAM needed for the `INT8` variant. If that variant is unavailable, the planner falls back to the nearest alternative and surfaces a warning.

### Run a model with a specific quant type

```bash
whichllm run "Mistral-7B-Instruct" --quant Q5_K_M

```

If a `Q5_K_M` GGUF exists, the model downloads and runs with that quantization. If not, you will see a **fallback** warning and the tool will run the best available alternative instead.

### Generate a snippet for a GPTQ model

```bash
whichllm snippet "Llama-2-7b-chat" --quant GPTQ

```

The generated Python snippet imports `auto-gptq` and loads the model with the requested GPTQ quantization baked into the construction code.

## Summary

- The `--quant` / `-q` flag in `whichllm` gives you fine-grained control over model quantization across rankings, planning, execution, and snippet generation.
- The option is defined centrally in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) and parsed by Typer as `quant: Optional[str]`.
- Filtering is case-insensitive inside [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) via `v.quant_type.upper() == quant_filter.upper()`.
- If an exact quant is unavailable, `_pick_gguf_variant` falls back using `whichllm.constants.QUANT_PREFERENCE_ORDER`.
- Non-GGUF models are supported through `infer_non_gguf_quant_type` in [`src/whichllm/engine/quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/quantization.py).

## Frequently Asked Questions

### What is the short form of the `--quant` flag?

The short form is `-q`. You can use it interchangeably with `--quant` on any command that accepts the option, including `whichllm`, `plan`, `run`, and `snippet`.

### Does `--quant` work if a model only has non-GGUF formats?

Yes. For models without GGUF builds, `whichllm` uses `infer_non_gguf_quant_type` in [`src/whichllm/engine/quantization.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/quantization.py) to derive the quantization type from the repository name. This allows the `--quant` filter to match `AWQ`, `GPTQ`, and other tagged formats.

### What happens if I request a quantization that does not exist?

If the exact quant type is missing, `_pick_gguf_variant` in [`src/whichllm/cli.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/cli.py) prints a yellow warning and automatically selects the next-best variant based on `QUANT_PREFERENCE_ORDER`. The command will still execute rather than failing.

### Is the quantization comparison case-sensitive?

No. The comparison is case-insensitive. You can pass `q4_k_m`, `Q4_K_M`, or `Q4`, and the filter in [`src/whichllm/engine/ranker.py`](https://github.com/Andyyyy64/whichllm/blob/main/src/whichllm/engine/ranker.py) will evaluate them identically using `.upper()`.