How to Specify Quantization Type in whichllm: A Complete Guide to the `--quant` Flag

Use the --quant or -q option with any whichllm command to filter rankings, plan GPU requirements, run models, or generate Python snippets using a specific quantization format such as Q4_K_M, INT8, or GPTQ.

The whichllm CLI helps you discover and run large language models matched to your hardware. When you need to target a specific quantization scheme, you can specify quantization type in whichllm commands via a unified flag exposed across all model-selection subcommands. As implemented in Andyyyy64/whichllm, this option is defined centrally in src/whichllm/cli.py and forwarded to the ranking engine and variant resolver.

Supported Commands That Accept --quant

The --quant / -q option is available on every command that deals with model selection. Typer parses the argument into quant: Optional[str] and passes it through the call chain accordingly.

• whichllm (ranking) — Defined in src/whichllm/cli.py for the main command. The flag filters ranking results to only GGUF variants whose quant_type matches the supplied string. The comparison is case-insensitive.

• whichllm plan — Defined in src/whichllm/cli.py for the plan subcommand. Shows the GPU resources required for a model if it were available in the requested quantization.

• whichllm run — Defined in src/whichllm/cli.py for the run subcommand. Chooses a GGUF variant that exactly matches the supplied quant type; if none exists, the command falls back to the best available variant and prints a warning.

• whichllm snippet — Defined in src/whichllm/cli.py for the snippet subcommand. Generates a ready-to-run Python snippet that loads the model using the requested quantization when available.

How the --quant Filter Works Internally

Parsing and forwarding in cli.py

Typer captures the user-supplied value and stores it in the quant variable. Every command forwards this value to the core engine, typically as quant_filter in calls to rank_models.

In src/whichllm/cli.py, the ranking invocation looks similar to:

rank_models(..., quant_filter=quant)

The parameter is then applied during iteration over candidate GGUF variants.

Case-insensitive matching in the ranker

Inside the ranking logic in src/whichllm/engine/ranker.py, the filter is applied when evaluating each variant:

if quant_filter:
    if v.quant_type.upper() == quant_filter.upper():
        return v

Because the comparison uses .upper(), inputs like q4_k_m, Q4_K_M, and Q4 are all treated equivalently.

Fallback variant selection

When you request a quant type that does not exist for a given model, _pick_gguf_variant in src/whichllm/cli.py handles resolution. If no exact match is found, it emits a yellow warning and selects the next-best variant according to the global preference order stored in whichllm.constants.QUANT_PREFERENCE_ORDER.

This ensures the tool remains usable even when a specific quantization build is missing from the model repository.

Non-GGUF quantization inference

Not all models provide GGUF builds. For these cases, whichllm infers the quantization from the repository name using infer_non_gguf_quant_type in src/whichllm/engine/quantization.py. An awq-tagged repository is treated as AWQ, allowing the --quant filter to work on vanilla transformer models as well.

After a variant is chosen, _resolve_model_deps determines the required Python packages—llama-cpp-python for GGUF, or transformers plus torch and optional autoawq/auto-gptq for non-GGUF formats. The generated script from _generate_chat_script embeds the quant type into the download URL and loader initialization.

Practical Examples

Rank only Q4_K_M models

whichllm --quant Q4_K_M --top 5

This restricts the ranking results to GGUF variants whose quant_type equals Q4_K_M.

Estimate GPU usage for an INT8 variant

whichllm plan "StarCoder" --quant INT8

The command displays the estimated VRAM needed for the INT8 variant. If that variant is unavailable, the planner falls back to the nearest alternative and surfaces a warning.

Run a model with a specific quant type

whichllm run "Mistral-7B-Instruct" --quant Q5_K_M

If a Q5_K_M GGUF exists, the model downloads and runs with that quantization. If not, you will see a fallback warning and the tool will run the best available alternative instead.

Generate a snippet for a GPTQ model

whichllm snippet "Llama-2-7b-chat" --quant GPTQ

The generated Python snippet imports auto-gptq and loads the model with the requested GPTQ quantization baked into the construction code.

Summary

• The --quant / -q flag in whichllm gives you fine-grained control over model quantization across rankings, planning, execution, and snippet generation.
• The option is defined centrally in src/whichllm/cli.py and parsed by Typer as quant: Optional[str].
• Filtering is case-insensitive inside src/whichllm/engine/ranker.py via v.quant_type.upper() == quant_filter.upper().
• If an exact quant is unavailable, _pick_gguf_variant falls back using whichllm.constants.QUANT_PREFERENCE_ORDER.
• Non-GGUF models are supported through infer_non_gguf_quant_type in src/whichllm/engine/quantization.py.

Frequently Asked Questions

What is the short form of the --quant flag?

The short form is -q. You can use it interchangeably with --quant on any command that accepts the option, including whichllm, plan, run, and snippet.

Does --quant work if a model only has non-GGUF formats?

Yes. For models without GGUF builds, whichllm uses infer_non_gguf_quant_type in src/whichllm/engine/quantization.py to derive the quantization type from the repository name. This allows the --quant filter to match AWQ, GPTQ, and other tagged formats.

What happens if I request a quantization that does not exist?

If the exact quant type is missing, _pick_gguf_variant in src/whichllm/cli.py prints a yellow warning and automatically selects the next-best variant based on QUANT_PREFERENCE_ORDER. The command will still execute rather than failing.

Is the quantization comparison case-sensitive?

No. The comparison is case-insensitive. You can pass q4_k_m, Q4_K_M, or Q4, and the filter in src/whichllm/engine/ranker.py will evaluate them identically using .upper().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}