Bytes per Weight Values for Quantization Types in WhichLLM: Complete Reference Guide

WhichLLM defines a comprehensive mapping of quantization formats to their exact bytes-per-weight ratios in the QUANT_BYTES_PER_WEIGHT constant, enabling precise calculation of model storage requirements and VRAM usage.

WhichLLM is an open-source recommendation engine that matches language models to hardware constraints. At the core of its size estimation logic lies a precise lookup table that converts quantization format names into their actual storage costs per parameter, as implemented in Andyyyy64/whichllm.

Where WhichLLM Defines Bytes per Weight Values

The authoritative source for these metrics resides in src/whichllm/data/quantization.py. Specifically, lines 4-44 define the constant QUANT_BYTES_PER_WEIGHT as a dictionary mapping format identifiers to floating-point storage values.

These figures derive from underlying bit-widths (e.g., 4 bits equals 0.5 bytes) and include overhead for per-block scaling factors where applicable, such as in MXFP4 and NVFP4 formats.

Complete Bytes per Weight Reference Table

The following table lists every quantization type supported by WhichLLM and its corresponding bytes-per-weight value:

Quantization	Bytes/weight
F32	4.0
F16	2.0
BF16	2.0
Q8_0	1.0625
Q6_K	0.8125
Q5_K_M	0.6875
Q5_K_S	0.6875
Q5_0	0.625
Q4_K_M	0.5625
Q4_K_S	0.5625
Q4_0	0.5
Q3_K_M	0.4375
Q3_K_S	0.4375
Q3_K_L	0.4375
Q2_K	0.3125
IQ4_XS	0.5
IQ3_XXS	0.375
IQ2_XXS	0.25
MXFP4	0.53125
NVFP4	0.5625
Q1_0	0.28
Q2_0	0.28
TQ1_0	0.21
TQ2_0	0.28
IQ1_S	0.21
IQ1_M	0.22
IQ2_S	0.275
IQ2_M	0.30
IQ3_S	0.40
IQ3_M	0.42
IQ3_XS	0.41
IQ4_NL	0.5

How WhichLLM Uses Bytes per Weight Data

The quantization constants power the engine's size estimation and ranking systems across three critical modules:

• Weight Size Estimation: The estimate_weight_bytes function in src/whichllm/engine/quantization.py multiplies model parameter counts by these bytes-per-weight values to predict total storage requirements.
• VRAM Calculation: src/whichllm/engine/vram.py integrates these metrics to forecast GPU memory requirements, accounting for the actual compressed size of model weights.
• Model Ranking: src/whichllm/engine/ranker.py leverages these storage costs to balance the trade-off between quantization compression and output quality when recommending models.

Practical Code Examples

Looking Up Specific Quantization Values

from whichllm.data.quantization import QUANT_BYTES_PER_WEIGHT

# Retrieve bytes per weight for Q4_K_M

bytes_per_weight = QUANT_BYTES_PER_WEIGHT["Q4_K_M"]
print(f"Q4_K_M uses {bytes_per_weight:.4f} bytes per weight")

# Output: Q4_K_M uses 0.5625 bytes per weight

Estimating Total Model Size

from whichllm.models.types import ModelInfo
from whichllm.engine.quantization import estimate_weight_bytes

# Create a model with 10 billion parameters

model = ModelInfo(id="example-llm", parameter_count=10_000_000_000)

# Calculate total weight size in bytes

estimated_bytes = estimate_weight_bytes(model, variant=None)
size_gb = estimated_bytes / (1024**3)

print(f"Estimated weight size: {size_gb:.2f} GB")

Custom VRAM Calculator

def required_vram_gb(model, variant):
    """
    Calculate required VRAM including 10% overhead for activations.
    """
    weight_bytes = estimate_weight_bytes(model, variant)
    # Add 10% headroom for inference overhead

    total_bytes = weight_bytes * 1.10
    return total_bytes / (1024**3)  # Convert to GB

Summary

• WhichLLM defines bytes-per-weight values in src/whichllm/data/quantization.py within the QUANT_BYTES_PER_WEIGHT constant, covering formats from F32 (4.0 bytes) to TQ1_0 (0.21 bytes).
• The engine uses these metrics in estimate_weight_bytes to calculate model storage requirements and predict VRAM usage.
• Fractional values account for compression overhead and per-block scaling factors in advanced formats like MXFP4 and NVFP4.
• Integration across modules ensures consistent size estimation from the data layer through the ranking engine.

Frequently Asked Questions

What is the bytes per weight value for Q4_K_M quantization in WhichLLM?

According to the source code in src/whichllm/data/quantization.py, the Q4_K_M quantization format uses 0.5625 bytes per weight. This represents 4.5 bits per weight on average, accounting for the mixed-bit compression strategy used by this format.

How does WhichLLM calculate total model size from bytes per weight?

The estimate_weight_bytes function in src/whichllm/engine/quantization.py multiplies the model's parameter count by the corresponding bytes-per-weight value from QUANT_BYTES_PER_WEIGHT. This yields the total compressed storage size required for the model weights, excluding overhead for activations or KV cache.

Why do some quantization formats use fractional bytes per weight?

Fractional values occur because quantization compresses weights below the byte boundary. For example, 4-bit quantization yields 0.5 bytes per weight, while formats like MXFP4 (0.53125 bytes) and NVFP4 (0.5625 bytes) include additional overhead for per-block scaling factors that slightly increase the average storage cost beyond the raw bit-width conversion.

Where can I find the complete list of quantization types supported by WhichLLM?

The complete dictionary mapping quantization names to bytes-per-weight values is defined in src/whichllm/data/quantization.py (lines 4-44). This file contains the QUANT_BYTES_PER_WEIGHT constant that serves as the single source of truth for all quantization metrics used throughout the WhichLLM codebase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}