How to Configure KV Cache Quantization for VRAM Savings in Forge

Use the --cache-type-k and --cache-type-v CLI flags when initializing a Forge server to quantize the key-value cache to 8-bit (q8_0) or 4-bit (q4_0), reducing VRAM usage by 4–8× and preventing costly CPU offloading.

Forge orchestrates inference backends—including llama-server, llamafile, and Ollama—that store attention state in a KV cache directly on the GPU. According to docs/ARCHITECTURE.md (line 65), this cache competes with model weights for limited VRAM, often forcing larger quantized models to fall back to system RAM and suffer 5–20× performance penalties. Configuring KV cache quantization minimizes this footprint without requiring changes to the model weights themselves.

Understanding KV Cache Memory Pressure

The KV cache maintains two floating-point tensors per token—keys and values—dimensioned by the model's hidden size and head count. By default, these are stored as 32-bit floats (f32), consuming substantial VRAM proportional to context length and batch size.

On consumer hardware with ≤12GB VRAM, running a 14B parameter model at Q4 precision can exhaust available memory, leaving insufficient room for the cache. When VRAM is exhausted, the backend offloads the cache to system RAM, causing severe token-generation slowdowns. Quantizing the cache itself—rather than just the weights—provides the only mechanism to retain GPU-resident inference under these constraints.

Configuring KV Cache Quantization Parameters

Forge exposes quantization controls through the ServerManager class, specifically within the start method implementation in src/forge/server.py.

CLI Flags for Keys and Values

Two optional string parameters control the quantization scheme:

• --cache-type-k – Defines precision for attention keys (e.g., q8_0, q4_0).
• --cache-type-v – Defines precision for attention values.

These flags accept GGUF quantization type identifiers. Valid options include f32 (default), q8_0 (8-bit), and q4_0 (4-bit).

Implementation in ServerManager

The ServerManager.start method signature (lines 79–84) accepts cache_type_k and cache_type_v as optional arguments. When non-null, these values are injected into the backend launch command at lines 160–162 of src/forge/server.py. Forge passes these flags directly to the underlying process without modification, allowing the backend to handle the quantized cache allocation.

Practical Implementation Examples

Configure an 8-bit KV cache when starting a server via Python:

from forge.server import ServerManager

sm = ServerManager(backend="llamafile", port=8080, models_dir="/models")
await sm.start(
    model="ministral-13b",
    gguf_path="/models/ministral-13b.gguf",
    mode="native",
    cache_type_k="q8_0",   # 8-bit quantization for keys

    cache_type_v="q8_0",   # 8-bit quantization for values

    n_slots=2,             # two concurrent agents share the cache

    kv_unified=False,
)

The equivalent configuration using the llama-server CLI directly:

llama-server \
  -m /models/ministral-13b.gguf \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --parallel 2

For maximum VRAM savings on severely constrained hardware, use 4-bit quantization:

await sm.start(
    model="ministral-13b",
    gguf_path="/models/ministral-13b.gguf",
    cache_type_k="q4_0",
    cache_type_v="q4_0",
)

Performance Implications and Trade-offs

VRAM Savings and Multi-Slot Deployments

Quantizing from f32 to q8_0 reduces cache memory consumption by 75%, while q4_0 achieves approximately 87.5% reduction. This compression is multiplicative across concurrent slots—each slot instantiates independent key and value caches—enabling higher parallelism on single-GPU workstations.

Accuracy Considerations

The backend performs on-the-fly de-quantization during attention computations. While extreme quantization (q4_0) theoretically degrades numerical precision, production workloads typically exhibit negligible quality degradation. Keys and values are less sensitive to precision loss than weight matrices, making cache quantization safer than aggressive weight quantization.

Backend Compatibility

KV cache quantization requires backend binaries built with GGUF support. Legacy llama-server versions predating April 2026 lack support for --cache-type-k and --cache-type-v, silently ignoring the flags. Verify your llamafile or llama-server build includes the quantization schemas before deployment.

Summary

• KV cache quantization uses --cache-type-k and --cache-type-v flags to reduce VRAM pressure.
• Configuration is handled via ServerManager.start in src/forge/server.py (lines 79–84 and 160–162).
• Memory reduction scales by 4–8× when using q8_0 or q4_0 precision instead of f32.
• Use cases include limited VRAM hardware (≤12GB), multi-slot concurrent deployments, and latency-sensitive workloads.
• Compatibility requires modern backend binaries with GGUF quantization support.

Frequently Asked Questions

How much VRAM does KV cache quantization actually save?

Quantizing from the default 32-bit floating point (f32) to 8-bit integers (q8_0) reduces the KV cache footprint by approximately 75%, while 4-bit (q4_0) quantization achieves roughly 87.5% memory reduction. For a 14B model with 4096 context length, this can free several gigabytes of VRAM, enabling the model to remain GPU-resident rather than falling back to system RAM.

Will quantizing the KV cache reduce model accuracy?

In most production scenarios, no. The backend performs on-the-fly de-quantization during attention operations, and empirical testing shows negligible quality degradation for typical prompt workloads. While aggressive q4_0 quantization can theoretically affect attention precision, the impact is substantially lower than equivalent weight quantization because cached activations are less sensitive to precision loss than model parameters.

Which Forge backends support KV cache quantization?

Support depends entirely on the underlying binary. Modern builds of llama-server, llamafile, and Ollama compiled with GGUF support recognize the --cache-type-k and --cache-type-v flags. Versions predating April 2026 will ignore these parameters silently, falling back to f32 precision without raising errors. Always verify your backend version against the quantization schema requirements.

Can I use different quantization levels for keys and values?

Yes. Forge allows asymmetric configuration—for example, cache_type_k="q8_0" combined with cache_type_v="q4_0"—to balance memory constraints against specific attention requirements. However, using matching precision for both components is generally recommended to ensure predictable latency and avoid potential numerical instability in attention head computations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}