KV-Cache Budgeting in Needle 2: How the Sliding-Window Size Is Computed

Needle 2 calculates the effective KV-cache window size by taking the minimum of a raw memory budget derived from kv_bits and an optional user-defined kv_window limit, ensuring inference respects both quantization constraints and context-length requirements.

Needle 2 implements a sliding-window KV cache that is constrained by two independent budget dimensions stored in the model header. The runtime determines the actual number of tokens preserved in the cache through a deterministic budgeting algorithm defined in needle/model/architecture.py.

The Two Dimensions of KV-Cache Budgeting

The .cact model header in needle/model/export.py (lines 19-30) defines two fields that jointly control memory allocation:

• kv_window: The desired maximum number of tokens the cache can hold. This represents the sliding-window width the model was trained to use. When set to 0, the runtime defers entirely to the memory budget calculated from kv_bits.
• kv_bits: The bit-width allocated to each KV entry (e.g., 8 for int8, 4 for int4). This determines the total memory available for the KV cache and therefore the maximum theoretical window size that fits within the allocated buffer.

These fields are written during model export and read at runtime to enforce memory constraints.

Computing the Effective Window Size

The function effective_kv_window() in needle/model/architecture.py (lines 614-616) implements the core budgeting logic:

def effective_kv_window(config):
    # `budget` is the maximum window that fits within the KV-bits budget.

    # If the user supplied a non-zero kv_window, we cap it to that budget.

    return min(budget, config.kv_window) if config.kv_window else budget

This single line encapsulates Needle 2's KV-cache budgeting policy: the system always respects the memory constraint imposed by kv_bits, but optionally applies a stricter, user-defined context limit.

Deriving the Raw Memory Budget

The budget variable represents the maximum number of tokens that can be stored given the chosen kv_bits value. This raw budget is computed from the model's dimensionality—specifically the number of attention heads and head size—and the total KV-cache memory allowed by the bit-width quantization scheme. The calculation ensures that the cache never exceeds the physical memory allocation determined by the quantization configuration.

Applying the Hard Window Cap

When config.kv_window is non-zero, the runtime applies a hard cap by returning min(budget, config.kv_window). This guarantees that even if the memory budget could theoretically support a larger window, the system never retains more tokens than the user-specified limit. If kv_window remains at its default value of 0, the function returns the raw budget directly, allowing the quantization settings to exclusively determine the window size.

Configuration and Runtime Usage

The TransformerConfig class in needle/model/architecture.py (lines 77-78) stores these budget parameters, which are then consumed by the decoding logic in needle/model/decode.py to mask attention and enforce causal limits during inference.

Practical Examples

The following examples demonstrate how different configurations interact to produce the effective window size:

from needle.model.architecture import TransformerConfig, effective_kv_window

# Example 1: Let the KV-budget decide the window size (kv_window=0)

cfg = TransformerConfig(kv_bits=8, kv_window=0)
window = effective_kv_window(cfg)
print(window)   # → budget derived from kv_bits (e.g., 4096 tokens)

# Example 2: Explicit window smaller than the memory budget

cfg = TransformerConfig(kv_bits=8, kv_window=1024)
window = effective_kv_window(cfg)
print(window)   # → 1024 (the explicit limit is respected)

# Example 3: Explicit window exceeding the budget is capped

cfg = TransformerConfig(kv_bits=4, kv_window=8192)
window = effective_kv_window(cfg)
print(window)   # → budget (e.g., 4096) because 8k exceeds the int4 memory limit

These behaviors are validated in tests/test_build.py, which exercises effective_kv_window with various quantization and window configurations to ensure the budgeting logic correctly handles edge cases where memory constraints override user preferences.

Summary

• KV-cache budgeting in Needle 2 is governed by two independent parameters: kv_bits (memory allocation) and kv_window (context length).
• The effective window is computed in needle/model/architecture.py by taking the minimum of the memory-derived budget and the optional user limit.
• Setting kv_window to 0 disables the explicit cap, allowing the quantization bit-width to solely determine the maximum cache size.
• The logic ensures that inference never exceeds available KV-cache memory, even when the requested context window is larger than the quantized buffer allows.

Frequently Asked Questions

What is KV-cache budgeting in Needle 2?

KV-cache budgeting is the mechanism by which Needle 2 constrains the attention key/value cache using two dimensions: a memory budget defined by kv_bits (quantization width) and an optional token limit defined by kv_window. The system computes the actual window size as the tighter of these two constraints to prevent out-of-memory errors during inference while respecting user-specified context limits.

How does kv_bits affect the window size?

The kv_bits parameter determines how many bits are used to store each KV entry. A lower bit-width (e.g., int4) reduces memory consumption per token but also reduces the total number of tokens that fit within the fixed-size KV buffer. The raw budget is calculated from this bit-width and the model's head dimensions, establishing the theoretical maximum window size supported by the hardware allocation.

What happens if kv_window is set to 0?

When kv_window is 0, the effective_kv_window function ignores the explicit window parameter and returns only the raw budget derived from kv_bits. This configuration allows the quantization settings to exclusively determine the sliding-window size, maximizing context length within the available memory budget without imposing an artificial token limit.

Where is the effective window size calculated in the source code?

The effective window size is calculated by the effective_kv_window() function in needle/model/architecture.py (lines 614-616). This function is called during model initialization and decoding to set the actual cache bounds used by the attention mechanism in needle/model/decode.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}