# KV-Cache Budgeting in Needle 2: How the Sliding-Window Size Is Computed

> Discover KV-cache budgeting in Needle 2. Learn how the sliding-window size is computed using raw memory and user-defined limits for efficient inference. Optimize your context length.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-08-24

---

**Needle 2 calculates the effective KV-cache window size by taking the minimum of a raw memory budget derived from `kv_bits` and an optional user-defined `kv_window` limit, ensuring inference respects both quantization constraints and context-length requirements.**

Needle 2 implements a **sliding-window KV cache** that is constrained by two independent budget dimensions stored in the model header. The runtime determines the actual number of tokens preserved in the cache through a deterministic budgeting algorithm defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).

## The Two Dimensions of KV-Cache Budgeting

The *.cact* model header in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 19-30) defines two fields that jointly control memory allocation:

- **`kv_window`**: The desired maximum number of tokens the cache can hold. This represents the sliding-window width the model was trained to use. When set to `0`, the runtime defers entirely to the memory budget calculated from `kv_bits`.
- **`kv_bits`**: The bit-width allocated to each KV entry (e.g., `8` for int8, `4` for int4). This determines the total memory available for the KV cache and therefore the maximum theoretical window size that fits within the allocated buffer.

These fields are written during model export and read at runtime to enforce memory constraints.

## Computing the Effective Window Size

The function `effective_kv_window()` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 614-616) implements the core budgeting logic:

```python
def effective_kv_window(config):
    # `budget` is the maximum window that fits within the KV-bits budget.

    # If the user supplied a non-zero kv_window, we cap it to that budget.

    return min(budget, config.kv_window) if config.kv_window else budget

```

This single line encapsulates Needle 2's KV-cache budgeting policy: the system always respects the memory constraint imposed by `kv_bits`, but optionally applies a stricter, user-defined context limit.

### Deriving the Raw Memory Budget

The `budget` variable represents the maximum number of tokens that can be stored given the chosen `kv_bits` value. This raw budget is computed from the model's dimensionality—specifically the number of attention heads and head size—and the total KV-cache memory allowed by the bit-width quantization scheme. The calculation ensures that the cache never exceeds the physical memory allocation determined by the quantization configuration.

### Applying the Hard Window Cap

When `config.kv_window` is non-zero, the runtime applies a hard cap by returning `min(budget, config.kv_window)`. This guarantees that even if the memory budget could theoretically support a larger window, the system never retains more tokens than the user-specified limit. If `kv_window` remains at its default value of `0`, the function returns the raw `budget` directly, allowing the quantization settings to exclusively determine the window size.

## Configuration and Runtime Usage

The `TransformerConfig` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 77-78) stores these budget parameters, which are then consumed by the decoding logic in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) to mask attention and enforce causal limits during inference.

### Practical Examples

The following examples demonstrate how different configurations interact to produce the effective window size:

```python
from needle.model.architecture import TransformerConfig, effective_kv_window

# Example 1: Let the KV-budget decide the window size (kv_window=0)

cfg = TransformerConfig(kv_bits=8, kv_window=0)
window = effective_kv_window(cfg)
print(window)   # → budget derived from kv_bits (e.g., 4096 tokens)

# Example 2: Explicit window smaller than the memory budget

cfg = TransformerConfig(kv_bits=8, kv_window=1024)
window = effective_kv_window(cfg)
print(window)   # → 1024 (the explicit limit is respected)

# Example 3: Explicit window exceeding the budget is capped

cfg = TransformerConfig(kv_bits=4, kv_window=8192)
window = effective_kv_window(cfg)
print(window)   # → budget (e.g., 4096) because 8k exceeds the int4 memory limit

```

These behaviors are validated in [`tests/test_build.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py), which exercises `effective_kv_window` with various quantization and window configurations to ensure the budgeting logic correctly handles edge cases where memory constraints override user preferences.

## Summary

- **KV-cache budgeting** in Needle 2 is governed by two independent parameters: `kv_bits` (memory allocation) and `kv_window` (context length).
- The effective window is computed in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) by taking the minimum of the memory-derived budget and the optional user limit.
- Setting `kv_window` to `0` disables the explicit cap, allowing the quantization bit-width to solely determine the maximum cache size.
- The logic ensures that inference never exceeds available KV-cache memory, even when the requested context window is larger than the quantized buffer allows.

## Frequently Asked Questions

### What is KV-cache budgeting in Needle 2?

KV-cache budgeting is the mechanism by which Needle 2 constrains the attention key/value cache using two dimensions: a memory budget defined by `kv_bits` (quantization width) and an optional token limit defined by `kv_window`. The system computes the actual window size as the tighter of these two constraints to prevent out-of-memory errors during inference while respecting user-specified context limits.

### How does `kv_bits` affect the window size?

The `kv_bits` parameter determines how many bits are used to store each KV entry. A lower bit-width (e.g., int4) reduces memory consumption per token but also reduces the total number of tokens that fit within the fixed-size KV buffer. The raw budget is calculated from this bit-width and the model's head dimensions, establishing the theoretical maximum window size supported by the hardware allocation.

### What happens if `kv_window` is set to 0?

When `kv_window` is `0`, the `effective_kv_window` function ignores the explicit window parameter and returns only the raw budget derived from `kv_bits`. This configuration allows the quantization settings to exclusively determine the sliding-window size, maximizing context length within the available memory budget without imposing an artificial token limit.

### Where is the effective window size calculated in the source code?

The effective window size is calculated by the `effective_kv_window()` function in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 614-616). This function is called during model initialization and decoding to set the actual cache bounds used by the attention mechanism in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py).