# How to Inspect the Effective KV Window Size in Needle 2

> Inspect the effective KV window size in Needle 2 by calling effective_kv_window(cfg) in architecture.py. Learn how this value is determined by user config or memory budget.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-29

---

**Call `effective_kv_window(cfg)` from [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) to retrieve the runtime KV window size, which represents the actual cache limit determined by either the user configuration or the memory budget calculation.**

The Needle 2 inference engine from the `cactus-compute/needle` repository implements a budget-aware sliding-window algorithm to manage key-value (KV) cache memory during transformer inference. While you can request a specific window size via `TransformerConfig`, the system dynamically adjusts this value based on hardware constraints. Understanding how to inspect the effective KV window size in Needle 2 is essential for verifying that your model operates within the intended memory boundaries.

## Understanding the KV Window Calculation

Needle 2 determines the effective KV window size by comparing two values: the explicit window requested by the user and the maximum window allowed by the current memory budget. The final runtime value is the more conservative of the two.

According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the budget calculation occurs in `kv_budget_window` (lines 1003–1011), which computes the theoretical maximum based on available KV cache memory. The function `effective_kv_window` (lines 1014–1016) then returns the effective size by selecting the minimum between the user-specified `kv_window` and the budget-limited value. If the user sets `kv_window` to `0`, the system interprets this as "no explicit limit" and defaults entirely to the budget calculation.

## Methods to Retrieve the Effective KV Window

You can inspect the KV window size at runtime using three distinct approaches, depending on whether you need the final computed value, the raw configuration setting, or the memory budget ceiling.

### Query the Effective Window Directly

The `effective_kv_window()` function provides the definitive window size that Needle 2 will use during forward passes. This method accounts for both user preferences and hardware constraints.

```python
from needle.model.architecture import TransformerConfig, effective_kv_window

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    kv_window=1024,  # User request

)

print(effective_kv_window(cfg))  # Returns the actual window size (e.g., 1024 or budget limit)

```

### Read the Raw Configuration Value

To see the explicit window setting without budget adjustments, access the `kv_window` attribute directly on your `TransformerConfig` instance. A value of `0` indicates that no explicit limit has been set, allowing the budget calculation to dictate the window size.

```python
print(cfg.kv_window)  # Returns 1024 (user request) or 0 (unlimited)

```

### Inspect the Memory Budget Limit

To understand the upper boundary imposed by memory constraints, call `kv_budget_window()`. This reveals the maximum window size supported by the current KV cache budget before considering the user configuration.

```python
from needle.model.architecture import kv_budget_window

print(kv_budget_window(cfg))  # Returns budget-limited maximum (e.g., 1536)

```

## Practical Code Examples

The following example demonstrates how to compare all three values to understand which constraint is currently active:

```python
from needle.model.architecture import TransformerConfig, effective_kv_window, kv_budget_window

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    max_seq_len=4096,
    kv_window=1024,
)

print(f"User-requested kv_window: {cfg.kv_window}")
print(f"Budget-limited kv_window: {kv_budget_window(cfg)}")
print(f"Effective KV window: {effective_kv_window(cfg)}")

```

Example output:

```bash
User-requested kv_window: 1024
Budget-limited kv_window: 1536
Effective KV window: 1024

```

When the user-requested window exceeds the budget, or when `kv_window` is set to `0`, the effective size falls back to the budget calculation:

```python
cfg.kv_window = 0
print(effective_kv_window(cfg))  # Returns budget-limited value (e.g., 1536)

```

## Inspecting Window Size During Model Execution

You can also retrieve the effective window from a running model instance. The `SimpleAttentionNetwork` class stores the configuration internally, allowing you to verify the window size after model initialization.

```python
from needle.model.simple_attention_network import SimpleAttentionNetwork

model = SimpleAttentionNetwork(config=cfg)
effective_size = effective_kv_window(model.config)
print(f"Effective KV window inside model: {effective_size}")

```

This approach is useful when loading pretrained configurations where the window settings might differ from your explicit instantiation.

## Summary

- **`effective_kv_window(cfg)`** returns the final runtime window size used by Needle 2, implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at lines 1014–1016.
- **`cfg.kv_window`** provides the raw user configuration, where `0` signifies no explicit limit.
- **`kv_budget_window(cfg)`** calculates the memory-imposed ceiling at lines 1003–1011 in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- When `kv_window` is `0` or exceeds the budget, the effective window defaults to the budget-limited value.
- Model instances like `SimpleAttentionNetwork` expose the configuration via `.config`, allowing runtime inspection.

## Frequently Asked Questions

### What is the difference between `kv_window` and `effective_kv_window`?

The `kv_window` attribute on `TransformerConfig` stores the user's explicit request or `0` for unlimited, while `effective_kv_window()` returns the actual window size that Needle 2 enforces during inference. The effective value represents the intersection of user intent and hardware constraints.

### What happens when `kv_window` is set to 0?

Setting `kv_window=0` removes the explicit user constraint, causing `effective_kv_window()` to return the value computed by `kv_budget_window()`. This allows Needle 2 to utilize the maximum cache size permitted by the current memory budget without artificial limitations.

### Where is the KV budget calculation implemented?

The memory budget logic resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) within the `kv_budget_window` function, specifically spanning lines 1003–1011. This function computes the theoretical maximum KV window based on available cache memory and model architecture parameters.

### How does the effective window affect model inference?

The effective KV window determines how many previous tokens remain in the key-value cache during autoregressive generation. A smaller window reduces memory usage but may degrade performance on long-context tasks, while the budget-aware calculation in Needle 2 prevents out-of-memory errors by automatically constraining the cache size to available resources.