# How Needle 2's Memory Management Differs from Traditional Transformers

> Discover how Needle 2 optimizes memory management beyond traditional transformers. Learn about its hybrid system featuring a sliding-window KV cache and persistent Engram table for efficient, budget-constrained operation.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-11

---

**Needle 2 replaces the exponential memory growth of traditional transformers with a deterministic, budget-constrained hybrid system that combines a sliding-window KV cache and a persistent Engram table.**

Needle 2, developed in the `cactus-compute/needle` repository, implements a Simple Attention Network that fundamentally restructures how transformer architectures handle memory. While conventional models store key-value pairs for every token in the sequence—leading to linear memory growth and quadratic attention costs—Needle 2 caps memory usage at approximately 28 MiB through specialized caching mechanisms and fixed-size embedding tables. This technical analysis explores the specific source code implementations that enable Needle 2's constant-memory scaling.

## The Sliding-Window KV Cache vs. Full-Context Storage

Traditional transformers maintain a **full KV cache** that stores key and value tensors for every token in the sequence. This approach consumes memory proportional to $O(L \times d_{\text{model}})$, where $L$ represents sequence length, and incurs quadratic attention computation costs during generation.

Needle 2 eliminates this bottleneck through a **budget-constrained sliding window**. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the `effective_kv_window()` function (lines 614-618) calculates the maximum number of recent tokens to retain based on a fixed byte budget rather than sequence length. The implementation restricts the KV cache to a few hundred tokens—typically around 400 tokens—regardless of how long the conversation grows.

```python
from needle.model.architecture import effective_kv_window
from needle.model.run import NeedleModel

# Load model with default 28 MiB memory budget

model = NeedleModel.from_pretrained("cactus-needle-2")

# Query the deterministic KV window size

kv_win = effective_kv_window(model.config)
print(f"Effective KV window: {kv_win} tokens")  # Output: ~400 tokens

```

This sliding window ensures that RAM usage remains constant during inference, contrasting sharply with traditional transformers where memory requirements grow indefinitely with context length.

## The Engram Module: Persistent Long-Range Memory

While sliding windows limit memory, they risk losing distant context. Needle 2 solves this through the **Engram** module, a fixed-size embedding table that provides persistent long-range storage without the memory overhead of full KV caching.

Located in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 81-96), the Engram module hashes token sequences into a fixed-size table and retrieves them on-the-fly during attention computation. The `engram_indices()` function (lines 46-58) generates hash-based indices that can be reused across layers, enabling the model to reference distant tokens without storing their full key-value representations.

**Key characteristics:**

- **Fixed-size storage**: The Engram table maintains constant capacity regardless of sequence length.
- **Cross-layer reuse**: Engram indices generated once can be shared across multiple transformer layers.
- **Low bandwidth**: Retrieval operates on compressed hash representations rather than full-dimensional KV tensors.

This architecture allows Needle 2 to capture long-range dependencies that would normally require the full KV cache in traditional transformers, but with bandwidth comparable to retrieval-augmented models.

## Deterministic Memory Budget Calculation

Unlike traditional transformers where memory limits depend on available GPU/CPU RAM, Needle 2 implements explicit memory budgeting. The `kv_budget_window` formula in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 1004-1011) calculates the exact memory footprint based on:

- Model width (hidden dimensions)
- Number of transformer layers
- KV-grouping configuration (GQA/MQA settings)
- Allocated Engram slots

This calculation guarantees that the model fits within approximately **28 MiB of RAM** for a complete inference session—a stark contrast to traditional models where memory scales linearly with sequence length.

```python

# The budget automatically accounts for all memory consumers:

# - Sliding-window KV cache

# - Engram table slots

# - Quantization overhead

# Result: deterministic ~28 MiB usage regardless of prompt length

output = model.generate("Extremely long prompt..." * 1000)  # Still ~28 MiB

```

## Per-Token Quantization Strategy

Traditional transformer quantization typically applies uniform weight compression, but Needle 2 extends compression to **per-token quantization** of activation caches. The `maybe_quant_kv` function in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 42-44) applies dynamic quantization to both the KV cache and Engram outputs.

This approach reduces the memory footprint of cached representations beyond standard compression techniques. By quantizing on a per-token basis rather than applying global schemes, Needle 2 maintains precision for recent tokens while aggressively compressing older cached values.

## Attention Mechanism Implementation

The attention computation itself reflects these memory constraints. While traditional transformers use standard dot-product attention over the full $L \times L$ matrix, Needle 2 leverages JAX's `dot_product_attention` with **flash-attention** optimization—but only over the constrained `kv_window` and optional Engram-derived keys/values.

This implementation keeps attention complexity close to linear in sequence length $O(L \times \text{window\_size})$ rather than quadratic $O(L^2)$. When the backend supports GPU acceleration, the flash-attention path further optimizes the memory-bandwidth bound operations over the limited KV window.

## Summary

- **Sliding-window caching**: `effective_kv_window()` enforces a fixed token budget (typically ~400 tokens) instead of storing all past KV pairs.
- **Engram persistence**: Fixed-size hash tables provide long-range memory without linear growth, implemented in the Engram module (lines 81-96).
- **Deterministic budgeting**: The `kv_budget_window` formula (lines 1004-1011) guarantees ~28 MiB RAM usage regardless of sequence length.
- **Per-token quantization**: `maybe_quant_kv` (lines 42-44) compresses both cache and Engram outputs dynamically.
- **Linear attention cost**: Flash-attention operates only over the constrained window and Engram indices, avoiding quadratic scaling.

## Frequently Asked Questions

### What is the Engram table in Needle 2?

The Engram table is a fixed-size embedding storage mechanism that hashes token sequences into persistent memory slots. Located in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 81-96), it allows the model to retrieve distant context without storing full key-value tensors for every past token. The `engram_indices()` function (lines 46-58) generates reusable hash indices that work across layers, providing long-range dependency modeling while maintaining constant memory usage.

### How does Needle 2 maintain constant memory usage during long conversations?

Needle 2 combines three mechanisms: a sliding-window KV cache that retains only recent tokens (calculated by `effective_kv_window()`), a fixed-size Engram table for distant context, and per-token quantization via `maybe_quant_kv()`. The `kv_budget_window` formula explicitly calculates memory requirements based on model architecture rather than sequence length, capping usage at approximately 28 MiB regardless of input size.

### How does sliding-window attention differ from standard transformer attention?

Standard transformers compute attention over the full sequence length, requiring $O(L^2)$ memory and computation. Needle 2 restricts attention to a budget-constrained window (typically ~400 tokens) using `dot_product_attention` with flash-attention optimization. This limits computation to $O(L \times \text{window\_size})$, while the Engram module provides additional context without requiring full attention over distant tokens.

### What is the default memory budget for Needle 2 models?

According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the default configuration targets approximately **28 MiB of RAM** for a complete inference session. This budget accommodates the sliding-window KV cache, Engram table slots, and quantization overhead through the deterministic `kv_budget_window` calculation (lines 1004-1011), ensuring the model fits on resource-constrained devices.