# Default Parameters for the 'needle' Preset in TransformerConfig

> Discover the default parameters for the 'needle' preset in TransformerConfig: d_model=768, num_heads=12, num_kv_heads=6, num_layers=27, and engram_layers=(2, 15). Learn more about this configuration.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: api-reference
- Published: 2026-08-14

---

**The `needle` preset configures `TransformerConfig` with `d_model=768`, `num_heads=12`, `num_kv_heads=6`, `num_layers=27`, and `engram_layers=(2, 15)` as default architectural values sourced from [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).**

The `needle` preset provides a pre-tuned transformer architecture for the cactus-compute/needle repository's Needle model. When applied to a `TransformerConfig` instance, these default parameters initialize the model's embedding dimensions, attention heads, and layer topology without requiring explicit per-field configuration.

## Default Parameter Values for the 'needle' Preset

The **`needle` preset** supplies the following architectural defaults when loaded into `TransformerConfig`:

- **`d_model`**: `768` — The dimensionality of the model's embeddings and hidden states
- **`num_heads`**: `12` — The number of attention heads in the multi-head attention mechanism
- **`num_kv_heads`**: `6` — The number of key-value heads for grouped query attention
- **`num_layers`**: `27` — The total number of transformer decoder layers
- **`engram_layers`**: `(2, 15)` — Specific layer indices where engram (memory) mechanisms are activated

These values are hardcoded in the `PRESETS` dictionary within **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** at lines 38‑41 according to the source code. When instantiating `TransformerConfig` with this preset, any unspecified fields retain these values while other parameters like `vocab_size=8192` and `max_seq_len=2048` use their own independent defaults.

## Source Code Location

The preset definitions reside in **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)**, which contains the `TransformerConfig` dataclass and the `PRESETS` mapping. The `needle` preset is specifically defined in the `PRESETS` dictionary at lines 38‑41, making it available for both direct configuration instantiation and high-level model initialization.

```python

# Conceptual view of the PRESETS structure in needle/model/architecture.py

PRESETS = {
    "needle": {
        "d_model": 768,
        "num_heads": 12,
        "num_kv_heads": 6,
        "num_layers": 27,
        "engram_layers": (2, 15),
    },
    # ... other presets

}

```

## How to Use the 'needle' Preset

You can apply the default parameters by unpacking the `PRESETS` dictionary directly into `TransformerConfig` or by using the high-level `Needle` API.

### Direct TransformerConfig Instantiation

Access the preset through the `PRESETS` export to create a configuration with the default parameters:

```python
from needle.model.architecture import TransformerConfig, PRESETS

# Create a config using the "needle" preset

needle_cfg = TransformerConfig(**PRESETS["needle"])

print("d_model:", needle_cfg.d_model)           # 768

print("num_heads:", needle_cfg.num_heads)       # 12

print("num_kv_heads:", needle_cfg.num_kv_heads) # 6

print("num_layers:", needle_cfg.num_layers)     # 27

print("engram_layers:", needle_cfg.engram_layers)  # (2, 15)

```

### High-Level Model Initialization

The public API exposes the preset through the `Needle` class constructor:

```python
from needle import Needle

# Initialize a Needle model with the preset configuration

model = Needle(preset="needle")   # internally loads the same values

```

## Summary

- The **`needle` preset** defines five core architectural parameters: `d_model=768`, `num_heads=12`, `num_kv_heads=6`, `num_layers=27`, and `engram_layers=(2, 15)`.
- These defaults originate from the `PRESETS` dictionary in **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** (lines 38‑41).
- You can instantiate configurations via `TransformerConfig(**PRESETS["needle"])` or use `Needle(preset="needle")` for simplified model creation.
- Additional fields like `vocab_size` and `max_seq_len` maintain separate defaults independent of the architectural preset.

## Frequently Asked Questions

### What is the purpose of the engram_layers parameter in the needle preset?

The **`engram_layers`** parameter specifies which transformer layers implement the engram mechanism, set to `(2, 15)` in the needle preset. This tuple indicates that layers 2 and 15 incorporate specialized memory or retrieval components distinct from standard attention blocks.

### Can I override individual parameters when using the needle preset?

Yes, you can override any default value by passing explicit keyword arguments after unpacking the preset. For example, `TransformerConfig(**PRESETS["needle"], d_model=512)` creates a configuration with the needle preset's 12 attention heads but a reduced embedding dimension of 512.

### Where are additional configuration fields like vocab_size defined?

Fields such as `vocab_size=8192` and `max_seq_len=2048` are defined as dataclass defaults within the **`TransformerConfig`** class itself in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), separate from the `PRESETS` dictionary. They apply universally unless explicitly overridden.

### Where are the needle preset defaults validated in the test suite?

The **[`tests/test_build.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py)** file contains test suites that validate preset handling and model construction, ensuring that `TransformerConfig` correctly applies the default parameters from `PRESETS["needle"]` during model initialization.