# How to Tune LLM Consolidation Prompt Sizing for Maximum Context Utilization in ai-memory

> Learn to tune LLM consolidation prompt sizing in ai-memory. Adjust DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS to maximize context utilization and fit more observations per request. Prevent overflows with careful limits.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: performance
- Published: 2026-08-27

---

**Tune LLM consolidation prompt sizing in ai-memory by adjusting the `DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS` constant (default 24,000) in [`auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/auto_improve.rs) to fit more observations per request, while managing the final body cap and advisory prompt limits to prevent context window overflows.**

The ai-memory project (akitaonrails/ai-memory) uses an LLM-driven consolidation pipeline to compress session observations into concise wiki pages. By default, the system enforces strict hard-coded boundaries to prevent exceeding LLM context windows, but these limits can be modified to squeeze maximum information into every request.

## Understanding the Hard-Coded Token Limits

The consolidation engine exposes three primary constraints that govern how much data travels to and from the LLM. Each limit serves a distinct purpose in the pipeline.

| Limit | Source File | Default Value | Purpose |
|-------|-------------|---------------|---------|
| **Input token budget** | [`crates/ai-memory-consolidate/src/auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/auto_improve.rs) | 24,000 tokens | `DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS` caps the total tokens sent to the LLM during consolidation. |
| **Final body size** | [`crates/ai-memory-consolidate/src/auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/auto_improve.rs) | 100,000 characters | `DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS` truncates the LLM's output to keep wiki pages concise. |
| **Advisory prompt length** | [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs) | 2,000 characters | Hard-coded cap for the [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) payload embedded in system prompts. |

## How Consolidation Limits Are Applied

The ai-memory router applies these constraints in three distinct phases to ensure the LLM never receives an oversized payload.

1. **Pre-flight admission** – Before invoking the LLM, the router checks the token budget defined by `DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS`. If the batched observations overflow this budget, the request automatically splits into smaller chunks.

2. **System prompt construction** – The consolidator embeds the project-level [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) into the system prompt template ([`single_consolidate_system.md`](https://github.com/akitaonrails/ai-memory/blob/main/single_consolidate_system.md) or [`batch_consolidate_system.md`](https://github.com/akitaonrails/ai-memory/blob/main/batch_consolidate_system.md)), but truncates it to 2,000 characters to preserve token headroom for user content.

3. **Post-LLM truncation** – After the model returns a draft, the system enforces `DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS` to ensure downstream storage and retrieval tools remain performant.

## Step-by-Step Tuning Guide

### Identify the Bottleneck

Run a consolidation and inspect the log output for the token count line:

```bash
ai-memory memory_consolidate --session-id 42

```

Look for entries resembling `consolidate-batch(session …): X page(s) — Y tokens`. If you see "exceeds max_input_tokens", the input budget is your bottleneck.

### Raise the Token Budget

Increase `DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS` in [`crates/ai-memory-consolidate/src/auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/auto_improve.rs) to fit larger sessions into a single request:

```rust
// crates/ai-memory-consolidate/src/auto_improve.rs
pub const DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS: usize = 30_000; // Increased from 24_000

```

Recompile the workspace after editing:

```bash
cargo build --workspace

```

### Adjust Final Body Size

If consolidated pages grow too large for downstream tooling—or if you need richer summaries—modify `DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS`:

```rust
pub const DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS: usize = 150_000; // Larger pages

```

Lowering this value forces terser outputs; raising it allows comprehensive summaries without truncation.

### Control Advisory Prompt Length

The project-level [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) file is automatically truncated to 2,000 characters during system prompt assembly. If you require more advisory content, either split guidance across multiple helper pages referenced by the main prompt, or increase the hard-coded limit in [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs):

```rust
// crates/ai-memory-consolidate/src/consolidator.rs
const ADVISORY_PROMPT_MAX_CHARS: usize = 3_000; // Expand system prompt allowance

```

**Warning:** Increasing this value consumes tokens from your input budget, reducing space available for actual observations.

### Enable Batch Consolidation for Large Sessions

For sessions that inevitably exceed even raised limits, invoke the `multi_page=true` flag via the `memory_consolidate` tool. This forces the system to split the payload across several LLM calls, each respecting the token budget:

```bash
ai-memory memory_consolidate --session-id 42 --multi-page

```

## Practical Example

The following workflow demonstrates tuning the token budget after encountering a limit error:

```bash

# 1. Run consolidation with default 24,000 token budget

ai-memory memory_consolidate --session-id 42

# 2. Observe "token budget exceeded" warning in logs

# 3. Edit the constant and rebuild

sed -i 's/24_000/30_000/' crates/ai-memory-consolidate/src/auto_improve.rs
cargo build --workspace

# 4. Re-run with expanded context window

ai-memory memory_consolidate --session-id 42

```

## Summary

- **Input budget tuning:** Modify `DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS` in [`crates/ai-memory-consolidate/src/auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/auto_improve.rs) to control how many tokens reach the LLM per request (default 24,000).

- **Output size management:** Adjust `DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS` in the same file to cap consolidated page length (default 100,000 characters).

- **System prompt limits:** The advisory prompt from [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) is truncated at 2,000 characters in [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs).

- **Batch fallback:** Enable `multi_page=true` to automatically shard oversized sessions across multiple LLM calls without manual intervention.

## Frequently Asked Questions

### What happens if I set `DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS` higher than my LLM's context window?

The LLM provider will return a context length exceeded error, or the underlying tokenizer will truncate the input mid-sentence, potentially corrupting the semantic meaning of your observations. Always set this value at least 1,000–2,000 tokens below your model's absolute limit to reserve space for the system prompt and response generation.

### Why is the advisory prompt limited to 2,000 characters instead of tokens?

The ai-memory codebase measures the [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) payload in characters rather than tokens because the Rust consolidator performs a simple string truncation before sending the request to the tokenizer. This ensures predictable byte sizes and avoids complex token-counting logic during prompt assembly, though it may slightly under-utilize the context window for character-efficient languages.

### How do I know if I should increase the token budget or switch to batch mode?

Increase the token budget if your sessions consistently fall just above the 24,000-token threshold (e.g., 25,000–35,000 tokens) and you want single-pass consolidation for coherence. Switch to batch mode (`multi_page=true`) if individual sessions regularly exceed 50,000 tokens or if you observe memory pressure during the Rust compilation of extremely high token constants.

### Where are the system prompt templates stored?

The consolidation system prompts reside in [`crates/ai-memory-consolidate/prompts/single_consolidate_system.md`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/prompts/single_consolidate_system.md) and [`batch_consolidate_system.md`](https://github.com/akitaonrails/ai-memory/blob/main/batch_consolidate_system.md). These templates define how the LLM interprets the advisory prompt and observation batches, but you should not edit the truncation logic here; instead, adjust the hard-coded constants in [`auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/auto_improve.rs) and [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs) to change input boundaries.