How to Tune LLM Consolidation Prompt Sizing for Maximum Context Utilization in ai-memory

Tune LLM consolidation prompt sizing in ai-memory by adjusting the DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS constant (default 24,000) in auto_improve.rs to fit more observations per request, while managing the final body cap and advisory prompt limits to prevent context window overflows.

The ai-memory project (akitaonrails/ai-memory) uses an LLM-driven consolidation pipeline to compress session observations into concise wiki pages. By default, the system enforces strict hard-coded boundaries to prevent exceeding LLM context windows, but these limits can be modified to squeeze maximum information into every request.

Understanding the Hard-Coded Token Limits

The consolidation engine exposes three primary constraints that govern how much data travels to and from the LLM. Each limit serves a distinct purpose in the pipeline.

Limit	Source File	Default Value	Purpose
Input token budget	crates/ai-memory-consolidate/src/auto_improve.rs	24,000 tokens	DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS caps the total tokens sent to the LLM during consolidation.
Final body size	crates/ai-memory-consolidate/src/auto_improve.rs	100,000 characters	DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS truncates the LLM's output to keep wiki pages concise.
Advisory prompt length	crates/ai-memory-consolidate/src/consolidator.rs	2,000 characters	Hard-coded cap for the _prompts/consolidation.md payload embedded in system prompts.

How Consolidation Limits Are Applied

The ai-memory router applies these constraints in three distinct phases to ensure the LLM never receives an oversized payload.

1. Pre-flight admission – Before invoking the LLM, the router checks the token budget defined by DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS. If the batched observations overflow this budget, the request automatically splits into smaller chunks.

2. System prompt construction – The consolidator embeds the project-level _prompts/consolidation.md into the system prompt template (single_consolidate_system.md or batch_consolidate_system.md), but truncates it to 2,000 characters to preserve token headroom for user content.

3. Post-LLM truncation – After the model returns a draft, the system enforces DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS to ensure downstream storage and retrieval tools remain performant.

Step-by-Step Tuning Guide

Identify the Bottleneck

Run a consolidation and inspect the log output for the token count line:

ai-memory memory_consolidate --session-id 42

Look for entries resembling consolidate-batch(session …): X page(s) — Y tokens. If you see "exceeds max_input_tokens", the input budget is your bottleneck.

Raise the Token Budget

Increase DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS in crates/ai-memory-consolidate/src/auto_improve.rs to fit larger sessions into a single request:

// crates/ai-memory-consolidate/src/auto_improve.rs
pub const DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS: usize = 30_000; // Increased from 24_000

Recompile the workspace after editing:

cargo build --workspace

Adjust Final Body Size

If consolidated pages grow too large for downstream tooling—or if you need richer summaries—modify DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS:

pub const DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS: usize = 150_000; // Larger pages

Lowering this value forces terser outputs; raising it allows comprehensive summaries without truncation.

Control Advisory Prompt Length

The project-level _prompts/consolidation.md file is automatically truncated to 2,000 characters during system prompt assembly. If you require more advisory content, either split guidance across multiple helper pages referenced by the main prompt, or increase the hard-coded limit in consolidator.rs:

// crates/ai-memory-consolidate/src/consolidator.rs
const ADVISORY_PROMPT_MAX_CHARS: usize = 3_000; // Expand system prompt allowance

Warning: Increasing this value consumes tokens from your input budget, reducing space available for actual observations.

Enable Batch Consolidation for Large Sessions

For sessions that inevitably exceed even raised limits, invoke the multi_page=true flag via the memory_consolidate tool. This forces the system to split the payload across several LLM calls, each respecting the token budget:

ai-memory memory_consolidate --session-id 42 --multi-page

Practical Example

The following workflow demonstrates tuning the token budget after encountering a limit error:

# 1. Run consolidation with default 24,000 token budget

ai-memory memory_consolidate --session-id 42

# 2. Observe "token budget exceeded" warning in logs

# 3. Edit the constant and rebuild

sed -i 's/24_000/30_000/' crates/ai-memory-consolidate/src/auto_improve.rs
cargo build --workspace

# 4. Re-run with expanded context window

ai-memory memory_consolidate --session-id 42

Summary

• Input budget tuning: Modify DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS in crates/ai-memory-consolidate/src/auto_improve.rs to control how many tokens reach the LLM per request (default 24,000).

• Output size management: Adjust DEFAULT_AUTO_IMPROVE_MAX_FINAL_BODY_CHARS in the same file to cap consolidated page length (default 100,000 characters).

• System prompt limits: The advisory prompt from _prompts/consolidation.md is truncated at 2,000 characters in crates/ai-memory-consolidate/src/consolidator.rs.

• Batch fallback: Enable multi_page=true to automatically shard oversized sessions across multiple LLM calls without manual intervention.

Frequently Asked Questions

What happens if I set DEFAULT_AUTO_IMPROVE_MAX_INPUT_TOKENS higher than my LLM's context window?

The LLM provider will return a context length exceeded error, or the underlying tokenizer will truncate the input mid-sentence, potentially corrupting the semantic meaning of your observations. Always set this value at least 1,000–2,000 tokens below your model's absolute limit to reserve space for the system prompt and response generation.

Why is the advisory prompt limited to 2,000 characters instead of tokens?

The ai-memory codebase measures the _prompts/consolidation.md payload in characters rather than tokens because the Rust consolidator performs a simple string truncation before sending the request to the tokenizer. This ensures predictable byte sizes and avoids complex token-counting logic during prompt assembly, though it may slightly under-utilize the context window for character-efficient languages.

How do I know if I should increase the token budget or switch to batch mode?

Increase the token budget if your sessions consistently fall just above the 24,000-token threshold (e.g., 25,000–35,000 tokens) and you want single-pass consolidation for coherence. Switch to batch mode (multi_page=true) if individual sessions regularly exceed 50,000 tokens or if you observe memory pressure during the Rust compilation of extremely high token constants.

Where are the system prompt templates stored?

The consolidation system prompts reside in crates/ai-memory-consolidate/prompts/single_consolidate_system.md and batch_consolidate_system.md. These templates define how the LLM interprets the advisory prompt and observation batches, but you should not edit the truncation logic here; instead, adjust the hard-coded constants in auto_improve.rs and consolidator.rs to change input boundaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}