# How ai-memory Implements the Karpathy-Style Consolidation Pipeline for Observations to Wiki Pages

> Discover how ai-memory implements the Karpathy-style consolidation pipeline. Transform observations into version-controlled wiki pages with a three-stage LLM process.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-30

---

**ai-memory implements a three-stage consolidation pipeline that transforms raw session observations into durable, version-controlled markdown wiki pages by collecting observations into bounded prompts, using structured LLM completion, and atomically writing results with Git-based supersession.**

The `akitaonrails/ai-memory` project provides a Rust-based memory system inspired by Andrej Karpathy's LLM wiki concept: ephemeral observations accumulate during agent sessions, then get periodically distilled into clean, enduring knowledge pages. This article breaks down exactly how the `ai-memory-consolidate` crate implements this pipeline.

## What Is the Karpathy-Style Consolidation Pattern?

Karpathy's original vision treats LLM sessions as producing raw, stream-of-consciousness observations that must be periodically rewritten into structured, persistent documentation. The key characteristics are:

- **Immutable raw data**: observations are append-only and never modified
- **LLM-powered rewrite**: a capable model condenses noise into signal
- **Versioned supersession**: new pages replace old ones while preserving history
- **Atomic commits**: the entire operation succeeds or rolls back cleanly

ai-memory realizes this through a dedicated consolidation crate that bridges the observation store, LLM provider, and wiki subsystem.

## Stage 1: Collecting and Projecting Observations

The pipeline begins in [`projection.rs`](https://github.com/akitaonrails/ai-memory/blob/main/projection.rs) with the `Consolidator` fetching session data via `ReaderPool::observations_for_session`.

The `project_observations` function transforms raw observations into a compact text representation that respects configurable character budgets. Each observation body gets trimmed to safe size, ensuring the final prompt fits within LLM context windows.

Key implementation in [[`crates/ai-memory-consolidate/src/projection.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/projection.rs)](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/projection.rs):

```rust
// Observations are chronologically ordered and size-trimmed
let projected = project_observations(observations, budget);
// Result: "2024-01-15T09:23:00Z [user]: Started debugging the auth flow..."

```

This projection step is critical for the Karpathy pattern—it prevents prompt overflow while preserving temporal structure and attribution.

## Stage 2: Building the LLM Request

The prompt assembly logic lives in [[`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs)](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs). Two system prompt templates ship with the crate:

- **[`single_consolidate_system.md`](https://github.com/akitaonrails/ai-memory/blob/main/single_consolidate_system.md)**: for consolidating one page at a time
- **[`batch_consolidate_system.md`](https://github.com/akitaonrails/ai-memory/blob/main/batch_consolidate_system.md)**: for multi-page updates up to five slots

These compile-time embedded prompts define the exact contract the LLM must follow.

### Request Structure

The `build_request` and `build_batch_request_with_slots` functions assemble:

1. **Prefix**: session ID and metadata
2. **Observation dump**: the projected content from Stage 1
3. **Context slot**: current page body (single) or slot snapshots (batch)
4. **Custom instructions**: optional [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) per project
5. **JSON schema suffix**: mandatory output format specification

Prompt budgeting via `PromptBudgets` (see [line 70 of consolidator.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs#line%2070)) ensures the total payload stays within provider limits.

```rust
let request = build_request(
    session_id,
    projected_observations,
    current_body,           // None for new pages
    project_instructions,   // Optional custom guidance
    budgets,                // PromptBudgets { total: 100_000, response: 32_000 }
);

```

## Stage 3: LLM Completion and Wiki Write

The structured request flows through `ai_memory_llm::complete_structured`, which enforces response schema compliance. The LLM must return:

- **`ConsolidatedPage`** (single): title, body, tags, description
- **`ConsolidatedBatch`** (multi): vector of `ConsolidatedPageUpdate` objects

### Front-Matter Construction and Atomic Commit

Before writing, `build_frontmatter` stamps each page with:

```rust
fn build_frontmatter(page: &ConsolidatedPage, session_id: Uuid, agent_kind: &str) -> BTreeMap<String, FrontMatterValue> {
    let mut fm = BTreeMap::new();
    fm.insert("session_id".into(), session_id.to_string().into());
    fm.insert("consolidated".into(), true.into());
    fm.insert("agent_kind".into(), agent_kind.into());
    fm.extend(page.tags.iter().map(|t| (format!("tag:{}", t), true.into())));
    fm
}

```

The `Wiki` component then handles atomic storage:

- **`Wiki::write_page`** for single-page consolidation
- **`Wiki::apply_batch`** for multi-page atomic updates

Both trigger `wiki.commit_all("consolidate(session ...)")`, creating a new Git revision that preserves the supersession chain.

## Single-Page Consolidation Flow

The `consolidate_session` method orchestrates the complete pipeline in [[`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs) lines 31–225](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs):

```rust
// 1. Fetch observations
let observations = reader.observations_for_session(session_id).await?;

// 2. Resolve workspace/project
let (ws, proj) = resolve_target(&writer, workspace_id, project_id).await?;

// 3. Preflight admission check
wiki.preflight_admission(&ws, &proj, &path).await?;

// 4. Read existing page body if present
let current_body = wiki.read_page(&ws, &proj, &path).await.ok().map(|p| p.body);

// 5. Build and send LLM request
let request = build_request(session_id, observations, current_body, instructions, budgets);
let page: ConsolidatedPage = llm.complete_structured(request).await?;

// 6. Assemble and write
let frontmatter = build_frontmatter(&page, session_id, agent_kind);
let id = wiki.write_page(WritePageRequest {
    workspace_id: ws.id,
    project_id: proj.id,
    path: page.title.to_slug(),
    body: page.body,
    frontmatter,
}).await?;

// 7. Atomic commit
wiki.commit_all(&format!("consolidate({})", session_id)).await?;

```

Lines [62–90](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs), [119–130](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs), [140–158](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs), and [176–194](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs) contain the specific implementation segments.

## Multi-Page Batch Consolidation

For long sessions producing disparate knowledge, `consolidate_session_multi` extends the pipeline with slot-aware batching. Key additions in [[`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs) lines 218–380](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs):

1. **Slot snapshot retrieval**: fetches pinned memory contexts via [`curator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/curator.rs) integration
2. **Batch prompt construction**: `build_batch_request_with_slots` prepares a single prompt returning up to five page updates
3. **Atomic batch write**: `Wiki::apply_batch` commits all updates or none

The slot mechanism lets operators pin critical facts that should persist across consolidations—essential for maintaining continuity in long-running agent workflows.

## Practical Usage Example

```rust
use ai_memory_consolidate::Consolidator;
use ai_memory_store::{ReaderPool, WriterHandle};
use ai_memory_wiki::Wiki;
use std::sync::Arc;
use ai_memory_llm::providers::OpenAiProvider;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize core components
    let reader: ReaderPool = /* configured elsewhere */;
    let writer: WriterHandle = /* configured elsewhere */;
    let wiki = Wiki::new(/* config */);
    let llm = Arc::new(OpenAiProvider::new("gpt-4o-mini")?);

    // Create consolidator with optional limits
    let consolidator = Consolidator::new(
        reader, writer, wiki, llm,
        workspace_id, project_id
    ).with_prompt_limits(100_000, 32_000);

    // Single-page consolidation
    let outcome = consolidator
        .consolidate_session(session_id, false, actor_ctx, None, None)
        .await?;
    println!("Page written: {}", outcome.path);

    // Multi-page batch for complex sessions
    let batch = consolidator
        .consolidate_session_multi(session_id, false, actor_ctx, None, None)
        .await?;
    for update in batch {
        println!("Updated {} ({:?})", update.path, update.page_id);
    }

    Ok(())
}

```

## Error Handling and Observability

All failure modes wrap in `ConsolidatorError`:

- **Store errors**: observation fetch or metadata resolution failures
- **Wiki errors**: preflight rejection, write conflicts, or commit failures
- **LLM errors**: structured completion parse failures or schema violations

The dry-run parameter (`false` in examples above) previews the operation without persisting changes—essential for testing prompt changes or custom instructions.

## Summary

- ai-memory's consolidation pipeline implements Karpathy's vision through three tightly coupled stages: observation projection, structured LLM prompting, and atomic wiki writes
- The `ai-memory-consolidate` crate provides both single-page (`consolidate_session`) and multi-page batch (`consolidate_session_multi`) variants
- Prompt budgeting and compile-time templates ensure reliable operation across different LLM providers and context window sizes
- Git-based atomic commits preserve complete version history while enabling clean supersession of outdated knowledge
- Slot snapshots in multi-page mode support operator-pinned memory for continuity across long agent sessions

## Frequently Asked Questions

### What makes this "Karpathy-style" specifically?

The term refers to Andrej Karpathy's described pattern of treating LLM session outputs as raw observations that get periodically rewritten into clean, durable wiki pages. ai-memory matches this with immutable observation logs, LLM-powered condensation, and versioned markdown storage where new pages atomically replace old ones while preserving Git history.

### How does prompt budgeting prevent context overflow?

The `PromptBudgets` struct (defined in [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs) line 70) enforces hard limits on total input tokens and reserved response capacity. The `project_observations` function aggressively trims observation bodies until the projected content fits within `budget.total - budget.response`, ensuring the LLM request always succeeds.

### Can I customize the consolidation behavior per project?

Yes. The pipeline checks for [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md) in each project root and appends its contents to the system prompt. This lets project owners inject domain-specific guidance (e.g., "prefer bullet points" or "always include code examples") without modifying the core templates.

### What happens if the LLM returns malformed JSON?

`complete_structured` validates responses against the `ConsolidatedPage` or `ConsolidatedBatch` schema defined in [`types.rs`](https://github.com/akitaonrails/ai-memory/blob/main/types.rs). Parse failures propagate as `ConsolidatorError::Llm` variants, allowing callers to retry with exponential backoff or escalate to manual review.