What is the ai-memory-consolidate Crate? Karpathy-Style LLM Ingest Pipeline Explained

The ai-memory-consolidate crate implements the Karpathy-style LLM ingest pipeline that transforms raw session observations into structured, versioned wiki pages through automated consolidation, embedding backfill, and iterative improvement.

The ai-memory-consolidate crate serves as the core knowledge engine within the akitaonrails/ai-memory repository, converting fleeting session data into durable, searchable documentation. It orchestrates the transformation of unstructured observations into structured markdown pages while maintaining version history and semantic search capabilities.

Core Responsibilities of the ai-memory-consolidate Crate

The crate handles ten distinct areas of responsibility that together form a complete knowledge management pipeline.

Single-Page Consolidation

The Consolidator::consolidate_session method in crates/ai-memory-consolidate/src/consolidator.rs (lines 34-47) handles the rewrites of individual sessions/<id>.md files. This function reads raw observations, executes an LLM prompt to generate a summary, writes the updated page, and auto-commits the supersession chain to maintain version history.

Multi-Page Batch Consolidation

For complex sessions spanning multiple domains, Consolidator::consolidate_session_multi (lines 28-35) produces atomic updates across sessions/, concepts/, decisions/, and _slots/ directories. This method builds a batch request, receives a ConsolidatedBatch from the LLM, and writes all pages atomically to prevent partial state corruption.

Prompt Construction and Token Budgeting

The crate implements sophisticated prompt management through build_request, build_batch_request_with_slots, and the PromptBudgets struct (lines 98-115). These utilities limit input/output tokens, inject project-level instructions, slot snapshots, and current-page context to maximize information density within LLM context windows.

Auxiliary Processing Pipelines

Beyond basic consolidation, the crate runs several maintenance workflows:

• Embedding back-fill – run_embedding_backfill in embed.rs generates vector embeddings for newly created pages to enable semantic search.
• Linting – run_lint in lint.rs validates page front-matter, titles, and rule locations after consolidation.
• Sweep (pruning) – run_sweep in sweep.rs removes stale observations and pages that no longer contribute to active knowledge.
• Auto-improve – run_auto_improve_review in auto_improve.rs periodically reviews consolidated content, proposes edits, and applies accepted changes automatically.

Bootstrap and Projection

The Bootstrap struct in bootstrap.rs discovers repository roots, chunks raw sources, and prepares the initial observation set for new projects. For handling large observation logs, project_observations in projection.rs converts massive logs into token-bounded snippets suitable for LLM prompts.

The Consolidation Pipeline Workflow

The ai-memory-consolidate crate implements a seven-stage pipeline that processes raw data into structured knowledge:

1. Collects raw observations via the ai-memory-core interface.
2. Projects observations into size-limited text blocks using token budgeting.
3. Adds project-level instructions and slot snapshots to the prompt context.
4. Sends structured prompts to a configured LLM provider.
5. Receives structured JSON responses (ConsolidatedPage or ConsolidatedBatch).
6. Writes pages atomically while preserving versioned supersession chains.
7. Runs auxiliary steps (embedding, lint, sweep, auto-improve) to maintain knowledge base hygiene.

Code Examples

Basic Single-Page Consolidation

use ai_memory_consolidate::Consolidator;
use ai_memory_store::{ReaderPool, WriterHandle};
use ai_memory_wiki::Wiki;
use std::sync::Arc;

// Assume we have already built the required handles …
let consolidator = Consolidator::new(reader, writer, wiki, llm_provider, workspace_id, project_id);

// Consolidate session 42, no dry‑run, default actor, no per‑call instructions.
let outcome = consolidator
    .consolidate_session(
        SessionId::from(42),
        false,
        ActorContext::default(),
        None,
        None,
    )
    .await
    .expect("consolidation failed");

// `outcome` tells you where the new page lives and its content.
println!("✅ Consolidated to {}", outcome.path);
println!("Title: {}", outcome.new_title);

Multi-Page Batch Consolidation

let outcomes = consolidator
    .consolidate_session_multi(
        SessionId::from(42),
        false,
        ActorContext::default(),
        None,
        Some("Prefer short, bullet‑point summaries."),
    )
    .await
    .expect("batch consolidation failed");

// Print a quick summary of all pages updated.
for o in outcomes {
    println!("🔖 {} → {}", o.path, o.new_title);
}

Building Custom Batch Requests

use ai_memory_consolidate::{build_batch_request, SessionId};

let request = build_batch_request(SessionId::from(42), &observations);
// `request` can be sent directly to any `LlmProvider` that implements the
// `complete_structured_with_operation_id` contract.

Key Source Files and Architecture

Understanding the crate's architecture requires familiarity with these specific files:

• crates/ai-memory-consolidate/src/lib.rs – Public API surface and module re-exports.
• crates/ai-memory-consolidate/src/consolidator.rs – Core Consolidator implementation with single and multi-page pipelines.
• crates/ai-memory-consolidate/src/types.rs – Data structures including ConsolidatedPage, ConsolidatedBatch, and ConsolidationOutcome.
• crates/ai-memory-consolidate/src/bootstrap.rs – Project bootstrapping logic for source discovery and chunking.
• crates/ai-memory-consolidate/src/curator.rs – Human-readable report generation via run_curator_report.
• crates/ai-memory-consolidate/src/embed.rs – Vector embedding generation via run_embedding_backfill.
• crates/ai-memory-consolidate/src/lint.rs – Page validation rules via run_lint.
• crates/ai-memory-consolidate/src/sweep.rs – Knowledge base pruning via run_sweep.
• crates/ai-memory-consolidate/src/auto_improve.rs – Self-improvement workflows via run_auto_improve_review.
• crates/ai-memory-consolidate/src/projection.rs – Observation compression via project_observations.
• crates/ai-memory-consolidate/prompts/* – System prompt templates for all pipeline stages.

Summary

• The ai-memory-consolidate crate transforms raw session observations into structured wiki pages using LLM-powered consolidation.
• Single-page consolidation handles individual sessions while batch consolidation manages complex multi-directory updates atomically.
• Token budgeting via PromptBudgets ensures efficient LLM utilization without context overflow.
• Auxiliary pipelines (embedding, linting, sweeping, auto-improve) maintain long-term knowledge base quality and searchability.
• All operations preserve versioned supersession chains to maintain complete audit trails of knowledge evolution.

Frequently Asked Questions

What does the ai-memory-consolidate crate do?

The ai-memory-consolidate crate functions as the central processing engine that converts unstructured session data into structured, versioned markdown documentation. It implements the Karpathy-style ingest pattern by collecting observations, projecting them into LLM-friendly snippets, generating summaries through structured prompts, and maintaining the resulting knowledge base through automated linting, embedding generation, and periodic pruning.

How does single-page consolidation differ from batch consolidation?

Single-page consolidation via Consolidator::consolidate_session processes one session file at a time, ideal for simple updates or testing. Batch consolidation via consolidate_session_multi handles complex sessions that spawn multiple pages across different directories (concepts, decisions, slots) in a single atomic transaction, ensuring consistency when sessions touch multiple knowledge domains simultaneously.

What is the Karpathy-style ingest pipeline referenced in the source code?

The Karpathy-style ingest pipeline refers to a specific pattern of LLM knowledge management that emphasizes turning raw, ephemeral data (session logs, observations) into structured, durable formats (wiki pages) through automated LLM processing. As implemented in akitaonrails/ai-memory, this pattern includes stages for collection, projection, consolidation, and continuous improvement via embeddings and auto-review cycles.

How does the crate handle large observation logs that exceed LLM context limits?

The crate handles token limitations through the project_observations function in projection.rs, which compresses large observation logs into token-bounded snippets. Additionally, the PromptBudgets struct manages input/output token allocation across the prompt construction pipeline, ensuring that project instructions, slot snapshots, and observation data fit within the configured LLM's context window without truncation of critical information.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}