How ai-memory Consolidates Session Data into Wiki Pages: A 5-Stage Pipeline

ai-memory transforms raw MCP session observations into structured, version-controlled wiki pages through a five-stage pipeline that validates permissions, constructs Karpathy-style prompts, generates structured JSON via LLM, and atomically commits markdown to Git.

The ai-memory project (akitaonrails/ai-memory) implements a persistent memory layer for AI agents that converts transient session logs into navigable knowledge bases. When you consolidate session data into wiki pages, the system orchestrates SQLite queries, admission checks, and structured LLM outputs to produce version-controlled markdown documentation with full provenance tracking.

The Consolidation Pipeline

The process of turning observations into wiki pages follows a strict sequence implemented primarily in crates/ai-memory-consolidate/src/consolidator.rs.

Stage 1: Gathering Observations from SQLite

The pipeline begins with ReaderPool::observations_for_session, defined in crates/ai-memory-store/src/reader.rs. This method pulls every Observation belonging to the target SessionId from the SQLite store, creating the raw dataset for consolidation.

Stage 2: Preflight Admission Validation

Before invoking any LLM, the system validates permissions via Wiki::preflight_admission (located at line 2234 in crates/ai-memory-wiki/src/wiki.rs). This check ensures the requesting actor has write access to the target page, preventing costly LLM calls when the scope or identity is denied.

Stage 3: Karpathy-Style Prompt Construction

The Consolidator builds a Karpathy-style prompt containing:

• The target session ID
• A trimmed excerpt of the current page body (if updating existing content)
• A token-budgeted projection of the observation log
• Optional project-wide instructions from _prompts/consolidation.md

The helper build_request (line 989 in consolidator.rs) assembles these elements into a ChatRequest ready for the LLM provider.

Stage 4: Structured LLM Completion

The system calls ai_memory_llm::LlmProvider::complete_structured_with_operation_id, requesting a JSON object conforming to the ConsolidatedPage schema defined in crates/ai-memory-consolidate/src/types.rs. The LLM returns structured data including the proposed markdown body, title, and metadata tier.

Stage 5: Atomic Page Write and Git Commit

Final persistence happens through several coordinated steps:

1. Front-matter assembly via build_frontmatter, injecting fields like title, tier, origin, and consolidated: true
2. Atomic write via Wiki::write_page (line 5091 in wiki.rs), which creates a supersession chain in the database
3. Version control via Wiki::commit_all, generating a commit message following the pattern consolidate(session <X>): <title> and auto-committing to the Git repository

Multi-Page Batch Consolidation

When multi_page mode is requested, the pipeline uses consolidate_session_multi instead of consolidate_session. The process diverges at the prompt stage:

• build_batch_request_with_slots (line 2774 in consolidator.rs) constructs a request for a ConsolidatedBatch output rather than a single page
• The LLM returns multiple page updates in one response
• build_update transforms each batch item into a WritePageRequest, respecting slot namespacing, rule routing, and invariant-slot protection
• Wiki::apply_batch executes all writes within a single SQL transaction
• A single Git commit captures the entire batch

Key Architectural Components

• Consolidator (in consolidator.rs): The main orchestrator exposing consolidate_session and consolidate_session_multi. It holds handles to the store, wiki, writer, and LLM provider.

• ReaderPool (in reader.rs): Handles data retrieval via observations_for_session and session_scope_from_observations.

• Wiki (in wiki.rs): Manages atomic markdown writes, admission checks, and version control through preflight_admission, write_page, apply_batch, and commit_all.

• LLM Provider (in ai-memory-llm): Generates structured consolidation output via complete_structured_with_operation_id.

• MCP Endpoint (in server.rs at line 2583): Exposes the memory_consolidate tool that external callers use to trigger the pipeline.

Implementation Examples

Triggering Consolidation via MCP

// Example using the generated client interface
let client = ai_memory_mcp::Client::new("http://localhost:49374");
let outcome = client.memory_consolidate(
    SessionId::from_str("a1b2c3d4")?,
    /*dry=*/ false,
    /*multi_page=*/ false,
    /*instructions=*/ None,
).await?;
println!("Consolidated page at {}", outcome.path);

Direct Consolidator Usage

use ai_memory_consolidate::Consolidator;
use ai_memory_store::{ReaderPool, WriterHandle};
use ai_memory_wiki::Wiki;
use ai_memory_llm::openai::OpenAiProvider;
use std::sync::Arc;

let llm = Arc::new(OpenAiProvider::new("gpt-4")?);
let consolidator = Consolidator::new(reader, writer, wiki, llm, ws_id, proj_id);

let outcome = consolidator
    .consolidate_session(
        session_id,
        /*dry_run=*/ false,
        actor_context,
        Some(author_user_id),
        None, // no per-call instructions
    )
    .await?;
println!("Page written with ID {}", outcome.page_id.unwrap());

Batch Multi-Page Consolidation

let outcomes = consolidator
    .consolidate_session_multi(
        session_id,
        false,
        actor_context,
        Some(author_user_id),
        None,
    )
    .await?;
for o in outcomes {
    println!("Updated {} (page id {:?})", o.path, o.page_id);
}

Summary

• ai-memory uses a five-stage pipeline (gather, validate, prompt, generate, commit) to turn session observations into wiki pages.
• Preflight admission checks in wiki.rs prevent unauthorized LLM usage before expensive calls are made.
• The system supports both single-page (ConsolidatedPage) and multi-page (ConsolidatedBatch) consolidation modes.
• All writes are atomic—either database transactions via apply_batch or Git commits via commit_all—ensuring data integrity.
• Prompts follow a Karpathy-style format and can include custom instructions from project configuration files.

Frequently Asked Questions

What triggers the consolidation pipeline in ai-memory?

The pipeline is triggered by the memory_consolidate MCP tool handler (found at line 2583 in crates/ai-memory-mcp/src/server.rs). External AI agents or scripts invoke this tool via the MCP protocol, passing a SessionId and optional flags like multi_page or dry_run to control behavior.

How does the system prevent unauthorized wiki modifications during consolidation?

Before constructing LLM prompts, Wiki::preflight_admission validates that the requesting actor has write permissions for the target page. This check occurs at line 2234 in crates/ai-memory-wiki/src/wiki.rs and prevents costly LLM operations when the actor lacks appropriate scope or access rights.

What distinguishes single-page from multi-page consolidation?

Single-page consolidation uses consolidate_session and expects the LLM to return a single ConsolidatedPage object. Multi-page consolidation uses consolidate_session_multi with build_batch_request_with_slots to request a ConsolidatedBatch, allowing the LLM to propose updates across multiple wiki slots in one operation. The batch mode respects slot namespacing and invariant-slot protection rules during the apply_batch transaction.

How does ai-memory ensure data integrity during batch operations?

Batch consolidation wraps all database writes in a single SQL transaction via Wiki::apply_batch. If any individual page write fails (due to invariant violations or rule routing conflicts), the entire transaction rolls back. After successful database persistence, Wiki::commit_all creates a single Git commit encompassing all modified files, maintaining consistency between the SQL store and the version-controlled markdown repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}