# How ai-memory Consolidates Session Data into Wiki Pages: A 5-Stage Pipeline

> Discover how ai-memory consolidates session data into wiki pages. Explore its 5-stage pipeline for structured, version-controlled knowledge management.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-09-09

---

**ai-memory transforms raw MCP session observations into structured, version-controlled wiki pages through a five-stage pipeline that validates permissions, constructs Karpathy-style prompts, generates structured JSON via LLM, and atomically commits markdown to Git.**

The ai-memory project (akitaonrails/ai-memory) implements a persistent memory layer for AI agents that converts transient session logs into navigable knowledge bases. When you consolidate session data into wiki pages, the system orchestrates SQLite queries, admission checks, and structured LLM outputs to produce version-controlled markdown documentation with full provenance tracking.

## The Consolidation Pipeline

The process of turning observations into wiki pages follows a strict sequence implemented primarily in [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs).

### Stage 1: Gathering Observations from SQLite

The pipeline begins with `ReaderPool::observations_for_session`, defined in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs). This method pulls every `Observation` belonging to the target `SessionId` from the SQLite store, creating the raw dataset for consolidation.

### Stage 2: Preflight Admission Validation

Before invoking any LLM, the system validates permissions via `Wiki::preflight_admission` (located at line 2234 in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs)). This check ensures the requesting **actor** has write access to the target page, preventing costly LLM calls when the scope or identity is denied.

### Stage 3: Karpathy-Style Prompt Construction

The `Consolidator` builds a **Karpathy-style** prompt containing:

- The target session ID
- A trimmed excerpt of the current page body (if updating existing content)
- A token-budgeted projection of the observation log
- Optional project-wide instructions from [`_prompts/consolidation.md`](https://github.com/akitaonrails/ai-memory/blob/main/_prompts/consolidation.md)

The helper `build_request` (line 989 in [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs)) assembles these elements into a `ChatRequest` ready for the LLM provider.

### Stage 4: Structured LLM Completion

The system calls `ai_memory_llm::LlmProvider::complete_structured_with_operation_id`, requesting a JSON object conforming to the `ConsolidatedPage` schema defined in [`crates/ai-memory-consolidate/src/types.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/types.rs). The LLM returns structured data including the proposed markdown body, title, and metadata tier.

### Stage 5: Atomic Page Write and Git Commit

Final persistence happens through several coordinated steps:

1. **Front-matter assembly** via `build_frontmatter`, injecting fields like `title`, `tier`, `origin`, and `consolidated: true`
2. **Atomic write** via `Wiki::write_page` (line 5091 in [`wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/wiki.rs)), which creates a supersession chain in the database
3. **Version control** via `Wiki::commit_all`, generating a commit message following the pattern `consolidate(session <X>): <title>` and auto-committing to the Git repository

## Multi-Page Batch Consolidation

When `multi_page` mode is requested, the pipeline uses `consolidate_session_multi` instead of `consolidate_session`. The process diverges at the prompt stage:

- `build_batch_request_with_slots` (line 2774 in [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs)) constructs a request for a `ConsolidatedBatch` output rather than a single page
- The LLM returns multiple page updates in one response
- `build_update` transforms each batch item into a `WritePageRequest`, respecting **slot namespacing**, **rule routing**, and **invariant-slot protection**
- `Wiki::apply_batch` executes all writes within a single SQL transaction
- A single Git commit captures the entire batch

## Key Architectural Components

- **`Consolidator`** (in [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs)): The main orchestrator exposing `consolidate_session` and `consolidate_session_multi`. It holds handles to the store, wiki, writer, and LLM provider.

- **`ReaderPool`** (in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs)): Handles data retrieval via `observations_for_session` and `session_scope_from_observations`.

- **`Wiki`** (in [`wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/wiki.rs)): Manages atomic markdown writes, admission checks, and version control through `preflight_admission`, `write_page`, `apply_batch`, and `commit_all`.

- **LLM Provider** (in `ai-memory-llm`): Generates structured consolidation output via `complete_structured_with_operation_id`.

- **MCP Endpoint** (in [`server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/server.rs) at line 2583): Exposes the `memory_consolidate` tool that external callers use to trigger the pipeline.

## Implementation Examples

### Triggering Consolidation via MCP

```rust
// Example using the generated client interface
let client = ai_memory_mcp::Client::new("http://localhost:49374");
let outcome = client.memory_consolidate(
    SessionId::from_str("a1b2c3d4")?,
    /*dry=*/ false,
    /*multi_page=*/ false,
    /*instructions=*/ None,
).await?;
println!("Consolidated page at {}", outcome.path);

```

### Direct Consolidator Usage

```rust
use ai_memory_consolidate::Consolidator;
use ai_memory_store::{ReaderPool, WriterHandle};
use ai_memory_wiki::Wiki;
use ai_memory_llm::openai::OpenAiProvider;
use std::sync::Arc;

let llm = Arc::new(OpenAiProvider::new("gpt-4")?);
let consolidator = Consolidator::new(reader, writer, wiki, llm, ws_id, proj_id);

let outcome = consolidator
    .consolidate_session(
        session_id,
        /*dry_run=*/ false,
        actor_context,
        Some(author_user_id),
        None, // no per-call instructions
    )
    .await?;
println!("Page written with ID {}", outcome.page_id.unwrap());

```

### Batch Multi-Page Consolidation

```rust
let outcomes = consolidator
    .consolidate_session_multi(
        session_id,
        false,
        actor_context,
        Some(author_user_id),
        None,
    )
    .await?;
for o in outcomes {
    println!("Updated {} (page id {:?})", o.path, o.page_id);
}

```

## Summary

- ai-memory uses a **five-stage pipeline** (gather, validate, prompt, generate, commit) to turn session observations into wiki pages.
- **Preflight admission** checks in [`wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/wiki.rs) prevent unauthorized LLM usage before expensive calls are made.
- The system supports both **single-page** (`ConsolidatedPage`) and **multi-page** (`ConsolidatedBatch`) consolidation modes.
- All writes are **atomic**—either database transactions via `apply_batch` or Git commits via `commit_all`—ensuring data integrity.
- Prompts follow a **Karpathy-style** format and can include custom instructions from project configuration files.

## Frequently Asked Questions

### What triggers the consolidation pipeline in ai-memory?

The pipeline is triggered by the `memory_consolidate` MCP tool handler (found at line 2583 in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs)). External AI agents or scripts invoke this tool via the MCP protocol, passing a `SessionId` and optional flags like `multi_page` or `dry_run` to control behavior.

### How does the system prevent unauthorized wiki modifications during consolidation?

Before constructing LLM prompts, `Wiki::preflight_admission` validates that the requesting actor has write permissions for the target page. This check occurs at line 2234 in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) and prevents costly LLM operations when the actor lacks appropriate scope or access rights.

### What distinguishes single-page from multi-page consolidation?

Single-page consolidation uses `consolidate_session` and expects the LLM to return a single `ConsolidatedPage` object. Multi-page consolidation uses `consolidate_session_multi` with `build_batch_request_with_slots` to request a `ConsolidatedBatch`, allowing the LLM to propose updates across multiple wiki slots in one operation. The batch mode respects slot namespacing and invariant-slot protection rules during the `apply_batch` transaction.

### How does ai-memory ensure data integrity during batch operations?

Batch consolidation wraps all database writes in a single SQL transaction via `Wiki::apply_batch`. If any individual page write fails (due to invariant violations or rule routing conflicts), the entire transaction rolls back. After successful database persistence, `Wiki::commit_all` creates a single Git commit encompassing all modified files, maintaining consistency between the SQL store and the version-controlled markdown repository.