How the Session Consolidation Process Compiles Observations into Wiki Pages in ai-memory

The session consolidation process in ai-memory transforms transient agent observations into durable wiki pages through a three-stage pipeline: capture via lifecycle hooks, rule-based session summarization, and LLM-driven consolidation into semantic knowledge families.

The session consolidation process is the core mechanism by which the akitaonrails/ai-memory repository turns fleeting coding session data into permanent, searchable knowledge. Every agent interaction feeds a steady-state loop that culminates in structured markdown output under the project's wiki directory. This pipeline lets operators maintain an accurate, versioned knowledge base without manual documentation overhead.

Three Stages of the Session Consolidation Process

Capture via Lifecycle Hooks

Agent CLIs emit short-lived HTTP POST requests to the /hook endpoint throughout a coding session. Each payload carries sanitized observations such as session-start, user-prompt, post-tool-use, and session-end. The server's hook router sanitizes the incoming payload, assigns an ObservationKind, and pushes a WriteCmd to the single-writer SQLite actor for durable storage.

This steady-state loop is documented in docs/ARCHITECTURE.md, which defines how raw observations enter the system before any summarization occurs.

Rule-Based Session Summary Generation

When the server processes a session-end event, it synthesizes a canonical session record at sessions/<id>.md. This summary page is generated entirely from stored observations using a deterministic, rule-based template. No LLM participates in this stage; the output is predictable and reproducible from the SQLite observation log.

The trigger for this step resides in hooks/opencode/session-end.sh. The resulting markdown file becomes the canonical record of the session and seeds the context for subsequent hand-offs.

LLM-Driven Consolidation

If an LLM provider is configured via the AI_MEMORY_LLM_PROVIDER environment variable, the system invokes the memory_consolidate tool automatically or on demand. The Consolidator struct, defined in crates/ai-memory-consolidate/src/lib.rs, reads the session summary, assembles a prompt containing the source material and related wiki pages, and delegates rewriting to the configured LLM.

The consolidation prompt uses ProjectNameStrategy and enforces chunk budgeting to stay within token limits. Output is written to semantic families under the wiki directory, such as concepts/, decisions/, and gotchas/, with an optional consolidated page placed in the original session location.

The MCP server registers this tool in crates/ai-memory-mcp/src/routes/api.rs, which also enforces the LLM-provider gate before allowing execution.

End-to-End Data Flow

The complete session consolidation process follows a strict sequence:

1. The agent CLI fires /hook requests until the session ends.
2. The session-end handler writes a rule-based summary to sessions/<id>.md.
3. If AI_MEMORY_LLM_PROVIDER is set, the memory_consolidate tool rewrites the content into richer wiki pages.
4. Every consolidation pass creates a new Git commit in <data_dir>/wiki/, making the compiled knowledge immutable and versioned.
5. SQLite triggers keep the search index synchronized with the latest wiki state.

Because the wiki directory serves as the single source of truth, previous versions remain reachable through Git history while the active index always reflects the newest consolidation output.

Running Consolidation via CLI and MCP

You can interact with the session consolidation process through the CLI, the MCP tool surface, or programmatically in Rust.

Command-Line Invocation

First, ensure the session-end hook has produced a summary page. Then run the LLM-driven consolidation:

# Finalize the session to guarantee the summary exists

ai-memory finalize-session --project myproject

# Run LLM-driven consolidation (requires AI_MEMORY_LLM_PROVIDER)

ai-memory memory_consolidate \
  --project myproject \
  --multi_page=true \
  --max_input_tokens=100000 \
  --max_output_tokens=32000

MCP Tool Call

MCP clients can invoke consolidation directly via the registered tool:

{
  "tool": "memory_consolidate",
  "arguments": {
    "project": "myproject",
    "targets": ["concepts/", "decisions/", "gotchas/"],
    "multi_page": true,
    "max_input_tokens": 100000,
    "max_output_tokens": 32000
  }
}

Programmatic Rust Usage

Internally, the server and CLI delegate to the Consolidator struct exposed by crates/ai-memory-consolidate/src/lib.rs:

use ai_memory_consolidate::{Consolidator, ConsolidatorError};

async fn run_consolidation(db: &Db, project_id: ProjectId) -> Result<(), ConsolidatorError> {
    let consolidator = Consolidator::new(db.clone());
    consolidator
        .consolidate_project(project_id, /*auto_approve=*/ true)
        .await
}

Key Files in the Consolidation Architecture

Several source files define the session consolidation process end to end:

• docs/ARCHITECTURE.md — Documents the steady-state observation loop and the consolidation step.
• hooks/opencode/session-end.sh — Triggers creation of the rule-based sessions/<id>.md summary.
• crates/ai-memory-consolidate/src/lib.rs — Implements the Consolidator struct, prompt construction, chunk budgeting, and page write logic.
• crates/ai-memory-mcp/src/routes/api.rs — Exposes the memory_consolidate tool and enforces the LLM-provider configuration gate.
• crates/ai-memory-cli/src/main.rs — Parses the memory_consolidate subcommand and forwards arguments to the MCP layer.

Summary

• The session consolidation process begins when agent lifecycle hooks POST observations to /hook, which are stored as ObservationKind records via a single-writer SQLite actor.
• A deterministic, rule-based template in hooks/opencode/session-end.sh generates sessions/<id>.md without LLM involvement.
• The memory_consolidate tool, gated by AI_MEMORY_LLM_PROVIDER, uses the Consolidator struct to rewrite session summaries into semantic wiki families.
• Every consolidation pass commits immutable output to <data_dir>/wiki/ and syncs the SQLite index through triggers.

Frequently Asked Questions

What triggers the session consolidation process?

The process starts automatically when an agent CLI sends a session-end observation to the /hook endpoint. This event causes the server to write a rule-based summary page. If an LLM provider is configured, the memory_consolidate tool then triggers the LLM-driven rewrite.

Does session consolidation require an LLM provider?

No. The initial stage that compiles observations into sessions/<id>.md is fully deterministic and requires no LLM. However, the enrichment pass that generates multi-page semantic wiki output under concepts/, decisions/, and gotchas/ requires AI_MEMORY_LLM_PROVIDER to be set.

Where are consolidated wiki pages stored?

All consolidated output lives in <data_dir>/wiki/. Each consolidation pass creates a new Git commit, so the wiki remains versioned and immutable. The SQLite search index stays synchronized with these files via database triggers.

How does ai-memory prevent data loss during consolidation?

The system treats the wiki directory as the single source of truth and uses Git for versioning. Because the Consolidator writes through Git commits, prior states are preserved. Meanwhile, the SQLite actor processes WriteCmd operations sequentially, ensuring the observation log remains consistent even during concurrent hook traffic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}