# How ai-memory Compiles Session Summaries into Markdown Wiki Pages: A Deep Dive into the Consolidation Pipeline > Discover how ai-memory compiles session summaries into markdown wiki pages using a three stage pipeline that gathers observations, constructs LLM prompts and writes structured git versioned markdown. - Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory) - Tags: deep-dive - Published: 2026-08-31 --- **ai-memory compiles session summaries into markdown wiki pages through a three-stage pipeline that gathers SQLite observations, constructs bounded LLM prompts with budget constraints, and atomically writes structured Markdown output with automatic git versioning.** The `akitaonrails/ai-memory` project transforms ephemeral AI session data into persistent, queryable knowledge using a sophisticated consolidation architecture. When you compile session summaries into markdown wiki pages, the system orchestrates data retrieval, intelligent truncation, and atomic storage operations to ensure reliable documentation of every interaction. ## The Three-Stage Consolidation Pipeline ### Stage 1: Gathering Observations from SQLite The consolidation process begins in [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs) by fetching all raw observations associated with a specific session. The `consolidate_session` method initializes the pipeline by querying the `ReaderPool` for chronological data: ```rust // Line 38 in consolidator.rs let observations = self.reader.observations_for_session(session_id).await?; ``` This iterator returns the complete chronological log of `Observation` rows from the SQLite store, establishing the factual foundation that the LLM will synthesize into prose. ### Stage 2: Building Bounded LLM Prompts To prevent context window overflow, the `build_request` function (lines 65-99) assembles a carefully bounded prompt using **PromptBudgets**. The system constructs a `ChatRequest` containing: - A fixed system message establishing the summarization task - A session-ID header identifying the source data - The observation dump filtered by `MAX_PROJECTED_OBSERVATIONS` - The current page body (if updating existing content), truncated to ≤ 20,000 characters - Optional user instructions for customization ```rust // Conceptual flow from build_request let prefix = format!("Session id {}...\nObservations (in order):", session_id); let budget = PromptBudgets::new(20_000); // Char limit for existing body // ... assembly logic ... let request = ChatRequest::new(messages); ``` This bounded approach ensures the LLM receives sufficient context without exceeding token limits, as implemented in the `ai-memory-consolidate` crate. ### Stage 3: Structured Generation and Atomic Storage The pipeline culminates in structured generation and durable storage. First, `complete_structured` (line 90) sends the prompt to the configured `LlmProvider`, expecting a JSON `ConsolidatedPage` response containing `title`, `body_markdown`, and `tags`. Then, `Wiki::write_page` in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) (lines 33-100) handles the atomic commit: ```rust // Key operations in write_page self.sanitizer.scrub(&body); // Remove secrets let markdown = emit(frontmatter, body); // Build final text replace_file_with_rollback_snapshot(path, markdown)?; // Atomic write self.writer.upsert_page(metadata).await?; // Update SQLite index self.wiki.commit_all("consolidate(session ...)"); // Git checkpoint ``` The method stamps front-matter with `last_modified_by` from the `ActorContext`, runs admission webhooks, and creates a git checkpoint before returning a `ConsolidationOutcome` containing the `PageId` and file path. ## Practical Usage Examples ### Command-Line Consolidation Test the compilation process without writing to disk using the dry-run flag: ```bash ai-memory consolidate --dry-run ``` ### Programmatic Session Consolidation Invoke the consolidator directly from Rust to compile session summaries into markdown wiki pages: ```rust use ai_memory_consolidate::Consolidator; use ai_memory_core::{SessionId, ActorContext}; let consolidator = Consolidator::new( reader_pool.clone(), writer_handle.clone(), llm_provider.clone(), workspace_id, project_id, ); let outcome = consolidator .consolidate_session( SessionId::new(), // Target session false, // dry_run = false ActorContext::anonymous(), None, // Optional author ID None, // Optional extra instructions ) .await?; println!("Created page {} with title {}", outcome.path, outcome.new_title); ``` ### Direct Wiki Page Creation For custom summary generation, use `Wiki::write_page` directly: ```rust use ai_memory_wiki::{Wiki, WritePageRequest}; use ai_memory_core::{WorkspaceId, ProjectId, PagePath, Tier}; let wiki = Wiki::new(&data_dir, writer_handle)?; let req = WritePageRequest { workspace_id: WorkspaceId::new(), project_id: ProjectId::new(), path: PagePath::new("sessions/abcd1234.md")?, frontmatter: serde_json::json!({ "title": "My Session" }), body: "## Summary\n…".to_string(), tier: Tier::Episodic, pinned: false, title: None, admission_ctx: None, author_id: None, actor: ai_memory_core::ActorContext::anonymous(), }; let page_id = wiki.write_page(req).await?; ``` ## Summary - **Observation Gathering**: The system queries `observations_for_session` in [`consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/consolidator.rs) to retrieve chronological session data from SQLite. - **Budget-Constrained Prompting**: `build_request` enforces `PromptBudgets` with a 20,000-character limit and `MAX_PROJECTED_OBSERVATIONS` to prevent context overflow. - **Structured Output**: The LLM returns a `ConsolidatedPage` JSON structure with `title`, `body_markdown`, and metadata fields. - **Atomic Persistence**: `Wiki::write_page` performs atomic file writes with rollback snapshots, secret scrubbing, front-matter injection, and automatic git commits. - **Type-Safe Results**: The pipeline returns a `ConsolidationOutcome` containing the `PageId`, file path, and generated title for downstream reference. ## Frequently Asked Questions ### How does ai-memory prevent token overflow when summarizing large sessions? The `build_request` function in [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs) implements **PromptBudgets** to enforce hard limits. It truncates existing page bodies to 20,000 characters and projects safe observation counts using `MAX_PROJECTED_OBSERVATIONS`, ensuring the final prompt fits within the LLM's context window before calling `complete_structured`. ### What happens if the wiki write operation fails mid-process? The `write_page` method in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) uses `replace_file_with_rollback_snapshot` for atomic file system operations. If any step fails—whether during content sanitization, SQLite indexing, or webhook execution—the system rolls back to the previous state without corrupting the wiki or leaving partial writes. ### Can I customize the consolidation prompt for specific domains? Yes. The `consolidate_session` method accepts an optional `instructions` parameter (string) that gets appended to the prompt in `build_request`. This allows you to inject domain-specific guidance, formatting requirements, or stylistic constraints without modifying the core source code. ### What file format and metadata structure does the consolidated output use? The system generates standard Markdown files with YAML front-matter. The `ConsolidatedPage` struct produces `title`, `body_markdown`, and `tags` fields, which are serialized into the document header alongside `last_modified_by` timestamps from the `ActorContext`. The files are stored as `sessions/.md` within the wiki hierarchy and tracked via git for version history.