Data Flow from Lifecycle Hooks to Wiki Page Compilation in ai-memory

The ai-memory system transforms AI agent lifecycle events into sanitized wiki documentation through a five-stage pipeline: shell hooks POST to a Rust router, a single-writer SQLite actor persists observations, and a wiki compiler renders atomic HTML pages.

The akitaonrails/ai-memory repository provides a memory layer for AI coding agents that automatically captures session events. This article traces the complete data flow from lifecycle hooks to wiki page compilation, demonstrating how the architecture enforces sanitization at every boundary.

Stage 1: Hook Ingestion and Payload Sanitization

When an AI coding agent (e.g., OpenCode, Claude-Code) starts or executes tools, shell scripts in the hooks/ directory trigger HTTP requests to the MCP server.

From Shell Scripts to Sanitized Observations

Agent-side hooks located under hooks/…/*.sh invoke curl to POST JSON payloads to the MCP endpoint at /hook. The request includes session metadata and event types.

# hooks/opencode/session-start.sh

curl -X POST "$AI_MEMORY_ENDPOINT/hook" \
  -H "Content-Type: application/json" \
  -d '{
        "session_id": "abcd1234",
        "event": "session-start",
        "payload": {"client":"opencode","version":"1.2.3"}
      }'

The MCP server creates a Router instance defined in crates/ai-memory-hooks/src/router.rs. Inside the router, raw payloads undergo immediate transformation into Sanitized observations using the built-in Sanitizer from crates/ai-memory-core/src/sanitizer.rs.

// crates/ai-memory-hooks/src/router.rs
let raw_obs = RawObservation::try_from(body)?;
let sanitized = Sanitized::new(raw_obs, &state.sanitizer);
store.write_observation(sanitized).await?;

The Sanitizer::scrub method removes secrets, control characters, and enforces length limits before any storage operation occurs.

Stage 2: Single-Writer SQLite Storage

The system maintains data consistency through a strict single-writer constraint that prevents race conditions during concurrent hook invocations.

The SQLite Writer Actor

Sanitized observations flow to the single-writer SQLite actor located in crates/ai-memory-store/src/writer.rs. This component serializes all writes to prevent race conditions.

The writer inserts rows into the observations table using a composite key of (workspace_id, project_id, path). This design satisfies the architecture's invariant that all writes go through a single writer, as documented in docs/ARCHITECTURE.md.

Reading Sanitized Observations

The Wiki subsystem reads recent observations through the reader pool implemented in crates/ai-memory-store/src/reader.rs. According to the source comments around line 708, the reader returns already-scrubbed observation bodies, ensuring that downstream components never process raw user input.

Stage 3: Wiki Rendering and Atomic Compilation

The final stage converts database rows into immutable HTML documentation through safe Markdown processing and atomic file operations.

Markdown Processing and HTML Sanitization

For each observation, the Wiki engine in crates/ai-memory-wiki/src/wiki.rs renders Markdown using the pulldown-cmark crate. During parsing, the renderer applies sanitize_event to strip unsafe HTML.

The HTML sanitization logic resides in crates/ai-memory-web/src/markdown.rs (lines 40-240). This ensures no stray markup or executable content appears in the final page.

// crates/ai-memory-wiki/src/wiki.rs
let markdown = self.sanitizer.scrub(&detail.body_markdown);
let parser = Parser::new_ext(&markdown, opts).map(sanitize_event);
let mut html = String::new();
html::push_html(&mut html, parser);
self.atomic_write(&path, &html)?;

Atomic File System Writes

The wiki compiler writes content using an atomic pattern: create temporary file, rename to target, then fsync. This prevents partially-written pages from being served to readers. The atomic_write method guarantees that wiki pages appear fully formed or not at all.

Complete Data Flow Architecture

The end-to-end pipeline follows this strict sequence:

1. Hook Execution: Shell scripts in hooks/*.sh POST to /hook
2. Router Sanitization: crates/ai-memory-hooks/src/router.rs creates Sanitized instances via Sanitized::new
3. Database Write: crates/ai-memory-store/src/writer.rs commits to SQLite with serialized access
4. Data Retrieval: crates/ai-memory-store/src/reader.rs returns sanitized rows (line 708)
5. Page Compilation: crates/ai-memory-wiki/src/wiki.rs renders Markdown and sanitizes HTML via crates/ai-memory-web/src/markdown.rs
6. Atomic Persistence: Files are written using tmp + rename + fsync patterns

This architecture ensures that secrets never persist and wiki pages remain immutable once written, as detailed in docs/lifecycle-ops.md.

Summary

• Sanitization occurs at ingestion: The Router in crates/ai-memory-hooks/src/router.rs scrubs all fields using Sanitized::new before database contact.
• Single-writer guarantee: All SQLite writes serialize through crates/ai-memory-store/src/writer.rs, keyed by (workspace_id, project_id, path).
• Defensive reading: The reader pool in crates/ai-memory-store/src/reader.rs (line 708) returns only pre-sanitized bodies.
• Safe rendering: crates/ai-memory-wiki/src/wiki.rs combines pulldown-cmark with HTML sanitization from crates/ai-memory-web/src/markdown.rs (lines 40-240).
• Atomic durability: Wiki pages are published via temporary file creation and rename operations, preventing corruption.

Frequently Asked Questions

How does ai-memory prevent secrets from leaking into wiki pages?

The system applies a typed sanitizer at the store boundary. When crates/ai-memory-hooks/src/router.rs receives a payload, it immediately constructs a Sanitized observation using the sanitizer defined in crates/ai-memory-core/src/sanitizer.rs. This scrubs secrets, control characters, and enforces length limits before the data reaches SQLite. The reader pool later returns these already-sanitized bodies, ensuring downstream renderers never handle raw secrets.

Why does the storage layer use a single-writer SQLite actor?

The single-writer design in crates/ai-memory-store/src/writer.rs eliminates write conflicts and maintains the invariant that observations are recorded sequentially. By forcing all database mutations through one actor, the system prevents race conditions when multiple hooks fire simultaneously, ensuring the (workspace_id, project_id, path) keyspace remains consistent.

Which component handles the Markdown-to-HTML conversion?

The Wiki engine in crates/ai-memory-wiki/src/wiki.rs drives the conversion using the pulldown-cmark parser. However, the critical safety logic resides in crates/ai-memory-web/src/markdown.rs (lines 40-240), where the sanitize_event function strips unsafe HTML tags during the parse stream. This dual-layer approach converts Markdown to safe HTML before atomic writing.

What guarantees atomicity when writing wiki files?

The atomic_write method in crates/ai-memory-wiki/src/wiki.rs implements a write-to-temp-then-rename pattern. It writes HTML content to a temporary file, calls fsync to flush to disk, then renames the temporary file to the target path. This ensures readers never encounter partially-written pages, even if the process crashes mid-write.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}