# Data Flow from Lifecycle Hooks to Wiki Page Compilation in ai-memory

> Understand the ai-memory data flow from lifecycle hooks to wiki page compilation. See how shell hooks, Rust router, and SQLite actor create atomic HTML pages.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-25

---

**The ai-memory system transforms AI agent lifecycle events into sanitized wiki documentation through a five-stage pipeline: shell hooks POST to a Rust router, a single-writer SQLite actor persists observations, and a wiki compiler renders atomic HTML pages.**

The `akitaonrails/ai-memory` repository provides a memory layer for AI coding agents that automatically captures session events. This article traces the complete data flow from lifecycle hooks to wiki page compilation, demonstrating how the architecture enforces sanitization at every boundary.

## Stage 1: Hook Ingestion and Payload Sanitization

When an AI coding agent (e.g., OpenCode, Claude-Code) starts or executes tools, shell scripts in the `hooks/` directory trigger HTTP requests to the MCP server.

### From Shell Scripts to Sanitized Observations

Agent-side hooks located under `hooks/…/*.sh` invoke `curl` to POST JSON payloads to the MCP endpoint at `/hook`. The request includes session metadata and event types.

```bash

# hooks/opencode/session-start.sh

curl -X POST "$AI_MEMORY_ENDPOINT/hook" \
  -H "Content-Type: application/json" \
  -d '{
        "session_id": "abcd1234",
        "event": "session-start",
        "payload": {"client":"opencode","version":"1.2.3"}
      }'

```

The MCP server creates a `Router` instance defined in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs). Inside the router, raw payloads undergo immediate transformation into `Sanitized` observations using the built-in `Sanitizer` from [`crates/ai-memory-core/src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitizer.rs).

```rust
// crates/ai-memory-hooks/src/router.rs
let raw_obs = RawObservation::try_from(body)?;
let sanitized = Sanitized::new(raw_obs, &state.sanitizer);
store.write_observation(sanitized).await?;

```

The `Sanitizer::scrub` method removes secrets, control characters, and enforces length limits before any storage operation occurs.

## Stage 2: Single-Writer SQLite Storage

The system maintains data consistency through a strict single-writer constraint that prevents race conditions during concurrent hook invocations.

### The SQLite Writer Actor

Sanitized observations flow to the **single-writer SQLite actor** located in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs). This component serializes all writes to prevent race conditions.

The writer inserts rows into the `observations` table using a composite key of `(workspace_id, project_id, path)`. This design satisfies the architecture's invariant that *all writes go through a single writer*, as documented in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md).

### Reading Sanitized Observations

The Wiki subsystem reads recent observations through the **reader pool** implemented in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs). According to the source comments around line 708, the reader returns already-scrubbed observation bodies, ensuring that downstream components never process raw user input.

## Stage 3: Wiki Rendering and Atomic Compilation

The final stage converts database rows into immutable HTML documentation through safe Markdown processing and atomic file operations.

### Markdown Processing and HTML Sanitization

For each observation, the Wiki engine in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) renders Markdown using the `pulldown-cmark` crate. During parsing, the renderer applies `sanitize_event` to strip unsafe HTML.

The HTML sanitization logic resides in [`crates/ai-memory-web/src/markdown.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-web/src/markdown.rs) (lines 40-240). This ensures no stray markup or executable content appears in the final page.

```rust
// crates/ai-memory-wiki/src/wiki.rs
let markdown = self.sanitizer.scrub(&detail.body_markdown);
let parser = Parser::new_ext(&markdown, opts).map(sanitize_event);
let mut html = String::new();
html::push_html(&mut html, parser);
self.atomic_write(&path, &html)?;

```

### Atomic File System Writes

The wiki compiler writes content using an atomic pattern: create temporary file, rename to target, then `fsync`. This prevents partially-written pages from being served to readers. The `atomic_write` method guarantees that wiki pages appear fully formed or not at all.

## Complete Data Flow Architecture

The end-to-end pipeline follows this strict sequence:

1. **Hook Execution**: Shell scripts in `hooks/*.sh` POST to `/hook`
2. **Router Sanitization**: [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) creates `Sanitized` instances via `Sanitized::new`
3. **Database Write**: [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) commits to SQLite with serialized access
4. **Data Retrieval**: [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) returns sanitized rows (line 708)
5. **Page Compilation**: [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) renders Markdown and sanitizes HTML via [`crates/ai-memory-web/src/markdown.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-web/src/markdown.rs)
6. **Atomic Persistence**: Files are written using `tmp + rename + fsync` patterns

This architecture ensures that **secrets never persist** and **wiki pages remain immutable** once written, as detailed in [`docs/lifecycle-ops.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/lifecycle-ops.md).

## Summary

- **Sanitization occurs at ingestion**: The `Router` in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) scrubs all fields using `Sanitized::new` before database contact.
- **Single-writer guarantee**: All SQLite writes serialize through [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs), keyed by `(workspace_id, project_id, path)`.
- **Defensive reading**: The reader pool in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (line 708) returns only pre-sanitized bodies.
- **Safe rendering**: [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) combines `pulldown-cmark` with HTML sanitization from [`crates/ai-memory-web/src/markdown.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-web/src/markdown.rs) (lines 40-240).
- **Atomic durability**: Wiki pages are published via temporary file creation and rename operations, preventing corruption.

## Frequently Asked Questions

### How does ai-memory prevent secrets from leaking into wiki pages?

The system applies a **typed sanitizer at the store boundary**. When [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) receives a payload, it immediately constructs a `Sanitized` observation using the sanitizer defined in [`crates/ai-memory-core/src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitizer.rs). This scrubs secrets, control characters, and enforces length limits before the data reaches SQLite. The reader pool later returns these already-sanitized bodies, ensuring downstream renderers never handle raw secrets.

### Why does the storage layer use a single-writer SQLite actor?

The single-writer design in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) eliminates write conflicts and maintains the invariant that observations are recorded sequentially. By forcing all database mutations through one actor, the system prevents race conditions when multiple hooks fire simultaneously, ensuring the `(workspace_id, project_id, path)` keyspace remains consistent.

### Which component handles the Markdown-to-HTML conversion?

The Wiki engine in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) drives the conversion using the `pulldown-cmark` parser. However, the critical safety logic resides in [`crates/ai-memory-web/src/markdown.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-web/src/markdown.rs) (lines 40-240), where the `sanitize_event` function strips unsafe HTML tags during the parse stream. This dual-layer approach converts Markdown to safe HTML before atomic writing.

### What guarantees atomicity when writing wiki files?

The `atomic_write` method in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) implements a write-to-temp-then-rename pattern. It writes HTML content to a temporary file, calls `fsync` to flush to disk, then renames the temporary file to the target path. This ensures readers never encounter partially-written pages, even if the process crashes mid-write.