# How Data Flows Through ai-memory: A Complete Architecture Guide

> Understand ai-memory data flow. Explore its two-layer pipeline, lifecycle hooks, SQLite serialization, LLM consolidation, and immutable markdown wiki source of truth.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: architecture
- Published: 2026-09-09

---

**Data flows through ai-memory via a strict two-layer pipeline where lifecycle hooks are sanitized, serialized by a single-writer actor to SQLite, and optionally consolidated by LLMs, while the markdown wiki remains the immutable source of truth.**

The ai-memory repository (`akitaonrails/ai-memory`) implements a memory layer for AI agents using a dual-layer storage architecture. Understanding how data flows through ai-memory enables developers to integrate robust observability, as the system guarantees atomic transactions, bounded write paths, and high-performance read operations through a derived SQLite index.

## The Two-Layer Architecture

The foundation of the data flow rests on two distinct layers. The **markdown-based wiki** serves as the canonical source of truth, storing human-readable session summaries and knowledge pages. All mutations ultimately persist to disk as atomic markdown files. The **SQLite index** operates as a derived, query-optimized layer that powers fast search, analytics, and tool use, but it never holds authoritative state. As documented in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) (lines 23-33), this separation ensures that even if the index is rebuilt or corrupted, the wiki remains intact and recoverable.

## The 8-Stage Data Flow Pipeline

The complete data flow follows a linear progression from event emission to retention management. Each stage enforces specific guarantees around data integrity and access patterns.

### Stage 1: Lifecycle Hook Emission

The pipeline initiates when agent CLIs or native `ai-memory hook` commands emit lifecycle events. These events include `SessionStart`, `UserPrompt`, `PostToolUse`, and `SessionEnd`. According to the architecture documentation (lines 45-57), clients POST these payloads to the `/hook` endpoint using short-timeout HTTP requests. This design ensures that agent execution never blocks on memory persistence, while the server accepts untrusted input for processing.

### Stage 2: Hook Routing and Sanitization

Incoming hooks enter the **router** defined in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs). Here, the system validates the JSON payload, strips private data fields, and assigns an `ObservationKind` classification. The router then constructs a `WriteCmd` structure and dispatches it to the **single-writer SQLite actor** (lines 61-68, [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md)). This single-channel architecture guarantees that all untrusted text enters the storage layer through one bounded, auditable path.

### Stage 3: Single-Writer Persistence

The writer actor, implemented in [`crates/ai-memory-core/src/actor.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/actor.rs), runs on a dedicated OS thread to prevent blocking the async runtime. Upon receiving a `WriteCmd`, it performs an atomic transaction that:
- Inserts the observation into the `observations` table
- Updates the `pages` table via supersession logic
- Appends a log line to [`log.md`](https://github.com/akitaonrails/ai-memory/blob/main/log.md) for audit trails

This stage (lines 69-74) ensures that the SQLite index and the markdown wiki move forward in lockstep, with the wiki write succeeding before the transaction commits.

### Stage 4: Session-End Synthesis

When the router processes a `SessionEnd` event, it triggers the synthesis subsystem. The server generates a summary page at `sessions/<id>.md` and creates a `handoff` row for the next agent context (lines 75-83). This operation executes as an atomic transaction, guaranteeing that the session is fully captured before any handoff becomes visible to subsequent queries. The implementation resides in [`crates/ai-memory-core/src/actor.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/actor.rs), coordinating between the wiki and store layers.

### Stage 5: Optional LLM Consolidation

If the environment variable `AI_MEMORY_LLM_PROVIDER` is configured, the `memory_consolidate` job activates. Located in [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs), this stage rewrites summary pages or fans out content to multiple thematic pages using the configured LLM (lines 93-98). This step is purely additive; it operates on the already-committed wiki state and produces refined, structured knowledge without blocking the core data flow.

### Stage 6: Auto-Improvement Scheduling

A background scheduler reviews newly completed sessions and generates structural proposals. It suggests new pages under `concepts/`, `decisions/`, or `gotchas/` directories, records these in a pending-writes audit trail, and by default approves them through the standard wiki-write path (lines 99-108). This autonomous maintenance ensures the knowledge base evolves without manual intervention while preserving the same write-guarantees as explicit client requests.

### Stage 7: Query Processing

Read operations follow a separate, lock-free path. The `memory_query` tool, implemented in [`crates/ai-memory-cli/src/commands/query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/query.rs), reads exclusively from the SQLite index. It executes **FTS5** full-text search, **entity-match** filtering, and optional **vector-based RRF** (Reciprocal Rank Fusion) retrieval (lines 117-124). For every result returned, the query processor atomically bumps `access_count` and updates `last_accessed_at` (lines 134-136), providing telemetry for the retention sweeps without interfering with the writer actor.

### Stage 8: Retention and Forget Sweeps

Periodic background tasks enforce storage policies. These sweeps delete expired pages based on TTL, evict cold pages using access-count heuristics, and prune raw observations that have been superseded by consolidated wiki pages (lines 141-162). This final stage preserves the invariant that the wiki remains the canonical source while keeping the derived index lean and performant.

## Key Source Files and Components

The data flow relies on specific implementations across the codebase:

- **[`crates/ai-memory-core/src/actor.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/actor.rs)** – Defines the **writer actor** that owns the exclusive SQLite connection and coordinates all mutations.
- **[`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs)** – Implements hook **routing and sanitization**, converting HTTP payloads into internal `WriteCmd` structures.
- **[`crates/ai-memory-store/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/lib.rs)** – Provides the public API for **reading and writing** observations, pages, and handoffs to the SQLite layer.
- **[`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs)** – Handles atomic **markdown wiki** operations including write, supersede, and link extraction.
- **[`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs)** – Contains the **LLM-driven consolidation** logic for summarization and page generation.
- **[`crates/ai-memory-cli/src/commands/query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/query.rs)** – Implements the **`memory_query`** MCP tool for read-only access.

## Practical Implementation Examples

To interact with the data flow programmatically, use the following patterns from the `akitaonrails/ai-memory` codebase.

Emit a lifecycle hook using the Rust client:

```rust
use ai_memory_hooks::payload::Payload;

let payload = Payload::new("session-start", "my-project", "{}", None);
ai_memory_hooks::router::handle(payload).await?;

```

Execute a read-only query via CLI:

```bash
ai-memory memory_query "how does the search work?" --scopes .

```

Write a page programmatically:

```rust
use ai_memory_cli::commands::write_page;

write_page::run(
    "knowledge/ai-memory-overview.md",
    "# Overview\nAI-memory stores ...",

    false,               // not global
    None,                // no TTL
)?;

```

Trigger the auto-improve scheduler manually:

```bash
ai-memory auto-improve --project my-project

```

## Summary

- **Data flows through ai-memory** via a linear pipeline: `hook → sanitize → writer → wiki + SQLite → optional LLM → auto-improve → query`.
- The **markdown wiki** is the single source of truth; the **SQLite index** is a derived, disposable cache for fast queries.
- All writes pass through a **single-writer actor** on a dedicated thread, guaranteeing atomicity and preventing corruption.
- **Read operations** are lock-free and execute against the SQLite index, never touching raw markdown files directly.
- Optional **LLM consolidation** and **auto-improvement** stages run asynchronously without blocking the core write path.
- **Retention sweeps** maintain storage bounds while preserving wiki integrity.

## Frequently Asked Questions

### What is the source of truth in ai-memory?

The **markdown-based wiki** serves as the immutable source of truth. Every mutation ultimately persists as atomic markdown files on disk, while the SQLite index functions as a derived, query-optimized layer that can be rebuilt from the wiki at any time.

### How does ai-memory ensure thread-safe writes?

All write operations route through a **single-writer SQLite actor** defined in [`crates/ai-memory-core/src/actor.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/actor.rs). This actor runs on a dedicated OS thread and processes `WriteCmd` messages sequentially, eliminating race conditions and ensuring that the wiki and SQLite index remain synchronized.

### Can ai-memory operate without an LLM provider?

Yes. The **LLM consolidation** stage is optional and only activates if the `AI_MEMORY_LLM_PROVIDER` environment variable is set. Without it, the system skips the `memory_consolidate` job and proceeds directly to auto-improvement and query processing, storing raw observations and manual edits without AI-driven summarization.

### How does querying interact with the data flow?

Querying follows a **read-only path** separate from the write pipeline. The `memory_query` tool reads from the SQLite index using FTS5 and vector search, bypassing the writer actor entirely. It updates access metadata (`access_count`, `last_accessed_at`) to inform retention policies but never modifies the underlying markdown files.