# What Is the Data Flow in the ai-memory System? A 3-Phase Pipeline Explained

> Understand the ai-memory data flow with our 3-phase pipeline explanation: ingest hooks, SQLite/markdown persistence, and read-only query pools. Learn more today!

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: architecture
- Published: 2026-09-05

---

**The ai-memory system uses a deterministic three-phase pipeline: ingest via lifecycle hooks, persist through a single-writer actor to SQLite and markdown, and serve through read-only query pools with optional LLM auto-improvement.**

The **ai-memory** system, written in Rust and designed for AI agent memory management, treats the **markdown wiki as the source of truth** while maintaining a **SQLite-derived index for fast search**. Understanding the data flow within the ai-memory system is essential for developers integrating agents, debugging performance, or extending the codebase. The architecture follows a strict separation between write-heavy ingest paths and read-heavy query paths, with all mutations funneled through a single thread to prevent database contention.

## Ingest Phase: From Agent Hook to Write Queue

The ingest phase transforms external agent events into sanitized, durable commands.

### Lifecycle Hooks Trigger the Flow

Agents emit events through the **MCP (Model Context Protocol) server** endpoint. These arrive as HTTP POST requests to `/hook`, handled by `ai_memory_mcp::server::Server` in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs).

Hooks can be emitted via CLI:

```bash
ai-memory hook --event SessionStart \
  --payload '{"session_id":"123","agent_kind":"codex"}'

```

Or directly via HTTP from shell scripts or other agents.

### Router Sanitization: The Trust Boundary

The `ai_memory_hooks::router::Router` in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) performs **the only trusted transformation** of untrusted input. It converts raw JSON into `Sanitized<NewObservation>` structs, rejecting malformed payloads before they reach storage.

### Write Commands Queue to Single-Writer Actor

The router constructs `WriteCmd` variants—`InsertObservation`, `UpsertPage`, or `InsertSession`—and sends them via an **mpsc channel** to `WriterHandle` in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs). This channel decouples the async HTTP handler from the synchronous writer thread, providing backpressure and preventing memory spikes under load.

## Persist Phase: Single-Writer Guarantees Atomicity

The persist phase enforces **sequential consistency** for all durable state.

### WriterHandle Owns the Sole SQLite Connection

The `WriterHandle` runs on a dedicated OS thread with exclusive access to the `rusqlite::Connection`. All mutations—pages, sessions, observations, handoffs, embeddings—execute inside this thread. This design eliminates "database is locked" errors without complex WAL tuning or connection pooling for writes.

Key operations in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs):

- `INSERT` or `UPSERT` entity rows
- Update **FTS5** full-text index
- Maintain vector similarity tables (if embeddings enabled)
- Apply batch operations atomically

### Wiki Files Update with Atomic Writes

After each successful SQLite transaction, the writer invokes `ai_memory_wiki::Wiki` methods in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs):

- `Wiki::write_page` for single-page updates
- `Wiki::apply_batch` for bulk operations

File writes use **atomic rename patterns**: write to temporary file, `fsync`, then rename to target. This ensures the markdown wiki never contains partial content, preserving the **source-of-truth invariant**.

## Serve Phase: Read-Only Queries and LLM Enrichment

The serve phase provides fast, concurrent access with optional background intelligence.

### ReaderPool Handles Concurrent Queries

Read operations use `ReaderPool` from [`crates/ai-memory-store/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/lib.rs), which distributes lightweight **read-only** SQLite connections across threads. The `memory_query` function combines:

1. **FTS5** full-text search for keyword matches
2. **Lexical entity matching** for structured references
3. **Vector similarity** (optional) for semantic search

Results are ranked and returned as markdown page references with relevance scores.

### Auto-Improvement Scheduler Runs Background Enrichment

When `AI_MEMORY_LLM_PROVIDER` is configured, the **auto-improvement scheduler** in [`crates/ai-memory-consolidate/src/auto_improve_schedule.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/auto_improve_schedule.rs) periodically:

1. Scans `SessionEnd` rows for recently completed sessions
2. Invokes the configured LLM to synthesize insights
3. Writes enriched pages back through the same `WriterHandle`

This creates a **feedback loop**: agent observations → storage → LLM analysis → improved documentation → better future retrieval.

Example scheduler usage:

```rust
use ai_memory_consolidate::auto_improve_schedule::AutoImproveScheduler;
use ai_memory_store::{ReaderPool, WriterHandle};

async fn run_scheduler(reader: ReaderPool, writer: WriterHandle) {
    let mut scheduler = AutoImproveScheduler::new(reader, writer);
    scheduler.run_once().await.unwrap();
}

```

## End-to-End Data Flow Diagram

```

Agent Hook ──► /hook (MCP) ──► Router (sanitize) ──► WriteCmd ──► WriterHandle
   │                                                    │
   │                                                    ▼
   │                                           SQLite + Wiki files
   │                                                    ▲
   │                                                    │
   │                                            Auto-Improve Scheduler
   │                                                    │
   ▼                                                    ▼
Query API (memory_query) ◄──── ReaderPool ◄──── SQLite index

```

Solid lines indicate request-response paths. Dashed lines indicate background maintenance. The **core invariant** holds throughout: markdown wiki remains the source of truth; SQLite remains the derived, searchable index.

## Key Source Files for Data Flow Analysis

| Component | File Path | Purpose |
|-----------|-----------|---------|
| MCP server entry | [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) | HTTP `/hook` handler |
| Request sanitization | [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) | Trust boundary for incoming data |
| Write serialization | [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) | Single-writer actor, SQLite + wiki persistence |
| Markdown operations | [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) | Atomic file writes, index coordination |
| Read query engine | [`crates/ai-memory-store/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/lib.rs) | `memory_query`, `ReaderPool` |
| LLM enrichment | [`crates/ai-memory-consolidate/src/auto_improve_schedule.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/auto_improve_schedule.rs) | Background improvement scheduler |
| Architecture docs | [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) | High-level design and diagrams |

## Summary

- **Ingest**: Agents send hooks → MCP server → Router sanitization → mpsc channel → `WriterHandle` queue
- **Persist**: Single-writer thread executes SQLite transactions, then atomically updates markdown wiki files
- **Serve**: `ReaderPool` enables concurrent FTS5 + vector queries; auto-improvement scheduler adds LLM-driven enrichment
- **Invariant**: Markdown wiki is source of truth; SQLite is derived index for search performance
- **Safety**: All mutations serialize through one thread; file writes are atomic; sanitization occurs at system boundary

## Frequently Asked Questions

### How does ai-memory prevent database corruption with concurrent writes?

All mutations route through `WriterHandle` in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs), which owns the sole `rusqlite::Connection` on a dedicated OS thread. This single-writer pattern eliminates write contention and "database is locked" errors without requiring WAL mode or complex locking schemes.

### Can multiple agents write to ai-memory simultaneously?

Yes. The `/hook` endpoint in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) is async and accepts concurrent requests. These are converted to `WriteCmd` messages and queued via an mpsc channel. The `WriterHandle` processes commands sequentially, so agents experience backpressure rather than lock failures.

### What happens if the wiki files and SQLite index become out of sync?

The code in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) performs atomic file updates (temp + rename + fsync) only after successful SQLite transactions. If either fails, the operation aborts. The wiki serves as the authoritative source; SQLite can be rebuilt from markdown if corruption occurs.

### How do I query ai-memory programmatically?

Use the `memory_query` function from [`crates/ai-memory-store/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/lib.rs) with a `ReaderPool`:

```rust
use ai_memory_store::{ReaderPool, memory_query};

let pool = ReaderPool::new("./memory.db")?;
let results = memory_query(&pool, "authentication token")?;

```

Results combine FTS5 text matches, entity references, and optional vector similarity rankings.