What Is the Data Flow in the ai-memory System? A 3-Phase Pipeline Explained

The ai-memory system uses a deterministic three-phase pipeline: ingest via lifecycle hooks, persist through a single-writer actor to SQLite and markdown, and serve through read-only query pools with optional LLM auto-improvement.

The ai-memory system, written in Rust and designed for AI agent memory management, treats the markdown wiki as the source of truth while maintaining a SQLite-derived index for fast search. Understanding the data flow within the ai-memory system is essential for developers integrating agents, debugging performance, or extending the codebase. The architecture follows a strict separation between write-heavy ingest paths and read-heavy query paths, with all mutations funneled through a single thread to prevent database contention.

Ingest Phase: From Agent Hook to Write Queue

The ingest phase transforms external agent events into sanitized, durable commands.

Lifecycle Hooks Trigger the Flow

Agents emit events through the MCP (Model Context Protocol) server endpoint. These arrive as HTTP POST requests to /hook, handled by ai_memory_mcp::server::Server in crates/ai-memory-mcp/src/server.rs.

Hooks can be emitted via CLI:

ai-memory hook --event SessionStart \
  --payload '{"session_id":"123","agent_kind":"codex"}'

Or directly via HTTP from shell scripts or other agents.

Router Sanitization: The Trust Boundary

The ai_memory_hooks::router::Router in crates/ai-memory-hooks/src/router.rs performs the only trusted transformation of untrusted input. It converts raw JSON into Sanitized<NewObservation> structs, rejecting malformed payloads before they reach storage.

Write Commands Queue to Single-Writer Actor

The router constructs WriteCmd variants—InsertObservation, UpsertPage, or InsertSession—and sends them via an mpsc channel to WriterHandle in crates/ai-memory-store/src/writer.rs. This channel decouples the async HTTP handler from the synchronous writer thread, providing backpressure and preventing memory spikes under load.

Persist Phase: Single-Writer Guarantees Atomicity

The persist phase enforces sequential consistency for all durable state.

WriterHandle Owns the Sole SQLite Connection

The WriterHandle runs on a dedicated OS thread with exclusive access to the rusqlite::Connection. All mutations—pages, sessions, observations, handoffs, embeddings—execute inside this thread. This design eliminates "database is locked" errors without complex WAL tuning or connection pooling for writes.

Key operations in crates/ai-memory-store/src/writer.rs:

• INSERT or UPSERT entity rows
• Update FTS5 full-text index
• Maintain vector similarity tables (if embeddings enabled)
• Apply batch operations atomically

Wiki Files Update with Atomic Writes

After each successful SQLite transaction, the writer invokes ai_memory_wiki::Wiki methods in crates/ai-memory-wiki/src/wiki.rs:

• Wiki::write_page for single-page updates
• Wiki::apply_batch for bulk operations

File writes use atomic rename patterns: write to temporary file, fsync, then rename to target. This ensures the markdown wiki never contains partial content, preserving the source-of-truth invariant.

Serve Phase: Read-Only Queries and LLM Enrichment

The serve phase provides fast, concurrent access with optional background intelligence.

ReaderPool Handles Concurrent Queries

Read operations use ReaderPool from crates/ai-memory-store/src/lib.rs, which distributes lightweight read-only SQLite connections across threads. The memory_query function combines:

1. FTS5 full-text search for keyword matches
2. Lexical entity matching for structured references
3. Vector similarity (optional) for semantic search

Results are ranked and returned as markdown page references with relevance scores.

Auto-Improvement Scheduler Runs Background Enrichment

When AI_MEMORY_LLM_PROVIDER is configured, the auto-improvement scheduler in crates/ai-memory-consolidate/src/auto_improve_schedule.rs periodically:

1. Scans SessionEnd rows for recently completed sessions
2. Invokes the configured LLM to synthesize insights
3. Writes enriched pages back through the same WriterHandle

This creates a feedback loop: agent observations → storage → LLM analysis → improved documentation → better future retrieval.

Example scheduler usage:

use ai_memory_consolidate::auto_improve_schedule::AutoImproveScheduler;
use ai_memory_store::{ReaderPool, WriterHandle};

async fn run_scheduler(reader: ReaderPool, writer: WriterHandle) {
    let mut scheduler = AutoImproveScheduler::new(reader, writer);
    scheduler.run_once().await.unwrap();
}

End-to-End Data Flow Diagram

Agent Hook ──► /hook (MCP) ──► Router (sanitize) ──► WriteCmd ──► WriterHandle
   │                                                    │
   │                                                    ▼
   │                                           SQLite + Wiki files
   │                                                    ▲
   │                                                    │
   │                                            Auto-Improve Scheduler
   │                                                    │
   ▼                                                    ▼
Query API (memory_query) ◄──── ReaderPool ◄──── SQLite index

Solid lines indicate request-response paths. Dashed lines indicate background maintenance. The core invariant holds throughout: markdown wiki remains the source of truth; SQLite remains the derived, searchable index.

Key Source Files for Data Flow Analysis

Component	File Path	Purpose
MCP server entry	crates/ai-memory-mcp/src/server.rs	HTTP /hook handler
Request sanitization	crates/ai-memory-hooks/src/router.rs	Trust boundary for incoming data
Write serialization	crates/ai-memory-store/src/writer.rs	Single-writer actor, SQLite + wiki persistence
Markdown operations	crates/ai-memory-wiki/src/wiki.rs	Atomic file writes, index coordination
Read query engine	crates/ai-memory-store/src/lib.rs	memory_query, ReaderPool
LLM enrichment	crates/ai-memory-consolidate/src/auto_improve_schedule.rs	Background improvement scheduler
Architecture docs	docs/ARCHITECTURE.md	High-level design and diagrams

Summary

• Ingest: Agents send hooks → MCP server → Router sanitization → mpsc channel → WriterHandle queue
• Persist: Single-writer thread executes SQLite transactions, then atomically updates markdown wiki files
• Serve: ReaderPool enables concurrent FTS5 + vector queries; auto-improvement scheduler adds LLM-driven enrichment
• Invariant: Markdown wiki is source of truth; SQLite is derived index for search performance
• Safety: All mutations serialize through one thread; file writes are atomic; sanitization occurs at system boundary

Frequently Asked Questions

How does ai-memory prevent database corruption with concurrent writes?

All mutations route through WriterHandle in crates/ai-memory-store/src/writer.rs, which owns the sole rusqlite::Connection on a dedicated OS thread. This single-writer pattern eliminates write contention and "database is locked" errors without requiring WAL mode or complex locking schemes.

Can multiple agents write to ai-memory simultaneously?

Yes. The /hook endpoint in crates/ai-memory-mcp/src/server.rs is async and accepts concurrent requests. These are converted to WriteCmd messages and queued via an mpsc channel. The WriterHandle processes commands sequentially, so agents experience backpressure rather than lock failures.

What happens if the wiki files and SQLite index become out of sync?

The code in crates/ai-memory-wiki/src/wiki.rs performs atomic file updates (temp + rename + fsync) only after successful SQLite transactions. If either fails, the operation aborts. The wiki serves as the authoritative source; SQLite can be rebuilt from markdown if corruption occurs.

How do I query ai-memory programmatically?

Use the memory_query function from crates/ai-memory-store/src/lib.rs with a ReaderPool:

use ai_memory_store::{ReaderPool, memory_query};

let pool = ReaderPool::new("./memory.db")?;
let results = memory_query(&pool, "authentication token")?;

Results combine FTS5 text matches, entity references, and optional vector similarity rankings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}