# ai-memory Data Flow Explained: 9 Core Components and Architecture

> Explore the ai-memory data flow and its 9 core components. Understand the dual-layer architecture leveraging markdown and SQLite for efficient retrieval and semantic search.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: internals
- Published: 2026-09-10

---

**The ai-memory data flow operates on a dual-layer architecture where markdown wiki files serve as the immutable source of truth, while a derived SQLite index with FTS5 enables fast retrieval and semantic search.**

The ai-memory system by akitaonrails implements a robust Rust-based pipeline for capturing, enriching, and retrieving AI agent interactions. Understanding the ai-memory data flow reveals how raw agent events transform into searchable knowledge through atomic writes, git-based versioning, and optional LLM consolidation.

## Dual-Layer Architecture Foundation

At the heart of ai-memory lies a **dual-layer data model** designed for durability and rebuildability. The **markdown wiki** (`data_dir/wiki/`) acts as the authoritative source of truth, storing human-readable pages with full git history. Separately, a **SQLite database** (`data_dir/db/memory.sqlite`) operates as a derived index powering fast retrieval via FTS5, entity matching, and vector similarity.

This separation ensures that index corruption can always be repaired by re-indexing the wiki, and the system can function in zero-LLM mode without external dependencies.

## The 9 Stages of the ai-memory Data Flow

### 1. Agent Lifecycle Hook Emission

CLI-based agents (Codex, Claude, and others) emit JSON events including `SessionStart`, `UserPrompt`, and `SessionEnd`. These events reach the system via `curl POST /hook` or through native `ai-memory hook` commands that spool events locally before handing them off to the server.

- **Core implementation**: [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs)
- **Event types**: `SessionStart`, `UserPrompt`, `SessionEnd`

### 2. Hook Router and Sanitisation

The server-side router in [`payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/payload.rs) receives payloads and runs them through the **sanitiser**—the only path from untrusted text into the store. The router classifies sanitized content as an `ObservationKind` before forwarding it downstream.

- **Security boundary**: The sanitiser is the sole entry point for external text
- **Core implementation**: [`crates/ai-memory-hooks/src/payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/payload.rs)

### 3. Writer Actor Persistence

A **single-writer SQLite actor** receives `WriteCmd` messages and persists observations to the `observations` table. This actor also maintains the `pages` table when wiki content is created or superseded, ensuring serialised writes to prevent database contention.

- **Concurrency model**: Single-writer actor pattern for SQLite
- **Core implementation**: [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs)

### 4. Wiki Layer (Source of Truth)

The **markdown wiki** stores durable, human-readable pages. The `ai-memory-wiki` crate performs atomic writes using a tmp-rename-fsync pattern and records each change as a git commit, guaranteeing durable history and enabling rollbacks.

- **Atomic write pattern**: tmp file → rename → fsync
- **Versioning**: Automatic git commits for every change
- **Core implementation**: [`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs)

### 5. Derived SQLite Index

After wiki writes, database triggers populate FTS5 (`pages_fts`) and auxiliary tables including `links`, `entities`, and `page_embeddings`. This derived index enables **FTS5 + entity-match + link-neighbour RRF** (Reciprocal Rank Fusion) queries and optional vector similarity when an embedder is configured.

- **Search capabilities**: Full-text search, entity extraction, link analysis
- **Core implementation**: [`crates/ai-memory-store/src/fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/fts_query.rs)

### 6. Consolidation Pipeline

When `AI_MEMORY_LLM_PROVIDER` is configured, the **consolidation pipeline** reads raw session summary pages, prompts the LLM to rewrite and enrich them, and writes the refined pages back through the wiki writer. This stage also generates derived concept and decision pages.

- **Trigger**: Environment variable `AI_MEMORY_LLM_PROVIDER`
- **Output**: Enriched pages, concept extractions, decision logs
- **Core implementation**: [`crates/ai-memory-consolidate/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/lib.rs)

### 7. Auto-Improvement Scheduler

A background scheduler (`auto_improve`) scans completed sessions and generates proposals for concepts, decisions, gotchas, and best practices via the LLM. Proposals stage in the `auto_improve_proposals` table and auto-approve by default, writing back through the wiki path to continuously refine the knowledge base.

- **Process**: Scan → Generate proposal → Stage → Auto-approve → Write
- **Core implementation**: [`crates/ai-memory-store/src/auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/auto_improve.rs)

### 8. Query and Feedback

Clients invoke `memory_query` (or `memory_read_page`) which combines the SQLite index, entity matches, and optional vector reranking to return ranked results. Page-level feedback via `memory_feedback` updates `page_feedback` records and adjusts salience scores used by decay-based retention algorithms.

- **Query methods**: FTS5 text search, entity matching, vector similarity
- **Feedback loop**: User feedback adjusts retention salience
- **Core implementations**: [`crates/ai-memory-store/src/fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/fts_query.rs) and [`crates/ai-memory-store/src/ops.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/ops.rs)

### 9. Maintenance and Decay

Periodic **forget-sweep** operations run against both wiki and SQLite layers, applying decay calculations and tombstone cleanup. Cold pages are evicted based on salience scores while pinned content remains preserved, preventing unbounded growth of the memory store.

- **Operations**: Decay calculation, tombstone cleanup, eviction
- **Core implementation**: [`crates/ai-memory-store/src/maintenance.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/maintenance.rs)

## Closed-Loop System Design

These nine stages form a continuous cycle:

1. **Capture**: Agents emit events → sanitiser → writer
2. **Persist**: Writer updates SQLite and triggers wiki writes
3. **Index**: SQLite/FTS5 tables synchronize with wiki content
4. **Enrich**: Optional LLM consolidation and auto-improvement convert raw observations into semantic knowledge
5. **Serve**: Queries read from the index; feedback drives salience adjustments; maintenance prunes stale data

## Practical Interaction Examples

### Emit a Lifecycle Hook

```rust
// Running from a CLI agent
std::process::Command::new("ai-memory")
    .arg("hook")
    .arg("--event")
    .arg("session-start")
    .arg("--payload")
    .arg(r#"{"session_id":"abc123","cwd":"/path/to/project"}"#)
    .status()
    .expect("hook failed");

```

This spools JSON locally and posts it to `/hook`, where the router sanitises the payload and enqueues a `WriteCmd`.

### Query the Knowledge Base

```bash

# Retrieve top 5 results for a query

ai-memory query --query "how to reset a project" --top_k 5

```

The `memory_query` tool combines FTS5, entity-match, and vector similarity (if configured) to return page paths and snippets.

### Write Explicit Knowledge

```bash
ai-memory write-page \
    --path "notes/quick-start.md" \
    --content "# Quick Start\nRun `ai-memory init` then `ai-memory status`." \

    --expires_at "2027-01-01"

```

The request reaches the wiki writer, which atomically creates the markdown file, commits to git, and updates the SQLite `pages` table.

### Trigger Manual Auto-Improvement

```bash
ai-memory auto-improve --session-id "$(ai-memory status --last-session-id)"

```

This pulls the session, runs the LLM consolidation pipeline, stages proposals in `auto_improve_proposals`, and writes approved pages back through the wiki path.

## Summary

- **ai-memory** implements a **dual-layer architecture** with markdown as source of truth and SQLite as a rebuildable index.
- **Nine distinct stages** handle everything from hook emission and sanitisation to LLM consolidation and maintenance decay.
- **Atomic writes** via [`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs) ensure durability through tmp-rename-fsync patterns and git commits.
- **Single-writer SQLite actor** in [`writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/writer.rs) prevents concurrency issues while maintaining high throughput.
- **Zero-LLM default** allows operation without external providers, with optional enrichment via `AI_MEMORY_LLM_PROVIDER`.

## Frequently Asked Questions

### What makes the markdown wiki the "source of truth" in ai-memory?

The markdown wiki in `data_dir/wiki/` serves as the authoritative store because every write operation in [`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs) performs atomic file operations followed by git commits before any database updates occur. If the SQLite index becomes corrupted or desynchronized, the system can rebuild it entirely by re-indexing the markdown files, ensuring the human-readable content always remains intact and authoritative.

### How does ai-memory handle data sanitisation and security?

All untrusted text enters through the **sanitiser** in [`crates/ai-memory-hooks/src/payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/payload.rs), which is the sole security boundary between external agents and the storage layer. The sanitiser validates and classifies incoming payloads into `ObservationKind` types before the writer actor processes them, preventing injection attacks and malformed data from reaching the SQLite store or wiki files.

### Can ai-memory function without an LLM provider?

Yes. The system operates in **zero-LLM mode** by default, storing raw observations and manual wiki entries without requiring `AI_MEMORY_LLM_PROVIDER`. LLM features—including the consolidation pipeline in [`crates/ai-memory-consolidate/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/lib.rs) and auto-improvement in [`auto_improve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/auto_improve.rs)—are strictly optional enhancements that enrich raw data into structured knowledge when configured.

### What triggers the maintenance and decay processes?

The **forget-sweep** and decay calculations in [`crates/ai-memory-store/src/maintenance.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/maintenance.rs) run periodically against both storage layers, evaluating page salience scores adjusted by user feedback from `page_feedback` records. Cold content below threshold salience is evicted during tombstone cleanup, while pinned or frequently accessed pages survive the decay process, ensuring the memory store remains bounded and relevant.