ai-memory Data Flow Explained: 9 Core Components and Architecture

The ai-memory data flow operates on a dual-layer architecture where markdown wiki files serve as the immutable source of truth, while a derived SQLite index with FTS5 enables fast retrieval and semantic search.

The ai-memory system by akitaonrails implements a robust Rust-based pipeline for capturing, enriching, and retrieving AI agent interactions. Understanding the ai-memory data flow reveals how raw agent events transform into searchable knowledge through atomic writes, git-based versioning, and optional LLM consolidation.

Dual-Layer Architecture Foundation

At the heart of ai-memory lies a dual-layer data model designed for durability and rebuildability. The markdown wiki (data_dir/wiki/) acts as the authoritative source of truth, storing human-readable pages with full git history. Separately, a SQLite database (data_dir/db/memory.sqlite) operates as a derived index powering fast retrieval via FTS5, entity matching, and vector similarity.

This separation ensures that index corruption can always be repaired by re-indexing the wiki, and the system can function in zero-LLM mode without external dependencies.

The 9 Stages of the ai-memory Data Flow

1. Agent Lifecycle Hook Emission

CLI-based agents (Codex, Claude, and others) emit JSON events including SessionStart, UserPrompt, and SessionEnd. These events reach the system via curl POST /hook or through native ai-memory hook commands that spool events locally before handing them off to the server.

• Core implementation: crates/ai-memory-hooks/src/router.rs
• Event types: SessionStart, UserPrompt, SessionEnd

2. Hook Router and Sanitisation

The server-side router in payload.rs receives payloads and runs them through the sanitiser—the only path from untrusted text into the store. The router classifies sanitized content as an ObservationKind before forwarding it downstream.

• Security boundary: The sanitiser is the sole entry point for external text
• Core implementation: crates/ai-memory-hooks/src/payload.rs

3. Writer Actor Persistence

A single-writer SQLite actor receives WriteCmd messages and persists observations to the observations table. This actor also maintains the pages table when wiki content is created or superseded, ensuring serialised writes to prevent database contention.

• Concurrency model: Single-writer actor pattern for SQLite
• Core implementation: crates/ai-memory-store/src/writer.rs

4. Wiki Layer (Source of Truth)

The markdown wiki stores durable, human-readable pages. The ai-memory-wiki crate performs atomic writes using a tmp-rename-fsync pattern and records each change as a git commit, guaranteeing durable history and enabling rollbacks.

• Atomic write pattern: tmp file → rename → fsync
• Versioning: Automatic git commits for every change
• Core implementation: crates/ai-memory-wiki/src/lib.rs

5. Derived SQLite Index

After wiki writes, database triggers populate FTS5 (pages_fts) and auxiliary tables including links, entities, and page_embeddings. This derived index enables FTS5 + entity-match + link-neighbour RRF (Reciprocal Rank Fusion) queries and optional vector similarity when an embedder is configured.

• Search capabilities: Full-text search, entity extraction, link analysis
• Core implementation: crates/ai-memory-store/src/fts_query.rs

6. Consolidation Pipeline

When AI_MEMORY_LLM_PROVIDER is configured, the consolidation pipeline reads raw session summary pages, prompts the LLM to rewrite and enrich them, and writes the refined pages back through the wiki writer. This stage also generates derived concept and decision pages.

• Trigger: Environment variable AI_MEMORY_LLM_PROVIDER
• Output: Enriched pages, concept extractions, decision logs
• Core implementation: crates/ai-memory-consolidate/src/lib.rs

7. Auto-Improvement Scheduler

A background scheduler (auto_improve) scans completed sessions and generates proposals for concepts, decisions, gotchas, and best practices via the LLM. Proposals stage in the auto_improve_proposals table and auto-approve by default, writing back through the wiki path to continuously refine the knowledge base.

• Process: Scan → Generate proposal → Stage → Auto-approve → Write
• Core implementation: crates/ai-memory-store/src/auto_improve.rs

8. Query and Feedback

Clients invoke memory_query (or memory_read_page) which combines the SQLite index, entity matches, and optional vector reranking to return ranked results. Page-level feedback via memory_feedback updates page_feedback records and adjusts salience scores used by decay-based retention algorithms.

• Query methods: FTS5 text search, entity matching, vector similarity
• Feedback loop: User feedback adjusts retention salience
• Core implementations: crates/ai-memory-store/src/fts_query.rs and crates/ai-memory-store/src/ops.rs

9. Maintenance and Decay

Periodic forget-sweep operations run against both wiki and SQLite layers, applying decay calculations and tombstone cleanup. Cold pages are evicted based on salience scores while pinned content remains preserved, preventing unbounded growth of the memory store.

• Operations: Decay calculation, tombstone cleanup, eviction
• Core implementation: crates/ai-memory-store/src/maintenance.rs

Closed-Loop System Design

These nine stages form a continuous cycle:

1. Capture: Agents emit events → sanitiser → writer
2. Persist: Writer updates SQLite and triggers wiki writes
3. Index: SQLite/FTS5 tables synchronize with wiki content
4. Enrich: Optional LLM consolidation and auto-improvement convert raw observations into semantic knowledge
5. Serve: Queries read from the index; feedback drives salience adjustments; maintenance prunes stale data

Practical Interaction Examples

Emit a Lifecycle Hook

// Running from a CLI agent
std::process::Command::new("ai-memory")
    .arg("hook")
    .arg("--event")
    .arg("session-start")
    .arg("--payload")
    .arg(r#"{"session_id":"abc123","cwd":"/path/to/project"}"#)
    .status()
    .expect("hook failed");

This spools JSON locally and posts it to /hook, where the router sanitises the payload and enqueues a WriteCmd.

Query the Knowledge Base

# Retrieve top 5 results for a query

ai-memory query --query "how to reset a project" --top_k 5

The memory_query tool combines FTS5, entity-match, and vector similarity (if configured) to return page paths and snippets.

Write Explicit Knowledge

ai-memory write-page \
    --path "notes/quick-start.md" \
    --content "# Quick Start\nRun `ai-memory init` then `ai-memory status`." \

    --expires_at "2027-01-01"

The request reaches the wiki writer, which atomically creates the markdown file, commits to git, and updates the SQLite pages table.

Trigger Manual Auto-Improvement

ai-memory auto-improve --session-id "$(ai-memory status --last-session-id)"

This pulls the session, runs the LLM consolidation pipeline, stages proposals in auto_improve_proposals, and writes approved pages back through the wiki path.

Summary

• ai-memory implements a dual-layer architecture with markdown as source of truth and SQLite as a rebuildable index.
• Nine distinct stages handle everything from hook emission and sanitisation to LLM consolidation and maintenance decay.
• Atomic writes via crates/ai-memory-wiki/src/lib.rs ensure durability through tmp-rename-fsync patterns and git commits.
• Single-writer SQLite actor in writer.rs prevents concurrency issues while maintaining high throughput.
• Zero-LLM default allows operation without external providers, with optional enrichment via AI_MEMORY_LLM_PROVIDER.

Frequently Asked Questions

What makes the markdown wiki the "source of truth" in ai-memory?

The markdown wiki in data_dir/wiki/ serves as the authoritative store because every write operation in crates/ai-memory-wiki/src/lib.rs performs atomic file operations followed by git commits before any database updates occur. If the SQLite index becomes corrupted or desynchronized, the system can rebuild it entirely by re-indexing the markdown files, ensuring the human-readable content always remains intact and authoritative.

How does ai-memory handle data sanitisation and security?

All untrusted text enters through the sanitiser in crates/ai-memory-hooks/src/payload.rs, which is the sole security boundary between external agents and the storage layer. The sanitiser validates and classifies incoming payloads into ObservationKind types before the writer actor processes them, preventing injection attacks and malformed data from reaching the SQLite store or wiki files.

Can ai-memory function without an LLM provider?

Yes. The system operates in zero-LLM mode by default, storing raw observations and manual wiki entries without requiring AI_MEMORY_LLM_PROVIDER. LLM features—including the consolidation pipeline in crates/ai-memory-consolidate/src/lib.rs and auto-improvement in auto_improve.rs—are strictly optional enhancements that enrich raw data into structured knowledge when configured.

What triggers the maintenance and decay processes?

The forget-sweep and decay calculations in crates/ai-memory-store/src/maintenance.rs run periodically against both storage layers, evaluating page salience scores adjusted by user feedback from page_feedback records. Cold content below threshold salience is evicted during tombstone cleanup, while pinned or frequently accessed pages survive the decay process, ensuring the memory store remains bounded and relevant.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}