How Data Flows Through ai-memory: A Complete Architecture Guide

Data flows through ai-memory via a strict two-layer pipeline where lifecycle hooks are sanitized, serialized by a single-writer actor to SQLite, and optionally consolidated by LLMs, while the markdown wiki remains the immutable source of truth.

The ai-memory repository (akitaonrails/ai-memory) implements a memory layer for AI agents using a dual-layer storage architecture. Understanding how data flows through ai-memory enables developers to integrate robust observability, as the system guarantees atomic transactions, bounded write paths, and high-performance read operations through a derived SQLite index.

The Two-Layer Architecture

The foundation of the data flow rests on two distinct layers. The markdown-based wiki serves as the canonical source of truth, storing human-readable session summaries and knowledge pages. All mutations ultimately persist to disk as atomic markdown files. The SQLite index operates as a derived, query-optimized layer that powers fast search, analytics, and tool use, but it never holds authoritative state. As documented in docs/ARCHITECTURE.md (lines 23-33), this separation ensures that even if the index is rebuilt or corrupted, the wiki remains intact and recoverable.

The 8-Stage Data Flow Pipeline

The complete data flow follows a linear progression from event emission to retention management. Each stage enforces specific guarantees around data integrity and access patterns.

Stage 1: Lifecycle Hook Emission

The pipeline initiates when agent CLIs or native ai-memory hook commands emit lifecycle events. These events include SessionStart, UserPrompt, PostToolUse, and SessionEnd. According to the architecture documentation (lines 45-57), clients POST these payloads to the /hook endpoint using short-timeout HTTP requests. This design ensures that agent execution never blocks on memory persistence, while the server accepts untrusted input for processing.

Stage 2: Hook Routing and Sanitization

Incoming hooks enter the router defined in crates/ai-memory-hooks/src/router.rs. Here, the system validates the JSON payload, strips private data fields, and assigns an ObservationKind classification. The router then constructs a WriteCmd structure and dispatches it to the single-writer SQLite actor (lines 61-68, docs/ARCHITECTURE.md). This single-channel architecture guarantees that all untrusted text enters the storage layer through one bounded, auditable path.

Stage 3: Single-Writer Persistence

The writer actor, implemented in crates/ai-memory-core/src/actor.rs, runs on a dedicated OS thread to prevent blocking the async runtime. Upon receiving a WriteCmd, it performs an atomic transaction that:

• Inserts the observation into the observations table
• Updates the pages table via supersession logic
• Appends a log line to log.md for audit trails

This stage (lines 69-74) ensures that the SQLite index and the markdown wiki move forward in lockstep, with the wiki write succeeding before the transaction commits.

Stage 4: Session-End Synthesis

When the router processes a SessionEnd event, it triggers the synthesis subsystem. The server generates a summary page at sessions/<id>.md and creates a handoff row for the next agent context (lines 75-83). This operation executes as an atomic transaction, guaranteeing that the session is fully captured before any handoff becomes visible to subsequent queries. The implementation resides in crates/ai-memory-core/src/actor.rs, coordinating between the wiki and store layers.

Stage 5: Optional LLM Consolidation

If the environment variable AI_MEMORY_LLM_PROVIDER is configured, the memory_consolidate job activates. Located in crates/ai-memory-consolidate/src/consolidator.rs, this stage rewrites summary pages or fans out content to multiple thematic pages using the configured LLM (lines 93-98). This step is purely additive; it operates on the already-committed wiki state and produces refined, structured knowledge without blocking the core data flow.

Stage 6: Auto-Improvement Scheduling

A background scheduler reviews newly completed sessions and generates structural proposals. It suggests new pages under concepts/, decisions/, or gotchas/ directories, records these in a pending-writes audit trail, and by default approves them through the standard wiki-write path (lines 99-108). This autonomous maintenance ensures the knowledge base evolves without manual intervention while preserving the same write-guarantees as explicit client requests.

Stage 7: Query Processing

Read operations follow a separate, lock-free path. The memory_query tool, implemented in crates/ai-memory-cli/src/commands/query.rs, reads exclusively from the SQLite index. It executes FTS5 full-text search, entity-match filtering, and optional vector-based RRF (Reciprocal Rank Fusion) retrieval (lines 117-124). For every result returned, the query processor atomically bumps access_count and updates last_accessed_at (lines 134-136), providing telemetry for the retention sweeps without interfering with the writer actor.

Stage 8: Retention and Forget Sweeps

Periodic background tasks enforce storage policies. These sweeps delete expired pages based on TTL, evict cold pages using access-count heuristics, and prune raw observations that have been superseded by consolidated wiki pages (lines 141-162). This final stage preserves the invariant that the wiki remains the canonical source while keeping the derived index lean and performant.

Key Source Files and Components

The data flow relies on specific implementations across the codebase:

• crates/ai-memory-core/src/actor.rs – Defines the writer actor that owns the exclusive SQLite connection and coordinates all mutations.
• crates/ai-memory-hooks/src/router.rs – Implements hook routing and sanitization, converting HTTP payloads into internal WriteCmd structures.
• crates/ai-memory-store/src/lib.rs – Provides the public API for reading and writing observations, pages, and handoffs to the SQLite layer.
• crates/ai-memory-wiki/src/lib.rs – Handles atomic markdown wiki operations including write, supersede, and link extraction.
• crates/ai-memory-consolidate/src/consolidator.rs – Contains the LLM-driven consolidation logic for summarization and page generation.
• crates/ai-memory-cli/src/commands/query.rs – Implements the memory_query MCP tool for read-only access.

Practical Implementation Examples

To interact with the data flow programmatically, use the following patterns from the akitaonrails/ai-memory codebase.

Emit a lifecycle hook using the Rust client:

use ai_memory_hooks::payload::Payload;

let payload = Payload::new("session-start", "my-project", "{}", None);
ai_memory_hooks::router::handle(payload).await?;

Execute a read-only query via CLI:

ai-memory memory_query "how does the search work?" --scopes .

Write a page programmatically:

use ai_memory_cli::commands::write_page;

write_page::run(
    "knowledge/ai-memory-overview.md",
    "# Overview\nAI-memory stores ...",

    false,               // not global
    None,                // no TTL
)?;

Trigger the auto-improve scheduler manually:

ai-memory auto-improve --project my-project

Summary

• Data flows through ai-memory via a linear pipeline: hook → sanitize → writer → wiki + SQLite → optional LLM → auto-improve → query.
• The markdown wiki is the single source of truth; the SQLite index is a derived, disposable cache for fast queries.
• All writes pass through a single-writer actor on a dedicated thread, guaranteeing atomicity and preventing corruption.
• Read operations are lock-free and execute against the SQLite index, never touching raw markdown files directly.
• Optional LLM consolidation and auto-improvement stages run asynchronously without blocking the core write path.
• Retention sweeps maintain storage bounds while preserving wiki integrity.

Frequently Asked Questions

What is the source of truth in ai-memory?

The markdown-based wiki serves as the immutable source of truth. Every mutation ultimately persists as atomic markdown files on disk, while the SQLite index functions as a derived, query-optimized layer that can be rebuilt from the wiki at any time.

How does ai-memory ensure thread-safe writes?

All write operations route through a single-writer SQLite actor defined in crates/ai-memory-core/src/actor.rs. This actor runs on a dedicated OS thread and processes WriteCmd messages sequentially, eliminating race conditions and ensuring that the wiki and SQLite index remain synchronized.

Can ai-memory operate without an LLM provider?

Yes. The LLM consolidation stage is optional and only activates if the AI_MEMORY_LLM_PROVIDER environment variable is set. Without it, the system skips the memory_consolidate job and proceeds directly to auto-improvement and query processing, storing raw observations and manual edits without AI-driven summarization.

How does querying interact with the data flow?

Querying follows a read-only path separate from the write pipeline. The memory_query tool reads from the SQLite index using FTS5 and vector search, bypassing the writer actor entirely. It updates access metadata (access_count, last_accessed_at) to inform retention policies but never modifies the underlying markdown files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}