How ai-memory's "Compile, Not Retrieve" Model Works: Technical Architecture Explained

ai-memory implements a "compile, not retrieve" architecture where LLMs write knowledge into a Git-versioned markdown wiki during consolidation phases, while queries execute against a local SQLite index without calling the LLM, ensuring deterministic, fast, and auditable knowledge retrieval.

The akitaonrails/ai-memory project redefines how AI agents retain information by treating knowledge as a compiled artifact rather than ephemeral retrieved memory. Instead of querying an LLM for every context lookup, the system uses language models only to compress and structure observations into durable markdown pages. This "compile, not retrieve" approach creates a Karpathy-style LLM wiki that serves as the single source of truth for agent memory.

The Core Philosophy: Compile Once, Retrieve Many

According to docs/ARCHITECTURE.md, ai-memory treats the knowledge base as "a git-versioned tree of markdown pages on disk that gets compiled over time, appended-to."

This design inverts the traditional RAG pattern. Rather than retrieving raw documents and prompting an LLM to synthesize answers for every query, ai-memory compiles knowledge upfront. The LLM processes observations into polished wiki pages during consolidation sessions. Subsequent queries retrieve pre-compiled content directly from a structured SQLite index, eliminating per-query LLM latency and cost.

The Compilation Pipeline

The compilation process transforms transient agent observations into permanent, structured knowledge through a lifecycle-driven pipeline.

Capture and Persistence

When agents finish sessions, they emit lifecycle-hook events such as session-end. The system captures these through ai_memory_hooks::sanitize, which validates and structures the incoming data into WriteCmd objects.

These commands persist to the observations table via a writer actor. This raw capture ensures no data is lost before compilation occurs.

Summarization and Consolidation

The critical compilation step occurs in crates/ai-memory-consolidate/src/lib.rs. When AI_MEMORY_LLM_PROVIDER is configured, the memory_consolidate worker processes accumulated observations:

1. Summarize: On session-end, the server creates an initial summary page (e.g., sessions/<id>.md).
2. Consolidate: The consolidation pipeline rewrites this page using LLM-generated prose while preserving the original markdown structure.

This bounded worker ensures LLM output is reviewed and structured before becoming source truth.

Version-Controlled Storage

All compiled knowledge lands in the wiki via crates/ai-memory-wiki/src/lib.rs. The Wiki::write_page function performs atomic writes using tmp-plus-rename patterns followed by Git commits. Every page write is versioned, creating an immutable history where knowledge updates remain traceable.

The Retrieval Layer: Query Without LLM

Once compiled, knowledge retrieval operates entirely against the SQLite index in crates/ai-memory-store/src/pages.rs. The memory_query tool exposed in crates/ai-memory-mcp/src/tools.rs executes three parallel search strategies:

• FTS5 full-text search over page titles and bodies.
• Entity-match using front-matter entities fields.
• Optional vector-search when embeddings are enabled.

The system merges these streams using reciprocal rank fusion (RRF), then applies authority multipliers based on tier, pinned status, and tags. No LLM participates in this retrieval flow unless the optional AI_MEMORY_RERANKER=llm environment variable is set—and even then, only snippet metadata is sent to the model.

Benefits of the Compile-Not-Retrieve Architecture

This architectural choice delivers specific technical advantages:

Deterministic answers emerge because the wiki is immutable and versioned. Every write through Wiki::write_page creates a Git commit, ensuring reproducibility.

Fast, cheap queries result from eliminating per-query LLM calls. The SQLite indices (pages_fts, entities, page_embeddings) operate in-process with bounded latency.

Safety and auditability come from gating LLM output through the memory_consolidate worker. Auto-improve approval workflows can review synthesized content before it enters the permanent wiki.

Long-term retention persists beyond LLM provider lifetimes. Pages survive even if AI_MEMORY_LLM_PROVIDER is later disabled, while raw observations may be pruned only after successful page compilation.

Implementation in Code

The following Rust examples demonstrate the compile and retrieve workflow:

// Compile: Consolidate a session's observations into a wiki page
let consolidate_req = memory_consolidate::Request {
    session_id: Some(session_id),
    multi_page: false,           // single-page rewrite
    ..Default::default()
};
client.memory_consolidate(consolidate_req).await?;

// Retrieve: Query the SQLite index (no LLM involved)
let query_req = memory_query::Request {
    q: "how to reset a project".into(),
    top_k: 5,
    ..Default::default()
};
let hits = client.memory_query(query_req).await?;

// Retrieve: Read full page body when more context is needed
let read_req = memory_read_page::Request {
    path: "procedures/reset-project.md".into(),
    ..Default::default()
};
let page = client.memory_read_page(read_req).await?;
println!("{}", page.body);

The first call triggers the compile step using the LLM; the subsequent calls perform retrieve operations entirely from the local SQLite index.

Summary

• ai-memory stores knowledge in a Git-versioned markdown wiki on disk, making the file system the single source of truth.
• The consolidation pipeline in crates/ai-memory-consolidate/src/lib.rs handles the "compile" phase, using LLMs to synthesize observations into wiki pages.
• Retrieval operates against SQLite indices (FTS5, entities, embeddings) via memory_query, without invoking the LLM.
• Atomic Git commits via Wiki::write_page ensure immutable, auditable knowledge history.
• This architecture eliminates per-query LLM costs while providing deterministic, fast, and long-term memory retention.

Frequently Asked Questions

What makes ai-memory's approach "compile, not retrieve"?

Traditional RAG systems retrieve raw documents and prompt an LLM to generate answers for every query. ai-memory instead compiles knowledge upfront: the LLM processes observations into structured markdown pages during consolidation. Subsequent queries retrieve these pre-compiled pages directly from SQLite, treating the LLM as a writer rather than a reader.

How does the compilation process ensure data safety?

The memory_consolidate worker runs as a bounded process that can be gated by auto-improve approval workflows. Additionally, Wiki::write_page in crates/ai-memory-wiki/src/lib.rs performs atomic tmp-plus-rename writes followed by Git commits, ensuring that compiled knowledge is versioned and immutable before it becomes queryable.

When does ai-memory use an LLM during retrieval?

By default, retrieval via memory_query uses only the SQLite indices (pages_fts, entities, page_embeddings). The LLM is invoked during retrieval only if the AI_MEMORY_RERANKER=llm environment variable is enabled, and even then, it processes only the metadata of retrieved snippets rather than the full content.

What happens to knowledge if the LLM provider is disabled?

Because the wiki is the single source of truth and stored as markdown files on disk, previously compiled pages persist indefinitely even if AI_MEMORY_LLM_PROVIDER is removed or changed. Raw observations in the observations table may be pruned after successful consolidation, but the compiled wiki pages remain available for retrieval via memory_query and memory_read_page.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}