How ai-memory Performs Memory Recall and Search: Architecture and Implementation

ai-memory performs memory recall and search by maintaining a dual-storage architecture where coding observations are stored as markdown files in a Git-backed wiki while a derived SQLite FTS5 index enables fast full-text retrieval, with optional semantic reranking via embeddings.

The ai-memory system, developed by akitaonrails, provides persistent memory for AI coding agents through a sophisticated recall pipeline that balances durability with query performance. Understanding how memory recall and search work in this repository requires examining its unique approach to dual storage: human-readable markdown files serve as the source of truth, while a SQLite-based index powers sub-second search without requiring calls to external LLMs.

The Dual-Storage Architecture (Wiki + SQLite Index)

At the core of ai-memory's recall capability is a write-optimized pipeline that synchronizes human-readable documentation with a machine-optimized search index. This design ensures that every observation captured from coding agents remains accessible both as plain text in a Git repository and as structured data in a relational database.

Ingestion Pipeline: From Capture to Index

When coding agents generate observations, the system processes them through a lifecycle described in the README as the capture → consolidate → recall flow. As detailed in README.md lines 61-68, agents first sanitize their observations through lifecycle hooks. When a session terminates, the server consolidates these discrete observations into a single markdown page representing that session's context.

The system then performs an atomic write operation: the wiki writer persists the markdown page to disk while simultaneously updating the search index. In crates/ai-memory-store/src/ops.rs lines 76-86, the upsert_page function handles this coordination, ensuring that the SQLite index remains consistent with the filesystem state.

Path Tokenization and FTS5 Indexing

During the upsert operation, the system transforms file paths into searchable tokens. The path_search_text function defined in crates/ai-memory-store/src/ops.rs lines 15-20 converts page paths into normalized, search-friendly token strings. These tokens are stored in an FTS5 virtual table named pages_fts, which provides built-in full-text search capabilities with BM25 ranking.

Full-Text Search Implementation with FTS5

The search layer leverages SQLite's FTS5 extension to deliver fast, relevance-ranked results without requiring external services. This implementation supports both structured boolean queries and natural language input, making it suitable for direct agent interaction as well as programmatic access.

Query Sanitization and Natural Language Processing

Raw user queries undergo preprocessing by the prepare_fts5_query function located in crates/ai-memory-store/src/fts_query.rs lines 9-22. This sanitizer performs three critical operations to prevent syntax errors and optimize recall:

• Operator Detection: The function checks for explicit FTS5 operators (OR, AND, NOT, NEAR, quotation marks, and parentheses). If detected, the query passes through unchanged to preserve user intent.
• OR-Join Strategy: For bare natural-language queries, the system splits terms and joins them with OR operators instead of the default AND behavior. This significantly increases recall; for example, "cross project search strategy" becomes cross OR project OR search OR strategy.
• Token Escaping: The sanitizer strips stop-words and escapes punctuation marks. Special characters in filenames like ai-memory or current.md receive proper quoting to prevent SQLite syntax errors, as verified by tests such as colon_is_not_column_syntax and dotted_filename_token_is_quoted.

Executing BM25 Ranked Searches

The prepared query executes against the FTS5 virtual table using standard SQL:

SELECT page_id, rank FROM pages_fts 
WHERE pages_fts MATCH ? 
ORDER BY rank;

The rank column utilizes BM25 scoring, assigning higher relevance to pages where query terms appear with greater frequency and proximity.

Recall Pipeline and Result Augmentation

Beyond basic full-text matching, ai-memory enriches search results through entity extraction and graph relationships, with optional semantic similarity scoring for enhanced precision.

Entity and Link-Based Reranking

Each indexed page contains extracted entities (tags and LLM-generated concepts) stored in the entities table. The system maintains relationship data in entity_page_links, which connects entities to specific page versions. During the retrieval phase, the pipeline joins these tables to boost results that share entities with the query context, effectively implementing graph-neighbor reranking based on conceptual similarity rather than purely lexical matching.

Optional Vector Fusion

When configured with an embedding provider, ai-memory extends its retrieval pipeline with semantic search capabilities. Each page maintains a vector representation in the page_embeddings table. As noted in README.md lines 73-77, the system can blend FTS5 BM25 scores with cosine similarity scores from these embeddings using Reciprocal Rank Fusion (RRF). This hybrid approach combines the precision of keyword matching with the conceptual understanding of vector similarity, though the system functions entirely without embeddings if desired.

Practical Implementation Examples

Developers can interact with the recall system either programmatically through the Rust API or via the HTTP interface.

Performing Search via Rust API

To execute a search directly from Rust code using the ai-memory-store crate:

use ai_memory_store::Reader;
use ai_memory_core::{WorkspaceId, ProjectId};

// Establish connection to the SQLite database
let mut conn = rusqlite::Connection::open("/path/to/ai-memory/data/db/memory.sqlite")?;
let reader = Reader::new(&mut conn)?;

// User-provided search query
let raw_query = "how does ai-memory perform memory recall and search";

// Sanitize for FTS5
let fts_query = ai_memory_store::prepare_fts5_query(raw_query);

// Execute search with limit of 10 results
let rows = reader.search_pages(&fts_query, 10)?;

for (page_id, rank) in rows {
    let page = reader.get_page(page_id)?;
    println!("Rank {} – {}", rank, page.title);
}

Querying via HTTP API

The server exposes the search functionality through a REST endpoint:

curl -G "http://127.0.0.1:49374/api/v1/search" \
     --data-urlencode "q=memory recall" \
     --data "limit=5"

The server internally invokes prepare_fts5_query, executes the FTS5 query, applies entity enrichment and optional embedding reranking, then returns a JSON array of matching pages.

Summary

• ai-memory implements a dual-storage architecture with markdown files as the source of truth and SQLite FTS5 as the search index.
• The upsert_page function in crates/ai-memory-store/src/ops.rs atomically synchronizes content between the wiki and the database.
• FTS5 virtual tables with BM25 ranking provide fast, relevance-sorted full-text search without LLM dependencies.
• Query sanitization via prepare_fts5_query (in fts_query.rs) converts natural language to OR-joined tokens while handling special characters safely.
• Entity augmentation improves results by considering graph relationships stored in entities and entity_page_links tables.
• Optional vector fusion enables semantic search when embeddings are available, combining cosine similarity with BM25 scores via RRF.

Frequently Asked Questions

Does ai-memory require an LLM to perform search?

No. The core search functionality operates entirely within SQLite using FTS5 full-text indexing. As implemented in crates/ai-memory-store/src/reader.rs, the system executes BM25-ranked queries against the pages_fts table without invoking external LLMs. Embeddings and LLM-based entity extraction are optional enhancements, not requirements for basic recall.

How does ai-memory handle natural language queries?

The system processes natural language through the prepare_fts5_query function in crates/ai-memory-store/src/fts_query.rs. For queries without explicit FTS5 operators (like AND, OR, NOT), it splits the input into individual terms and joins them with OR operators. This approach maximizes recall by returning pages matching any query term rather than requiring all terms to be present. The function also handles punctuation escaping and stop-word removal to prevent syntax errors.

What is the relationship between the Git-backed wiki and the SQLite index?

The Git-backed wiki serves as the durable source of truth containing human-readable markdown files. The SQLite database acts as a derived index optimized for query performance. During the consolidation phase, the upsert_page operation writes to both stores atomically. If the database is corrupted or deleted, it can be rebuilt from the markdown files, ensuring data durability remains independent of the search infrastructure.

Can ai-memory perform semantic similarity search?

Yes, when configured with an embedding provider. The system stores vector representations in the page_embeddings table and supports cosine similarity calculations. According to the architecture described in README.md lines 73-77, the recall pipeline can fuse FTS5 BM25 scores with embedding similarity scores using Reciprocal Rank Fusion (RRF). This hybrid approach allows agents to find conceptually related content even when specific keywords differ.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}