# How ai-memory Implements Long-Term Memory for AI Coding Agents

> Discover how ai-memory builds long-term memory for AI coding agents using tiered Markdown files, SQLite FTS5, and semantic indexing for durable, retrievable knowledge.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-26

---

**ai-memory stores an agent’s long-term knowledge as tiered Markdown files under a configurable `wiki/` directory, using atomic write-through to SQLite with FTS5 full-text search and semantic entity indexing for durable, retrievable memory across sessions.**

The `akitaonrails/ai-memory` repository provides a Rust-based persistence layer that enables AI coding agents to retain knowledge across process restarts. By treating Markdown files as the source of truth and maintaining a synchronized SQLite index, the system delivers both human-readable archives and machine-queryable long-term memory.

## The Four-Tier Memory Model

According to the source code in [[`crates/ai-memory-core/src/page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs)](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs) (lines 16‑31), ai-memory classifies every memory page into one of four semantic tiers that determine lifespan and stability:

- **Working** – Transient session data including recent observations and open files
- **Episodic** – Per-session summaries tagged with timestamps and touched file references
- **Semantic** – Distilled architectural knowledge, facts, and wiki-style content intended for long-term retention
- **Procedural** – Extracted patterns from clustered episodic memories representing repeatable workflows

This tiering system allows the consolidation pipeline to apply different retention and indexing strategies based on the anticipated longevity of the information.

## Atomic Write-Through Pipeline

All page creation and updates flow through **`Wiki::write_page`** in [[`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs)](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) (lines 31‑34). This function implements a reliable write-through protocol:

1. **Sanitization** – Removes secrets from body and frontmatter
2. **Generation** – Produces titles, wiki-links, expiry timestamps, and entity lists from the Markdown content
3. **Atomic Persistence** – Writes to a temporary file, renames it to the final location, and calls `fsync` before upserting the corresponding row into SQLite via a single-writer actor

The SQLite schema stores a row per page version, defined by the `NewPage` struct in [`page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/page.rs) (lines 33‑84). Older revisions remain accessible as superseded rows marked with `is_latest=false`, enabling time-travel queries through `Page::supersedes`.

## Hybrid Search and Retrieval

The system exposes long-term memory through two complementary indexing strategies implemented in [[`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs)](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs):

**FTS5 Full-Text Search**

The SQLite database maintains an FTS5 index on raw Markdown bodies and frontmatter, supporting rapid keyword lookups across all pages. The MCP admin API endpoint `/admin/search` queries this index directly.

**Entity and Embedding Indexes**

The consolidator extracts salient nouns (entities) and generates optional vector embeddings from each page. These indexes enable semantic retrieval that supplements lexical search.

When agents query memory via the `memory_query` endpoint in [[`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs)](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1802‑1825), the handler first attempts a scoped FTS5 query. If the compiled page misses, it falls back to raw observation search, ensuring both curated wiki content and recent observations remain discoverable.

## Versioning and Retention Policies

Pages support explicit lifecycle management through pinning and expiration mechanisms defined in [`wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/wiki.rs) (lines 16‑18 via `parse_expires_at`):

- **Pinning** – Set `pinned=true` to prevent automatic decay regardless of tier
- **Expiration** – Assign an `expires_at` timestamp after which a retention sweep deletes the page
- **Versioning** – Each update creates a new version while preserving historical rows for audit trails

This design ensures that long-term memory remains bounded and relevant, with procedural and semantic tiers receiving longer retention periods than working or episodic data.

## Practical Examples

To create a Semantic-tier page programmatically:

```rust
use ai_memory_core::{NewPage, Tier, PagePath, WorkspaceId, ProjectId};
use ai_memory_wiki::Wiki;

let wiki = Wiki::new(/* config */).await?;
let req = WritePageRequest {
    workspace_id: WorkspaceId::new("default"),
    project_id: ProjectId::new("my-project"),
    path: PagePath::new("notes/async.md")?,
    frontmatter: json!({ "title": "Async in Rust" }),
    body: r#"
        # Async in Rust

        Rust's async/await is powered by futures.
        "#.into(),
    tier: Tier::Semantic,
    pinned: false,
    title: None,
    admission_ctx: None,
    author_id: None,
    actor: ActorContext::default(),
};

let page_id = wiki.write_page(req).await?;
println!("Created page with id {}", page_id);

```

To query memory via the MCP client:

```rust
use ai_memory_mcp::client::AiMemoryClient;

let client = AiMemoryClient::new("http://127.0.0.1:49374")?;
let resp = client
    .admin_search("async futures", None, None, None)
    .await?;
println!("Search hits: {:?}", resp.hits);

```

For generating embeddings through the consolidator:

```rust
use ai_memory_consolidate::Consolidator;

let consolidator = Consolidator::new(/* config */).await?;
let page = consolidator.fetch_page(page_id).await?;
let embedding = consolidator.embed_page(&page).await?;
println!("Embedding vector length: {}", embedding.len());

```

## Summary

- **Markdown-based pages** serve as the atomic units of long-term memory, stored in a tiered hierarchy under `<data_dir>/wiki/`
- **Atomic write-through** via `Wiki::write_page` guarantees consistency between filesystem and SQLite using temp-file rename patterns
- **Hybrid retrieval** combines FTS5 full-text search with entity and vector embeddings for both lexical and semantic queries
- **Versioning and retention** mechanisms manage page lifecycle through pinning, expiration timestamps, and superseded row tracking

## Frequently Asked Questions

### What storage backends does ai-memory use for long-term memory?

ai-memory uses a dual-layer approach: Markdown files on disk serve as the human-readable source of truth, while SQLite provides the queryable index. The system stores raw page content in the filesystem and maintains metadata, FTS5 indexes, and entity relationships in SQLite, as implemented in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs).

### How does ai-memory prevent data loss when writing memory pages?

The `Wiki::write_page` function implements atomic write semantics by writing to a temporary file first, then renaming it to the final destination and calling `fsync` before updating the database. This ensures that either both the filesystem and SQLite reflect the change, or neither does, preventing partial writes during crashes.

### What distinguishes Semantic memory from Episodic memory in the tier system?

**Semantic** memory contains distilled facts and architectural knowledge intended for long-term retention, while **Episodic** memory captures per-session summaries with temporal tags and file references. According to [`crates/ai-memory-core/src/page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs), semantic content undergoes more aggressive consolidation and indexing, whereas episodic entries typically decay faster unless promoted.

### How does the search system handle queries that miss in the compiled pages?

When a query finds no matches in the curated wiki pages, the MCP server endpoint in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1802‑1825) automatically falls back to raw observation search. This layered retrieval ensures agents can access both formalized knowledge and recent transient observations.