How ai-memory Uses a Two-Layer Model for Storing Data

ai-memory persists all information in a two-layer architecture: a raw markdown (Wiki) layer for durability and portability, and a SQLite (Store) layer for fast full-text search and vector-based retrieval.

This Rust-based knowledge management tool, developed by Akita On Rails, separates human-readable content from machine-optimized indexes. The design ensures your data remains accessible, version-controllable, and searchable without trade-offs between readability and performance.

---

What Is the Two-Layer Model in ai-memory?

The two-layer model is the foundational storage architecture of ai-memory. Rather than choosing between file-based simplicity and database performance, ai-memory implements both as synchronized layers:

• Layer 1: Wiki (raw markdown) — Plain-text files stored atomically on the filesystem
• Layer 2: SQLite (indexed) — A structured database with FTS5 full-text search and optional vector embeddings

Every write operation traverses both layers in a single logical transaction, maintaining strict consistency between your files and your search indexes.

---

Layer 1: The Wiki (Raw Markdown) Layer

The Wiki layer serves as the canonical source of truth. All user-generated content—notes, observations, pages—exists first and foremost as markdown files on disk.

Atomic Write Implementation

In crates/ai-memory-wiki/src/lib.rs, the Wiki API implements atomic filesystem writes using a temp-file + rename + fsync pattern:

// Simplified representation of the atomic write pattern
let temp_path = format!("{}.tmp", final_path);
// Write to temp, fsync, then atomic rename
tokio::fs::write(&temp_path, content).await?;
tokio::fs::rename(&temp_path, &final_path).await?;

This guarantees that:

• Data survives application crashes or power failures
• Files are never partially written
• Content is instantly readable with any text editor

Benefits of the Markdown Layer

• Git-native versioning: Your knowledge base is a normal Git repository
• Portable across machines: No proprietary formats or migration tools needed
• Human-inspectable: Use cat, grep, or any editor without special tooling
• Future-proof: Markdown will outlive any specific application's database schema

The Wiki layer implementation resides in crates/ai-memory-wiki/src/lib.rs as documented in docs/ARCHITECTURE.md.

---

Layer 2: The SQLite (Indexed) Layer

The Store layer provides high-performance retrieval capabilities that would be impractical with filesystem scans alone.

Database Structure and Features

In crates/ai-memory-store/src/lib.rs, the SQLite layer implements:

Feature	Implementation Detail
Full-text search	SQLite FTS5 extension
Vector similarity	Optional embedding storage for LLM-driven retrieval
Relational queries	(workspace_id, project_id, path) triple indexing
Single-writer concurrency	Actor pattern prevents write conflicts

The database file memory.sqlite mirrors your markdown content without duplicating the full text unnecessarily. Instead, it maintains:

• Metadata triples for fast filtering
• FTS5 index tokens for sub-second search
• Optional vector embeddings when an LLM embedder is configured

Single-Writer Actor Pattern

The Store uses a single-writer SQLite actor to serialize all database mutations. This pattern, detailed in docs/ARCHITECTURE.md, eliminates SQLite's "database is locked" errors while maintaining high read concurrency through separate connections.

---

Synchronization: How Both Layers Stay Consistent

The two-layer model only works if mutations propagate atomically. ai-memory achieves this through a strict write-through policy:

1. All modifications flow through Wiki::write_page()
2. The Wiki layer performs the atomic filesystem write
3. The same code path invokes Store::index_page() to update SQLite

Example: Writing a Page Updates Both Layers

use ai_memory_wiki::Wiki;
use ai_memory_store::Store;

let path = "notes/ai-memory-overview.md";
let content = "# AI-Memory Overview\n\nTwo-layer model description …";

// Single call updates both filesystem and database
wiki.write_page(&path, content).await?;
store.index_page(&path, content).await?;

The docs/ARCHITECTURE.md document formalizes the invariants: at any stable point, the SQLite index must reflect the exact state of the markdown files. There are no background sync jobs that could drift or fail silently.

---

Query Patterns: Choosing the Right Layer

Different operations naturally target different layers:

Raw Content Retrieval (Wiki Layer)

use ai_memory_wiki::Wiki;

let page = wiki.read_page("notes/ai-memory-overview.md").await?;
println!("{}", page);  // Direct markdown content

Use this when you need the exact file content, want to stream large pages, or are exporting data.

Full-Text Search (SQLite Layer)

use ai_memory_store::Store;

let query = "two-layer model";
let results = store.search(query).await?;

for r in results {
    println!("Found in: {}", r.path);
    // Retrieve actual content from Wiki layer if needed
    let page = wiki.read_page(&r.path).await?;
}

The search hits only the SQLite layer, avoiding expensive filesystem traversal.

---

Why a Two-Layer Model? Design Rationale

According to docs/ARCHITECTURE.md, ai-memory's architecture deliberately rejects common alternatives:

Alternative	Why ai-memory Avoids It
Pure files (no database)	Directory scans and grep don't scale; no vector search
Pure database (no files)	Proprietary lock-in; loses Git versioning and editor accessibility
Object storage (S3-style)	Adds network latency and cost for local-first tool
Embedded document DB	Adds dependency complexity; harder to inspect and debug

The two-layer model preserves simplicity where it matters (human-readable files) and performance where needed (structured indexes).

---

Key Implementation Files

Path	Role
crates/ai-memory-wiki/src/lib.rs	Atomic markdown I/O, Wiki struct
crates/ai-memory-store/src/lib.rs	SQLite actor, FTS5 indexing, vector backend
docs/ARCHITECTURE.md	Two-layer invariants and data flow documentation
crates/ai-memory-cli/src/main.rs	CLI wiring both layers together

---

Summary

• Two-layer model: Markdown files (durability) + SQLite (performance)
• Wiki layer in ai-memory-wiki: Atomic writes, Git-native, human-readable
• Store layer in ai-memory-store: FTS5 search, vector embeddings, single-writer actor
• Strict synchronization: Every write updates both layers; no background drift
• Practical flexibility: Query SQLite for speed, Wiki for raw content, combine as needed

---

Frequently Asked Questions

What happens if the SQLite database is deleted?

Your data remains fully intact in the markdown files. Rebuilding the database requires only re-indexing: the Store can crawl the Wiki directory and reconstruct all indexes from scratch. No content is lost because SQLite is strictly a derived layer.

Does the two-layer model impact write performance?

Writes are serialized through the single-writer actor, which adds minimal latency for the atomic operations. The architecture document in docs/ARCHITECTURE.md notes that typical note-taking workloads (< 100 writes/second) experience no perceptible delay. Heavy batch imports can use the provided bulk indexing API.

Can I disable the SQLite layer and use only markdown files?

No—ai-memory requires the Store layer for core functionality including search and LLM context retrieval. However, the design ensures you can always extract your data as plain markdown even if you stop using ai-memory entirely.

How does vector search fit into the two-layer model?

Optional embeddings are stored in the SQLite layer alongside FTS5 indexes. When configured with an LLM embedder (OpenAI, local models, etc.), the ai-memory-store crate generates and queries vectors without modifying the Wiki layer. The architecture keeps vector operations separate from your canonical markdown files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}