# How ai-memory Uses a Two-Layer Model for Storing Data

> Discover how ai-memory uses a two-layer model for data storage. Explore its raw markdown and SQLite layers for durability, portability, and fast retrieval. Learn more!

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-28

---

**ai-memory persists all information in a two-layer architecture: a raw markdown (Wiki) layer for durability and portability, and a SQLite (Store) layer for fast full-text search and vector-based retrieval.**

This Rust-based knowledge management tool, developed by Akita On Rails, separates human-readable content from machine-optimized indexes. The design ensures your data remains accessible, version-controllable, and searchable without trade-offs between readability and performance.

---

## What Is the Two-Layer Model in ai-memory?

The **two-layer model** is the foundational storage architecture of ai-memory. Rather than choosing between file-based simplicity and database performance, ai-memory implements both as synchronized layers:

- **Layer 1: Wiki (raw markdown)** — Plain-text files stored atomically on the filesystem
- **Layer 2: SQLite (indexed)** — A structured database with FTS5 full-text search and optional vector embeddings

Every write operation traverses both layers in a single logical transaction, maintaining strict consistency between your files and your search indexes.

---

## Layer 1: The Wiki (Raw Markdown) Layer

The Wiki layer serves as the **canonical source of truth**. All user-generated content—notes, observations, pages—exists first and foremost as markdown files on disk.

### Atomic Write Implementation

In [`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs), the `Wiki` API implements **atomic filesystem writes** using a temp-file + rename + fsync pattern:

```rust
// Simplified representation of the atomic write pattern
let temp_path = format!("{}.tmp", final_path);
// Write to temp, fsync, then atomic rename
tokio::fs::write(&temp_path, content).await?;
tokio::fs::rename(&temp_path, &final_path).await?;

```

This guarantees that:
- Data survives application crashes or power failures
- Files are never partially written
- Content is instantly readable with any text editor

### Benefits of the Markdown Layer

- **Git-native versioning**: Your knowledge base is a normal Git repository
- **Portable across machines**: No proprietary formats or migration tools needed
- **Human-inspectable**: Use `cat`, `grep`, or any editor without special tooling
- **Future-proof**: Markdown will outlive any specific application's database schema

The Wiki layer implementation resides in [`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs) as documented in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md).

---

## Layer 2: The SQLite (Indexed) Layer

The Store layer provides **high-performance retrieval capabilities** that would be impractical with filesystem scans alone.

### Database Structure and Features

In [`crates/ai-memory-store/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/lib.rs), the SQLite layer implements:

| Feature | Implementation Detail |
|--------|----------------------|
| Full-text search | SQLite FTS5 extension |
| Vector similarity | Optional embedding storage for LLM-driven retrieval |
| Relational queries | `(workspace_id, project_id, path)` triple indexing |
| Single-writer concurrency | Actor pattern prevents write conflicts |

The database file `memory.sqlite` mirrors your markdown content without duplicating the full text unnecessarily. Instead, it maintains:

- Metadata triples for fast filtering
- FTS5 index tokens for sub-second search
- Optional vector embeddings when an LLM embedder is configured

### Single-Writer Actor Pattern

The Store uses a **single-writer SQLite actor** to serialize all database mutations. This pattern, detailed in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md), eliminates SQLite's "database is locked" errors while maintaining high read concurrency through separate connections.

---

## Synchronization: How Both Layers Stay Consistent

The two-layer model only works if mutations propagate atomically. ai-memory achieves this through a strict **write-through policy**:

1. All modifications flow through `Wiki::write_page()`
2. The Wiki layer performs the atomic filesystem write
3. The same code path invokes `Store::index_page()` to update SQLite

### Example: Writing a Page Updates Both Layers

```rust
use ai_memory_wiki::Wiki;
use ai_memory_store::Store;

let path = "notes/ai-memory-overview.md";
let content = "# AI-Memory Overview\n\nTwo-layer model description …";

// Single call updates both filesystem and database
wiki.write_page(&path, content).await?;
store.index_page(&path, content).await?;

```

The [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) document formalizes the invariants: at any stable point, the SQLite index must reflect the exact state of the markdown files. There are no background sync jobs that could drift or fail silently.

---

## Query Patterns: Choosing the Right Layer

Different operations naturally target different layers:

### Raw Content Retrieval (Wiki Layer)

```rust
use ai_memory_wiki::Wiki;

let page = wiki.read_page("notes/ai-memory-overview.md").await?;
println!("{}", page);  // Direct markdown content

```

Use this when you need the exact file content, want to stream large pages, or are exporting data.

### Full-Text Search (SQLite Layer)

```rust
use ai_memory_store::Store;

let query = "two-layer model";
let results = store.search(query).await?;

for r in results {
    println!("Found in: {}", r.path);
    // Retrieve actual content from Wiki layer if needed
    let page = wiki.read_page(&r.path).await?;
}

```

The search hits **only** the SQLite layer, avoiding expensive filesystem traversal.

---

## Why a Two-Layer Model? Design Rationale

According to [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md), ai-memory's architecture deliberately rejects common alternatives:

| Alternative | Why ai-memory Avoids It |
|------------|------------------------|
| Pure files (no database) | Directory scans and `grep` don't scale; no vector search |
| Pure database (no files) | Proprietary lock-in; loses Git versioning and editor accessibility |
| Object storage (S3-style) | Adds network latency and cost for local-first tool |
| Embedded document DB | Adds dependency complexity; harder to inspect and debug |

The two-layer model preserves **simplicity where it matters** (human-readable files) and **performance where needed** (structured indexes).

---

## Key Implementation Files

| Path | Role |
|------|------|
| [`crates/ai-memory-wiki/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/lib.rs) | Atomic markdown I/O, `Wiki` struct |
| [`crates/ai-memory-store/src/lib.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/lib.rs) | SQLite actor, FTS5 indexing, vector backend |
| [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) | Two-layer invariants and data flow documentation |
| [`crates/ai-memory-cli/src/main.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/main.rs) | CLI wiring both layers together |

---

## Summary

- **Two-layer model**: Markdown files (durability) + SQLite (performance)
- **Wiki layer** in `ai-memory-wiki`: Atomic writes, Git-native, human-readable
- **Store layer** in `ai-memory-store`: FTS5 search, vector embeddings, single-writer actor
- **Strict synchronization**: Every write updates both layers; no background drift
- **Practical flexibility**: Query SQLite for speed, Wiki for raw content, combine as needed

---

## Frequently Asked Questions

### What happens if the SQLite database is deleted?

Your data remains fully intact in the markdown files. Rebuilding the database requires only re-indexing: the `Store` can crawl the Wiki directory and reconstruct all indexes from scratch. No content is lost because SQLite is strictly a derived layer.

### Does the two-layer model impact write performance?

Writes are serialized through the single-writer actor, which adds minimal latency for the atomic operations. The architecture document in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) notes that typical note-taking workloads (< 100 writes/second) experience no perceptible delay. Heavy batch imports can use the provided bulk indexing API.

### Can I disable the SQLite layer and use only markdown files?

No—ai-memory requires the Store layer for core functionality including search and LLM context retrieval. However, the design ensures you can always extract your data as plain markdown even if you stop using ai-memory entirely.

### How does vector search fit into the two-layer model?

Optional embeddings are stored in the SQLite layer alongside FTS5 indexes. When configured with an LLM embedder (OpenAI, local models, etc.), the `ai-memory-store` crate generates and queries vectors without modifying the Wiki layer. The architecture keeps vector operations separate from your canonical markdown files.