Why ai-memory Treats Its Markdown Wiki as the Source of Truth

In ai-memory, the Markdown wiki serves as the single source of truth—all persistent knowledge, sessions, and project data live in a git-versioned tree of markdown files, while SQLite indexes function only as derived, read-only search layers.

The ai-memory project implements a Karpathy-style LLM wiki that inverts traditional application architecture. Instead of treating a database as the canonical store, the system persists every piece of knowledge as plain markdown on disk. This design ensures that human-readable files remain the authoritative artifact, with all other data structures functioning as performance-optimized mirrors.

The Canonical Store Architecture

According to docs/ARCHITECTURE.md (lines 16-22), the wiki is defined as a "Karpathy-style LLM wiki"—a git-versioned tree of markdown pages that is continually compiled and appended-to. The markdown itself is the source of truth, not a serialization format or cache layer.

Every write operation mutates the markdown directly. When the system records a new session summary, auto-improvement proposal, or handoff, it persists these changes via the wiki writer in crates/ai-memory-wiki/src/lib.rs. This guarantees that the filesystem representation remains the authoritative state, with Git providing immutable version history.

Atomic Persistence and Version Control

The wiki layer implements atomic write semantics to prevent data corruption. When updating a page, the system writes to a temporary file, executes a rename operation, and calls fsync to ensure durability. This tmp + rename + fsync pattern ensures that readers never observe partial writes, even during system crashes.

Because the markdown resides in a Git repository, the wiki naturally supports supersession: later edits do not erase earlier content but instead create new versions. This preserves the complete lineage of knowledge evolution without complex database migration schemes.

Derived Storage: The SQLite Index

While the markdown wiki holds the authoritative data, docs/ARCHITECTURE.md (lines 28-31) clarifies that the SQLite index is merely a derived structure. Implemented in crates/ai-memory-store/src/lib.rs, this index provides FTS5 full-text search and vector embeddings for fast retrieval, but it remains strictly secondary.

The SQLite store functions as a read-only mirror of the wiki contents. If the index becomes corrupted or obsolete, the system can rebuild it entirely from the markdown source without data loss. This separation of concerns ensures that complex search capabilities never compromise the integrity of the primary data store.

Writing to the Source of Truth

All read-write paths flow through the wiki layer. The Rust API exposes this through the ai_memory_wiki crate, while CLI commands delegate to the same underlying writer.

Writing Pages from Rust

use ai_memory_wiki::{Wiki, WriteOptions};

fn add_idea(project: &str, title: &str, body: &str) -> anyhow::Result<()> {
    // Resolve the wiki root for the current project
    let wiki = Wiki::open(project)?;
    // Path relative to the wiki root
    let page_path = format!("ideas/{}.md", title.replace(' ', "_"));
    // Write the markdown content atomically
    wiki.write_page(&page_path, body, WriteOptions::default())?;
    Ok(())
}

Appending via CLI

# After a session ends, synthesize a summary page

ai-memory write-page "sessions/$(date +%F).md" \
    "## Session Summary\n$(cat summary.txt)"

Querying the Derived Index

# Search uses the SQLite index, but results point back to markdown sources

ai-memory query "vector embeddings"

Every searchable entry in the SQLite index has an upstream markdown source in the wiki, maintaining the single-source-of-truth invariant.

Summary

• Markdown files are canonical: All knowledge persists in a Git-versioned wiki tree, making the filesystem the authoritative store.
• SQLite is derived: The index in crates/ai-memory-store/src/lib.rs provides fast search but functions only as a read-only mirror.
• Atomic writes guarantee durability: The wiki layer uses tmp-rename-fsync semantics in crates/ai-memory-wiki/src/lib.rs to ensure crash-safe updates.
• Supersession preserves history: Edits create new versions rather than overwriting, maintaining complete knowledge lineage.
• Human-readable foundation: Because truth lives in markdown, you can use standard Unix tools (grep, git, vim) to inspect and modify project data directly.

Frequently Asked Questions

Why does ai-memory use markdown files instead of a database as the source of truth?

Markdown provides human-readable, portable persistence that preserves complete knowledge lineage. Because files are plain text and Git-versioned, you maintain full history without complex schema migrations. The database serves only as a performance layer for vector and full-text search, while the markdown remains the durable, authoritative artifact.

How does ai-memory ensure data integrity when writing to markdown files?

The system employs atomic file operations implemented in crates/ai-memory-wiki/src/lib.rs. Every write creates a temporary file, renames it into place, and calls fsync to flush buffers to disk. This ensures that crashes never leave the wiki in a partially updated state, and Git provides additional versioning safety.

What happens if the SQLite index becomes corrupted or out of sync?

The index can be fully rebuilt from the markdown wiki without data loss. Since crates/ai-memory-store/src/lib.rs maintains only derived data (FTS5 tokens and vector embeddings), deleting and regenerating the SQLite file simply re-indexes the existing markdown source of truth.

Can I access ai-memory data without using the SQLite search index?

Yes—because all data lives in plain markdown files in the wiki directory. You can read, edit, and version control them directly using standard tools. The SQLite index is optional for querying but never required for data retrieval; the markdown files are always the primary interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}