How ai-memory Ensures Markdown Is the Source of Truth: File-First Architecture Explained

ai-memory guarantees that Markdown files remain the single source of truth through a file-first architecture where all knowledge persists as plain .md files in a Git-backed wiki/ directory, while the SQLite database serves only as a rebuildable derived index.

The ai-memory project treats Markdown not as an export format but as the authoritative storage layer. This design choice fundamentally shapes how data flows through the system, ensuring that even if the search index corrupts or diverges, the truth can always be reconstructed from the on-disk files. According to the project's design documents, "markdown in a git repo is source of truth, SQLite is the derived index" docs/design-decisions.md#L42-L45.

Core Design Principles

The Wiki Directory as Immutable Layer

All memory content lives in a standard Git repository located at <data_dir>/wiki/. This directory structure is intentional and documented: the wiki/ folder holds the markdown source of truth while db/ contains only derived indexes README.md#L99-L103. The README emphasizes this directly: "Your memory is plain markdown. The source of truth is a git‑backed wiki of ordinary .md files" README.md#L43-L46.

Two-Layer Architecture

• Layer 1: Markdown files — Human-readable, version-controlled, portable
• Layer 2: SQLite + vector indexes — Fast search, embeddings, and retrieval

The second layer exists purely for performance. It can be destroyed and regenerated at any time without data loss, precisely because it derives entirely from Layer 1.

Write Pipeline: Atomic Updates to Both Layers

The critical mechanism enforcing the source-of-truth guarantee is Wiki::write_page in [crates/ai-memory-wiki/src/wiki.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs). This function implements transactional dual writes: it updates the Markdown file and the SQLite index atomically, with automatic rollback on failure.

Transaction Semantics

When write_page executes:

1. Stage the Markdown file change using git2
2. Attempt to upsert the corresponding record in SQLite
3. If the store write fails, roll back the file change before returning an error wiki.rs#L1657-L1662 wiki.rs#L3166-L3170

This rollback behavior is tested explicitly in [write_page_rolls_back_file_when_store_upsert_fails.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/tests/write_page_rolls_back_file_when_store_upsert_fails.rs), ensuring the invariant holds even under failure conditions.

Git-Backed Immutability

Every successful write commits to Git atomically via git2::Repository::commit, creating an immutable audit trail wiki.rs#L219-L226. This means:

• Complete version history is preserved in the Markdown files themselves
• Manual edits outside the application are valid and trackable
• The source of truth remains readable with any text editor or standard Git tooling

Rebuilding Indexes from Source

The system provides Wiki::rebuild_store_from_wiki to reconstruct the entire SQLite index from the current Markdown tree wiki.rs#L1345-L1350. This capability reinforces the hierarchy: Markdown is primary, the database is disposable.

Common scenarios requiring rebuild:

• Manual edits to .md files outside the application
• Corruption or accidental deletion of the SQLite database
• Schema migrations that require re-indexing

Working with the Source-of-Truth API

use ai_memory_wiki::{Wiki, WritePageRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize wiki at data_dir/wiki (Git repo) and data_dir/db (SQLite)
    let wiki = Wiki::new(std::path::Path::new("/data/wiki"), writer_handle).await?;

    // Atomic write: updates both markdown file AND SQLite index
    let req = WritePageRequest::new("architecture.md", "# System Design\nDetails here...");

    let page_id = wiki.write_page(req).await?;
    
    // Read directly from markdown source (truth)
    let content = wiki.read_page("default", "main", "architecture.md")?;
    
    // Recover from any corruption by rebuilding index from files
    wiki.rebuild_store_from_wiki().await?;
    
    Ok(())
}

Why This Matters for AI Memory Systems

The markdown-as-source-of-truth approach solves several problems common to AI-assisted knowledge management:

• Vendor independence: Your data remains readable without the application
• Diff and merge: Git-native versioning for collaborative editing
• Recoverability: No complex backup strategy—git clone preserves everything
• Transparency: Inspect and modify memory with standard tools

Summary

• Markdown files in wiki/ are authoritative; SQLite in db/ is a derived cache
• Atomic writes through Wiki::write_page prevent divergence between layers
• Automatic rollback ensures failed database updates never corrupt the file state
• Git integration provides immutable version history at the source layer
• rebuild_store_from_wiki guarantees recoverability from any index corruption

Frequently Asked Questions

What happens if the SQLite database is deleted?

The database can be fully reconstructed by running Wiki::rebuild_store_from_wiki(). This function walks the wiki/ directory tree, parses every Markdown file, and rebuilds all indexes—including vector embeddings—from scratch. No data is lost because the SQLite store contains only derived data.

Can I edit Markdown files directly with a text editor?

Yes. Direct edits to files in the wiki/ directory are valid and encouraged. However, the application must be notified to re-index (or you must run rebuild_store_from_wiki) for those changes to appear in search results. The Markdown files remain the truth regardless of index state.

How does ai-memory handle write failures?

If the SQLite upsert fails during write_page, the function rolls back the Markdown file write before returning the error. This is verified by the test write_page_rolls_back_file_when_store_upsert_fails.rs, which injects store failures and confirms the file system remains unchanged.

Why use Markdown instead of a database as the primary storage?

Markdown provides human readability, Git-native versioning, and zero lock-in. As stated in the design documents, treating "markdown in a git repo [as] source of truth" ensures the memory system remains portable, transparent, and recoverable without specialized tooling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}