ai-memory Data Directory Structure: A Complete Guide to `<data_dir>` Layout and Organization

The ai-memory data directory uses a six-folder layout with wiki/ as the single source of truth, db/memory.sqlite as the searchable index, plus raw/, logs/, models/, and client-projects.json for work-streams, logging, future models, and client metadata.

The <data_dir> directory in the akitaonrails/ai-memory repository serves as the central on-disk workspace for all persistent state. Understanding this structure is essential for configuring deployments, debugging synchronization issues, or extending the system with custom tooling. This guide breaks down each component based on the actual source code implementation.

Top-Level Directory Structure

The ai-memory data directory contains six top-level subdirectories and several configuration files. All paths are resolved relative to a root data_dir passed to core components at initialization.

Sub-path	Purpose	Source Implementation
wiki/	Markdown source of truth; every page lives here and is version-controlled by git2	crates/ai-memory-wiki/src/wiki.rs
db/memory.sqlite	SQLite WAL database holding searchable index, page versions, sessions, handoffs, and embeddings	crates/ai-memory-store/src/lib.rs
raw/	Immutable, sanitized JSONL segments for managed work-streams	crates/ai-memory-hooks/src/workstream.rs
logs/	Daily-rolling tracing logs from server and CLI	crates/ai-memory-mcp/src/admin.rs
models/	Reserved folder for bundled embedding models (future local-model support)	docs/design-decisions.md
client-projects.json	Private JSON mapping of client-local checkout links	crates/ai-memory-mcp/src/admin.rs

Configuration Files in <data_dir>

Beyond the folder structure, these auxiliary files live at the root:

• config.toml — Global configuration overridable by environment variables
• auth.json — Stored provider refresh tokens for LLM authentication
• auth-token / auth-header — Files used by the native hook command for authentication

The wiki/ Folder: Single Source of Truth

The wiki/ directory is the only mutable source of truth in the entire ai-memory data directory structure. All other components derive their state from here.

According to the architecture, the SQLite database and raw work-stream files must stay synchronized with wiki/ contents—the system treats them as disposable rebuildable artifacts, not primary data.

use ai_memory_wiki::Wiki;
use ai_memory_store::WriterHandle;
use std::path::Path;

let data_dir = Path::new("/my/custom/data_dir");
let writer = WriterHandle::new()?;               // single-writer SQLite actor
let wiki = Wiki::new(data_dir, writer)?;         // roots at <data_dir>/wiki

The Wiki::new() constructor in crates/ai-memory-wiki/src/wiki.rs handles atomic markdown writes and ensures git2-backed version control for every page operation.

The db/memory.sqlite Database: Derived Search Index

The SQLite database at db/memory.sqlite uses WAL mode for concurrent readers and a single writer. It contains:

• Full-text search index
• Page version history
• Session and handoff records
• Vector embeddings

use ai_memory_store::Store;
use std::path::Path;

let data_dir = Path::new("/my/custom/data_dir");
let store = Store::open(data_dir)?;               // opens <data_dir>/db/memory.sqlite

The Store::open() function in crates/ai-memory-store/src/lib.rs enforces a single-writer actor model. Multiple processes cannot write simultaneously—this prevents corruption and guarantees consistency with the wiki/ folder.

The raw/ Directory: Immutable Work-Stream Segments

The raw/ folder stores immutable, sanitized JSONL segments for managed work-streams. These files serve two purposes:

1. Replayable observations — reconstruct any prior system state
2. Legacy fallback searches — operate when the SQLite index is rebuilding

// Writing work-stream segments happens in ai-memory-hooks
// See crates/ai-memory-hooks/src/workstream.rs for implementation

The workstream implementation in crates/ai-memory-hooks/src/workstream.rs appends to these files atomically—once written, segments are never modified or deleted, only rotated.

The logs/ Directory: Operational Visibility

Daily-rolling tracing logs from both server and CLI components:

use tracing::info;
use std::env;

let log_dir = std::env::var("AI_MEMORY_DATA_DIR")
    .unwrap_or_else(|_| "./data".into());
info!(target: "ai_memory", "server started – logs in {}/logs", log_dir);

Log path handling is centralized in crates/ai-memory-mcp/src/admin.rs alongside other administrative operations.

The models/ Folder: Future Local Embeddings

Currently reserved for bundled embedding models. The docs/design-decisions.md document outlines plans for local model support, eliminating external LLM dependencies for certain operations.

The client-projects.json File: Client Metadata

This private JSON file maintains checkout links between the central ai-memory instance and client-local working directories. It is not part of the public API and should not be manually edited.

use std::fs;
use std::path::Path;

let data_dir = Path::new("/my/custom/data_dir");
let json_path = data_dir.join("client-projects.json");
let contents = fs::read_to_string(json_path)?;
println!("Client projects: {}", contents);

Managed by crates/ai-memory-mcp/src/admin.rs alongside authentication and configuration files.

Architecture Invariants and Design Principles

The ai-memory data directory structure enforces these core invariants:

• Single source of truth — Only wiki/ holds authoritative data; all other stores are derived
• Immutable work-streams — raw/ segments are append-only, enabling perfect reproducibility
• Single-writer database — SQLite operations are serialized through the WriterHandle actor
• Git-native versioning — Markdown files in wiki/ use libgit2 for automatic version tracking

These constraints enable reliable backups (just wiki/ and config.toml), clean disaster recovery, and deterministic testing.

Customizing the ai-memory Data Directory Location

All components accept a data_dir: PathBuf parameter. The system respects the AI_MEMORY_DATA_DIR environment variable as a default fallback when no explicit path is provided.

Component	Initialization Function	Data Directory Usage
Wiki	Wiki::new(data_dir, writer)	Creates <data_dir>/wiki
Store	Store::open(data_dir)	Opens <data_dir>/db/memory.sqlite
MCP Admin	Internal path resolution	Manages config.toml, auth.json, client-projects.json

Summary

• The ai-memory <data_dir> contains six top-level elements: wiki/, db/memory.sqlite, raw/, logs/, models/, and client-projects.json
• wiki/ is the sole source of truth — all other data can be reconstructed from markdown files
• db/memory.sqlite provides fast search via single-writer SQLite with WAL mode
• raw/ stores immutable JSONL work-stream segments for replay and fallback
• Configuration and auth files (config.toml, auth.json, auth-token, auth-header) live at the root
• All paths resolve through a data_dir: PathBuf passed to Wiki::new(), Store::open(), and administrative functions

Frequently Asked Questions

Where is the actual content stored in ai-memory?

All user content lives in the wiki/ folder as markdown files. The SQLite database at db/memory.sqlite only holds derived indexes and embeddings for fast searching. If you back up only one folder, make it wiki/.

Can I move the ai-memory data directory to a different location?

Yes. Pass any PathBuf to Wiki::new() or Store::open(), or set the AI_MEMORY_DATA_DIR environment variable. All subdirectories are created relative to your specified path. The models/ folder and configuration files will initialize automatically on first run.

What happens if db/memory.sqlite becomes corrupted?

Delete it and restart ai-memory. The system will rebuild the entire database from the wiki/ markdown source. The raw/ work-stream segments are similarly rebuildable. This design makes corruption recovery straightforward—your actual data in wiki/ remains safe.

Is client-projects.json required for operation?

No. This file is optional and used only when connecting client-local checkouts to a central ai-memory instance. If absent, the system operates in standalone mode. The file is managed automatically by the MCP admin component and should not be manually edited.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}