Understanding the Core Data Structure of ai-memory: A Karpathy-Style LLM Wiki Implementation

The core data structure of ai-memory is the Page struct, a typed and versioned markdown record that implements Andrej Karpathy's "compile-not-retrieve" pattern through immutable versioning, content-addressed storage, and tier-based lifecycle management.

The ai-memory repository (akitaonrails/ai-memory) implements the Karpathy-style LLM wiki as a durable knowledge base for AI agents. At its foundation lies a strictly typed data model that treats every piece of knowledge as a compiled, versioned artifact rather than a mutable document. This article explores the Page struct and its supporting infrastructure that enables the compile-first workflow.

The Page Struct: Versioned Markdown at the Core

The heart of the system is the Page struct defined in crates/ai-memory-core/src/page.rs. This structure represents a single markdown page together with rich metadata required for the compile-not-retrieve workflow. Unlike simple file storage, the Page embeds cryptographic integrity checks, temporal metadata, and explicit version chaining.

Immutable Identity and Namespacing

Every page carries a stable id: PageId that serves as its immutable identifier across versions. The struct supports multi-tenant isolation through workspace_id: WorkspaceId and project_id: ProjectId, which mirror the directory layout <data_dir>/wiki/<workspace>/<project>/<path>. This namespacing ensures that agents operating in different contexts cannot accidentally contaminate knowledge bases.

The path: PagePath field stores the relative wiki path (e.g., src/lib.rs.md), providing a hierarchical namespace that aligns with the on-disk representation. Combined with the title: String field for human-readable display, these identifiers create a dual-naming system: stable machine IDs for references and semantic paths for human navigation.

Content Integrity and Frontmatter

To guarantee that compiled artifacts never drift from their indexed representations, the Page stores both body: String containing the raw markdown and body_sha256: [u8; 32] for cryptographic verification. This checksum ensures integrity during the consolidation pipeline.

The frontmatter: serde_json::Value field holds structured metadata including tags, expiration dates, and custom attributes. This implements the Karpathy-wiki metadata pattern where properties like kind, expires_at, and tier travel with the document itself, making each page self-describing.

Version Chains and Supersession

Versioning is explicit rather than implicit through the is_latest: bool flag and supersedes: Option<PageId> field. When an agent updates knowledge, the system creates a new Page record and links it to its predecessor via supersedes, implementing the "supersede-instead-of-append" rule. This creates an immutable audit trail where historical versions remain accessible while the latest version is clearly marked.

Additional lifecycle controls include pinned: bool to prevent decay, along with created_at, updated_at, and expires_at timestamps that drive the automated forgetting mechanisms.

The Wiki Facade: Enforcing the Compile-Not-Retrieve Contract

While Page defines the data structure, the Wiki struct in crates/ai-memory-wiki/src/wiki.rs provides the operational facade that enforces Karpathy's compile-first principles. This struct owns the root filesystem path and coordinates between on-disk storage and the SQLite index.

Atomic Write Operations

The Wiki maintains a WriterHandle that guarantees atomicity across the storage boundary. When writing, the write_page method (defined in crates/ai-memory-wiki/src/wiki.rs at line 86) performs three operations in a single step:

1. Serializes the Page to markdown with frontmatter
2. Writes the file to the filesystem at the computed path
3. Pushes an upsert command to the SQLite store

This atomicity ensures there is no "background indexing" lag—queries immediately see consistent data because the compilation happens synchronously with persistence.

Git Integration and Storage

The Wiki includes a GitAdapter for version-controlled history, ensuring that even after SQLite compaction, the full audit trail remains accessible through git. The admission chain in crates/ai-memory-wiki/src/admission.rs sanitizes incoming writes to guarantee that every write respects the compilation principles before touching disk.

The Karpathy-Style Workflow Implementation

The data structures support a four-phase workflow that distinguishes ai-memory from traditional RAG systems.

Capture Phase

Agents emit observations as Sanitized<NewObservation> events that get wrapped in WritePageRequest structures. These requests carry the raw content and metadata but have not yet entered the canonical store.

Compile Phase

The consolidation pipeline in crates/ai-memory-consolidate/src/consolidator.rs transforms observations into Page records. This compilation step digests raw observations into markdown artifacts with computed checksums and validated frontmatter. Once compiled, the Wiki::write_page method materializes the page at <data_dir>/wiki/<workspace>/<project>/<path>.

Decay and Forgetting

The Tier enum (with variants Working, Episodic, Semantic, and Procedural) classifies pages by retention priority. The sweep logic in crates/ai-memory-consolidate/src/sweep.rs uses these tiers along with the expires_at and pinned fields to prune low-signal pages while preserving high-value knowledge. This implements the M8-style forgetting mechanism without breaking version chains.

Query Phase

Searchers retrieve pages exclusively through the SQLite index (utilizing FTS5, entity graphs, and optional embeddings) but never re-retrieve from raw observations. They read the compiled, versioned Page artifacts only, ensuring that queries always see the sanitized, canonical form of knowledge.

Practical Implementation

The following Rust code demonstrates the core data structure usage:

use ai_memory_wiki::Wiki;
use ai_memory_core::{PagePath, WritePageRequest};
use std::path::Path;

// Initialize the Wiki facade rooted at the data directory
let data_dir = Path::new("/var/lib/ai-memory");
let writer = WriterHandle::new(...);
let wiki = Wiki::new(data_dir, writer)?;

// Compile and write a new page
let req = WritePageRequest {
    workspace_id,
    project_id,
    path: PagePath::new("src/lib.rs.md"),
    title: "ai-memory Core Library".into(),
    body: "# ai-memory\n\nRust core library for the LLM wiki.".into(),

    frontmatter: serde_json::json!({ 
        "tier": "semantic",
        "kind": "documentation" 
    }),
    ..Default::default()
};
wiki.write_page(req).await?;

// Retrieve the compiled artifact
let page = wiki.read_page(
    workspace_id, 
    project_id, 
    PagePath::new("src/lib.rs.md")
)?;
assert!(page.is_latest);
assert_eq!(page.title, "ai-memory Core Library");

The Page fields ensure data integrity throughout this process:

// Verify content hasn't been corrupted since compilation
use sha2::{Sha256, Digest};

let mut hasher = Sha256::new();
hasher.update(page.body.as_bytes());
let result = hasher.finalize();
assert_eq!(result.as_slice(), &page.body_sha256);

Summary

• The Page struct in crates/ai-memory-core/src/page.rs is the immutable, versioned unit of knowledge containing markdown content, SHA-256 checksums, and temporal metadata.
• Version chaining via supersedes and is_latest fields creates an immutable history while maintaining a clear "latest" pointer.
• The Wiki struct in crates/ai-memory-wiki/src/wiki.rs enforces atomic compile-not-retrieve semantics across filesystem and SQLite stores.
• Tier-based classification (Working, Episodic, Semantic, Procedural) drives automated decay in the sweep pipeline without destroying version history.
• All writes pass through the admission chain in crates/ai-memory-wiki/src/admission.rs to guarantee frontmatter validity and checksum integrity.

Frequently Asked Questions

What makes ai-memory's Page structure different from a simple markdown file?

The Page struct embeds a content-addressed SHA-256 checksum (body_sha256) and explicit version chaining (supersedes) that simple files lack. According to the source code in crates/ai-memory-core/src/page.rs, it also carries tier classifications and expiration metadata that enable automated lifecycle management, transforming static documents into managed knowledge artifacts with integrity guarantees.

How does ai-memory handle versioning without breaking existing references?

The system uses immutable version chains where each Page receives a unique PageId and optionally references its predecessor via supersedes: Option<PageId>. The is_latest: bool flag marks the current head of the chain, allowing queries to resolve "latest" while preserving historical references to specific versions. This structure, implemented in crates/ai-memory-core/src/page.rs, ensures that links to specific PageId values remain valid even as knowledge evolves.

What is the role of the Tier enum in the Page struct?

The Tier enum classifies pages into Working, Episodic, Semantic, or Procedural categories that drive the decay and retention policies in crates/ai-memory-consolidate/src/sweep.rs. This classification allows the system to aggressively prune temporary working memory while preserving semantic knowledge and procedures, implementing a tiered forgetting mechanism where pinned: true pages are exempt from sweeps regardless of tier.

Where are the compiled pages physically stored on disk?

Pages materialize at <data_dir>/wiki/<workspace>/<project>/<path> as maintained by the Wiki struct in crates/ai-memory-wiki/src/wiki.rs. The WriterHandle in crates/ai-memory-store/src/writer.rs ensures that markdown files and their SQLite index entries are written atomically to this location, with the GitAdapter providing version control for the filesystem layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}