# Understanding the Core Data Structure of ai-memory: A Karpathy-Style LLM Wiki Implementation

> Explore the core data structure of ai-memory, the Page struct. Discover its typed markdown records, immutable versioning, and content-addressed storage for efficient LLM wiki implementation.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-21

---

**The core data structure of ai-memory is the `Page` struct, a typed and versioned markdown record that implements Andrej Karpathy's "compile-not-retrieve" pattern through immutable versioning, content-addressed storage, and tier-based lifecycle management.**

The ai-memory repository (akitaonrails/ai-memory) implements the Karpathy-style LLM wiki as a durable knowledge base for AI agents. At its foundation lies a strictly typed data model that treats every piece of knowledge as a compiled, versioned artifact rather than a mutable document. This article explores the `Page` struct and its supporting infrastructure that enables the compile-first workflow.

## The `Page` Struct: Versioned Markdown at the Core

The heart of the system is the **`Page` struct** defined in [`crates/ai-memory-core/src/page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs). This structure represents a single markdown page together with rich metadata required for the compile-not-retrieve workflow. Unlike simple file storage, the `Page` embeds cryptographic integrity checks, temporal metadata, and explicit version chaining.

### Immutable Identity and Namespacing

Every page carries a stable **`id: PageId`** that serves as its immutable identifier across versions. The struct supports multi-tenant isolation through **`workspace_id: WorkspaceId`** and **`project_id: ProjectId`**, which mirror the directory layout `<data_dir>/wiki/<workspace>/<project>/<path>`. This namespacing ensures that agents operating in different contexts cannot accidentally contaminate knowledge bases.

The **`path: PagePath`** field stores the relative wiki path (e.g., [`src/lib.rs.md`](https://github.com/akitaonrails/ai-memory/blob/main/src/lib.rs.md)), providing a hierarchical namespace that aligns with the on-disk representation. Combined with the **`title: String`** field for human-readable display, these identifiers create a dual-naming system: stable machine IDs for references and semantic paths for human navigation.

### Content Integrity and Frontmatter

To guarantee that compiled artifacts never drift from their indexed representations, the `Page` stores both **`body: String`** containing the raw markdown and **`body_sha256: [u8; 32]`** for cryptographic verification. This checksum ensures integrity during the consolidation pipeline.

The **`frontmatter: serde_json::Value`** field holds structured metadata including tags, expiration dates, and custom attributes. This implements the Karpathy-wiki metadata pattern where properties like `kind`, `expires_at`, and `tier` travel with the document itself, making each page self-describing.

### Version Chains and Supersession

Versioning is explicit rather than implicit through the **`is_latest: bool`** flag and **`supersedes: Option<PageId>`** field. When an agent updates knowledge, the system creates a new `Page` record and links it to its predecessor via `supersedes`, implementing the "supersede-instead-of-append" rule. This creates an immutable audit trail where historical versions remain accessible while the latest version is clearly marked.

Additional lifecycle controls include **`pinned: bool`** to prevent decay, along with **`created_at`**, **`updated_at`**, and **`expires_at`** timestamps that drive the automated forgetting mechanisms.

## The `Wiki` Facade: Enforcing the Compile-Not-Retrieve Contract

While `Page` defines the data structure, the **`Wiki` struct** in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) provides the operational facade that enforces Karpathy's compile-first principles. This struct owns the root filesystem path and coordinates between on-disk storage and the SQLite index.

### Atomic Write Operations

The `Wiki` maintains a **`WriterHandle`** that guarantees atomicity across the storage boundary. When writing, the `write_page` method (defined in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) at line 86) performs three operations in a single step:

1. Serializes the `Page` to markdown with frontmatter
2. Writes the file to the filesystem at the computed path
3. Pushes an upsert command to the SQLite store

This atomicity ensures there is no "background indexing" lag—queries immediately see consistent data because the compilation happens synchronously with persistence.

### Git Integration and Storage

The `Wiki` includes a **`GitAdapter`** for version-controlled history, ensuring that even after SQLite compaction, the full audit trail remains accessible through git. The admission chain in [`crates/ai-memory-wiki/src/admission.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/admission.rs) sanitizes incoming writes to guarantee that every write respects the compilation principles before touching disk.

## The Karpathy-Style Workflow Implementation

The data structures support a four-phase workflow that distinguishes ai-memory from traditional RAG systems.

### Capture Phase

Agents emit observations as `Sanitized<NewObservation>` events that get wrapped in `WritePageRequest` structures. These requests carry the raw content and metadata but have not yet entered the canonical store.

### Compile Phase

The consolidation pipeline in [`crates/ai-memory-consolidate/src/consolidator.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs) transforms observations into `Page` records. This compilation step digests raw observations into markdown artifacts with computed checksums and validated frontmatter. Once compiled, the `Wiki::write_page` method materializes the page at `<data_dir>/wiki/<workspace>/<project>/<path>`.

### Decay and Forgetting

The **`Tier` enum** (with variants `Working`, `Episodic`, `Semantic`, and `Procedural`) classifies pages by retention priority. The sweep logic in [`crates/ai-memory-consolidate/src/sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/sweep.rs) uses these tiers along with the `expires_at` and `pinned` fields to prune low-signal pages while preserving high-value knowledge. This implements the M8-style forgetting mechanism without breaking version chains.

### Query Phase

Searchers retrieve pages exclusively through the SQLite index (utilizing FTS5, entity graphs, and optional embeddings) but never re-retrieve from raw observations. They read the compiled, versioned `Page` artifacts only, ensuring that queries always see the sanitized, canonical form of knowledge.

## Practical Implementation

The following Rust code demonstrates the core data structure usage:

```rust
use ai_memory_wiki::Wiki;
use ai_memory_core::{PagePath, WritePageRequest};
use std::path::Path;

// Initialize the Wiki facade rooted at the data directory
let data_dir = Path::new("/var/lib/ai-memory");
let writer = WriterHandle::new(...);
let wiki = Wiki::new(data_dir, writer)?;

// Compile and write a new page
let req = WritePageRequest {
    workspace_id,
    project_id,
    path: PagePath::new("src/lib.rs.md"),
    title: "ai-memory Core Library".into(),
    body: "# ai-memory\n\nRust core library for the LLM wiki.".into(),

    frontmatter: serde_json::json!({ 
        "tier": "semantic",
        "kind": "documentation" 
    }),
    ..Default::default()
};
wiki.write_page(req).await?;

// Retrieve the compiled artifact
let page = wiki.read_page(
    workspace_id, 
    project_id, 
    PagePath::new("src/lib.rs.md")
)?;
assert!(page.is_latest);
assert_eq!(page.title, "ai-memory Core Library");

```

The `Page` fields ensure data integrity throughout this process:

```rust
// Verify content hasn't been corrupted since compilation
use sha2::{Sha256, Digest};

let mut hasher = Sha256::new();
hasher.update(page.body.as_bytes());
let result = hasher.finalize();
assert_eq!(result.as_slice(), &page.body_sha256);

```

## Summary

- The **`Page` struct** in [`crates/ai-memory-core/src/page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs) is the immutable, versioned unit of knowledge containing markdown content, SHA-256 checksums, and temporal metadata.
- **Version chaining** via `supersedes` and `is_latest` fields creates an immutable history while maintaining a clear "latest" pointer.
- The **`Wiki` struct** in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) enforces atomic compile-not-retrieve semantics across filesystem and SQLite stores.
- **Tier-based classification** (`Working`, `Episodic`, `Semantic`, `Procedural`) drives automated decay in the sweep pipeline without destroying version history.
- All writes pass through the **admission chain** in [`crates/ai-memory-wiki/src/admission.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/admission.rs) to guarantee frontmatter validity and checksum integrity.

## Frequently Asked Questions

### What makes ai-memory's Page structure different from a simple markdown file?

The `Page` struct embeds a **content-addressed SHA-256 checksum** (`body_sha256`) and explicit **version chaining** (`supersedes`) that simple files lack. According to the source code in [`crates/ai-memory-core/src/page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs), it also carries tier classifications and expiration metadata that enable automated lifecycle management, transforming static documents into managed knowledge artifacts with integrity guarantees.

### How does ai-memory handle versioning without breaking existing references?

The system uses **immutable version chains** where each `Page` receives a unique `PageId` and optionally references its predecessor via `supersedes: Option<PageId>`. The `is_latest: bool` flag marks the current head of the chain, allowing queries to resolve "latest" while preserving historical references to specific versions. This structure, implemented in [`crates/ai-memory-core/src/page.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/page.rs), ensures that links to specific `PageId` values remain valid even as knowledge evolves.

### What is the role of the Tier enum in the Page struct?

The **`Tier` enum** classifies pages into `Working`, `Episodic`, `Semantic`, or `Procedural` categories that drive the decay and retention policies in [`crates/ai-memory-consolidate/src/sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/sweep.rs). This classification allows the system to aggressively prune temporary working memory while preserving semantic knowledge and procedures, implementing a tiered forgetting mechanism where `pinned: true` pages are exempt from sweeps regardless of tier.

### Where are the compiled pages physically stored on disk?

Pages materialize at **`<data_dir>/wiki/<workspace>/<project>/<path>`** as maintained by the `Wiki` struct in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs). The `WriterHandle` in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) ensures that markdown files and their SQLite index entries are written atomically to this location, with the `GitAdapter` providing version control for the filesystem layer.