How the Entity-Assisted Recall System Stores Canonical Entities in Page Frontmatter

The entity-assisted recall system persists canonical entities directly in YAML frontmatter as a list of structured EntityRef objects containing normalized names, kinds, and stable IDs, enabling entity-aware search and graph expansion at query time.

The ai-memory project implements a novel approach to semantic search by embedding structured entity metadata within wiki page frontmatter. This design—centered on the FrontMatter struct defined in crates/ai-memory-wiki/src/wiki.rs—allows the system to track which canonical entities appear on each page without requiring external databases or complex synchronization protocols.

How Canonical Entities Are Defined and Extracted

Canonical entities are those that pass a configurable threshold length and satisfy the is_canonical predicate during indexing. The system identifies these entities by scanning the markdown body of each page, normalizing their names to lowercase, and assigning stable identifiers.

The entity extraction logic resides around line 4060 of crates/ai-memory-wiki/src/wiki.rs, where entity_hits are collected and filtered for canonical status. This ensures that only meaningful, reusable entities—not incidental terms—are promoted to frontmatter storage.

The Frontmatter Structure for Entity Storage

When a page is written, the system constructs a FrontMatter struct (lines 1790–1835 in crates/ai-memory-wiki/src/wiki.rs). The entities field holds a Vec<EntityRef>, with each entry containing:

Field	Purpose
name	Normalized, lower-cased entity string as discovered
kind	Semantic category (e.g., person, project, topic, feature)
id	Stable EntityId hash guaranteeing cross-page identity

This structure enables the canonical entity principle: the first discovered form of an entity becomes its authoritative representation, and all subsequent mentions across the workspace reference the same id.

Example Frontmatter Output

---
title: "Design of the Recall Engine"
last_modified_by: "alice"
entities:
  - name: "ai-memory"
    kind: "project"
    id: "c2f9e7d5-a1b3-4f2c-9d1e-5a8b6c7d9e01"
  - name: "entity-assisted recall"
    kind: "feature"
    id: "e3b5c1a2-d4f6-7a9b-c0d1-e2f3a4b5c6d7"
---

Entity-Aware Retrieval Architecture

At query time, the entity-assisted recall system performs a multi-stage retrieval pipeline:

1. Entity extraction from the query text
2. Entity-match step that joins query entities against page frontmatter entities lists
3. IDF weighting to rank pages by entity rarity and relevance
4. Graph expansion to surface related pages sharing canonical entities

This pipeline is implemented in crates/ai-memory-web/src/routes/api.rs at lines 287–300, where the entity stream merges into the underlying FTS5 query.

Practical Usage Examples

Creating Pages with Canonical Entities (Rust API)

use ai_memory_wiki::Wiki;
use ai_memory_wiki::frontmatter::FrontMatter;

let wiki = Wiki::open("./wiki")?;
let mut fm = FrontMatter::default();
fm.title = Some("Introducing Entity-Assisted Recall".into());
fm.entities = vec![
    EntityRef::new("ai-memory", "project"),
    EntityRef::new("entity-assisted recall", "feature"),
];
wiki.write_page("notes/recall.md", fm, "# Content …")?;

Executing Entity-Aware Queries (CLI)

$ ai-memory query "How does entity-assisted recall work?" --entity-aware

The CLI dispatches to /api/v1/search, which runs the full entity-match pipeline and returns pages containing matching canonical entities.

Key Design Benefits of Frontmatter Storage

Storing canonical entities in frontmatter provides several architectural advantages:

• Git-native versioning — entity lists persist through standard version control
• Edit survival — re-indexing updates entities only when content changes
• Cross-system portability — YAML frontmatter is human-readable and tool-agnostic
• Source of truth — the entities list becomes the authoritative entity-stream for recall scoring

Summary

• The entity-assisted recall system stores canonical entities in page frontmatter via the FrontMatter struct in crates/ai-memory-wiki/src/wiki.rs
• Each EntityRef contains a normalized name, semantic kind, and stable id enabling cross-page identity
• Entity extraction occurs during indexing (around line 4060), with recall-time matching in api.rs (lines 287–300)
• Frontmatter-based storage provides Git versioning, edit resilience, and portable entity metadata

Frequently Asked Questions

What makes an entity "canonical" in ai-memory?

An entity becomes canonical when it exceeds a configurable length threshold and passes the is_canonical predicate during indexing. These filters eliminate noise from short or generic terms, ensuring only semantically meaningful concepts are promoted to frontmatter storage and used in entity-assisted recall.

How does the entity id maintain stability across pages?

The id field contains an EntityId hash generated from the entity's normalized name during its first discovery. Subsequent mentions of the same canonical entity—regardless of case variations or context—resolve to this same identifier, enabling consistent cross-page relationships and graph expansion.

Can canonical entities be modified or removed from frontmatter?

Entities are automatically maintained by the indexing system. Manual edits to the entities list in frontmatter will be overwritten on the next index operation unless the underlying page content changes. The authoritative source is always the entity extraction pipeline in wiki.rs, not hand-written frontmatter.

What happens if two pages reference the same canonical entity?

Both pages will contain identical EntityRef entries in their frontmatter, including the same id value. This duplication is intentional: it enables local entity-aware search without requiring a central lookup service, and powers the graph expansion phase where related pages are surfaced based on shared entity fingerprints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}