How Entity Extraction and Entity-Assisted Recall Improve Search Relevance in ai-memory

Entity extraction and entity-assisted recall improve search relevance by extracting up to 10 salient nouns from each page's front-matter and fusing them with full-text search results via Reciprocal Rank Fusion, surfacing relevant content even when query terms don't match body text exactly.

The akitaonrails/ai-memory project implements a lightweight semantic search layer that leverages explicit entity tagging to boost precision and recall. Unlike pure vector similarity or brittle keyword matching, this hybrid approach uses a curated list of entities stored in Markdown front-matter to bridge the gap between lexical and conceptual similarity.

Understanding the Entity Index Architecture

Every page in ai-memory is stored as Markdown with a typed front-matter section. One critical field is entities:, a YAML list containing up to 10 high-signal nouns that represent the page's core concepts.

Front-Matter Parsing with parse_entities

When pages are written or consolidated, the system invokes the parse_entities helper in [crates/ai-memory-wiki/src/wiki.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs#L1886) to extract the noun list from the front-matter. This function guarantees that the entity list round-trips unchanged through the write-path, ensuring consistency between the stored Markdown and the search index.

During consolidation—handled by **[consolidator.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/consolidator.rs)**—the extracted entities are stored alongside the page's canonical representation. Because this extraction happens at write-time, it requires zero runtime LLM inference during queries, making the system deterministic and performant.

The FTS5 and Entity Hybrid Index

The storage layer maintains two complementary indices:

• FTS5 index: A classic full-text search index over the page body that handles generic keyword matches and stemming.
• Entity column: A dedicated field storing the parsed entities: list as a compact, language-agnostic signal.

This dual-index strategy allows the query engine to compare user queries against both the full body text and the curated entity list simultaneously.

The Reciprocal Rank Fusion Query Pipeline

When a user issues a search, the query pipeline—implemented primarily in **[fts_query.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/fts_query.rs)**—constructs a Reciprocal Rank Fusion (RRF) stream that merges multiple relevance signals before applying final ranking rules.

Combining Lexical and Entity Signals

The RRF algorithm combines three distinct scoring streams:

1. Pure lexical matches: Standard FTS5 hits against the page body text.
2. Entity-match scores: Query terms are compared against stored entities: lists. Because entities are short, high-precision nouns, a page receives a strong relevance boost when the query matches its salient concepts, even if the exact wording doesn't appear in the body.
3. Optional vector similarity: If an embedding provider is configured, semantic similarity scores are incorporated into the fusion.

By merging these streams, ai-memory achieves semantic recall without exclusively relying on expensive vector operations. A search for "executors" will match a page tagged with the entity executor even if the body text only discusses "task schedulers" or "job runners."

Rule-Based Re-Ranking

After RRF fusion, the pipeline applies a set of rule-based weights defined in [reader.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs#L399). This post-processing step adjusts scores based on page metadata:

• Pages under _rules/, decisions/, or procedures/ paths receive explicit boosts.
• Tier, pin status, tag relevance, and graph-neighbor influences further refine the ranking.

The matched_entities field exposed in the search results indicates exactly which entities contributed to a page's ranking, providing transparency for debugging relevance issues.

Practical Implementation Examples

Defining Entities in Page Front-Matter

Authors specify entities directly in the Markdown front-matter. The parse_entities function then extracts these during the consolidation phase:

---
title: "Rust async tutorial"
entities:
  - async
  - future
  - tokio
  - executor
---

# Introduction

This guide covers cooperative multitasking using the Tokio runtime...

In this example, a query for "executors" will match the page via the entity list, even though the body text uses "runtime" instead of the exact term "executor."

Querying via CLI and Rust API

Command-line search:

# Finds pages tagged with 'executor' even if body text uses different terminology

ai-memory query "executors"

Programmatic access via the Rust API:

use ai_memory_store::{Reader, Query};

let reader = Reader::new(&conn)?;
let results = reader.search(Query {
    text: "executor".into(),
    // Additional filters omitted for brevity
})?;

for hit in results {
    println!("{} – score {}", hit.page_path, hit.rank);
    // hit.matched_entities reveals which entities boosted this result
}

The query engine automatically handles the RRF fusion between FTS5 hits and entity matches, returning a relevance-ordered result set without requiring manual intervention.

Summary

• Front-matter extraction: Up to 10 salient nouns per page are parsed by parse_entities in wiki.rs and stored during consolidation.
• Hybrid ranking: The RRF pipeline merges FTS5 lexical matches, entity-assisted signals, and optional vector similarity for robust relevance scoring.
• Zero inference cost: Entity matching uses pre-computed lexical comparisons, eliminating runtime LLM calls and reducing latency compared to pure semantic search.
• Transparent scoring: The matched_entities field in reader.rs exposes exactly which concepts boosted each result, aiding in search debugging and optimization.

Frequently Asked Questions

How does entity-assisted recall differ from vector similarity search?

Entity-assisted recall uses explicit noun lists extracted at write-time, while vector similarity relies on embedding models to compute semantic relationships. According to the ai-memory source code, entity matching is deterministic, requires no runtime LLM inference, and works offline, whereas vector similarity depends on external embedding providers. The two approaches complement each other in the RRF pipeline.

Why limit entities to 10 nouns per page?

The constraint ensures that the entity signal remains high-precision and compact. In wiki.rs, the parse_entities function enforces this limit to prevent entity lists from becoming noisy, low-signal keyword blobs. This restriction maintains fast index lookups and prevents dilution of the relevance boost during the RRF scoring phase.

Can I search for pages that match specific entities but not the query text?

Yes. The search pipeline in fts_query.rs treats entity matches as a separate scoring stream from FTS5 lexical matches. A page containing the entity tokio will receive relevance boosts for queries including "tokio," "async runtime," or "executor" even if those exact strings don't appear in the body text, provided the entities align with the query intent.

How are entity lists maintained when pages are updated?

During the consolidation process handled by consolidator.rs, the system re-parses the front-matter using parse_entities and updates the canonical page representation. This ensures that the entity index always reflects the current front-matter state. The round-trip guarantees in wiki.rs ensure that manual edits to the entities: field are preserved exactly as written.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}