How ai-memory Adjusts Search Relevance Using Frontmatter Tags and Page Kind

ai-memory adjusts search relevance by applying configurable multipliers to FTS5 base scores when frontmatter tags or page kind attributes match query parameters, implemented through a dedicated retrieval tuning layer in the Rust-based storage engine.

The ai-memory repository implements a semantic search system that goes beyond basic full-text matching by interpreting structured metadata embedded in Markdown frontmatter. This approach allows the engine to prioritize results based on categorical tags and document types, creating a context-aware ranking system that surfaces the most semantically relevant pages first.

The Role of Retrieval Tuning in Search Ranking

At the core of ai-memory's relevance adjustment lies the RetrievalTuning struct defined in crates/ai-memory-store/src/retrieval_tuning.rs. This configuration object stores the multipliers used to modify base FTS5 scores:

pub struct RetrievalTuning {
    pub abstract_vectors: bool,     // enable vector-based boost
    pub tag_boost: f32,             // multiplier for tag matches
    pub kind_boost: f32,            // multiplier for kind matches
    pub title_boost: f32,           // extra weight for title matches
    pub recency_decay: Option<f32>, // optional time-decay factor
}

The system applies these boosts after SQLite's FTS5 engine computes initial relevance scores. According to the source code in crates/ai-memory-store/src/fts_query.rs, the ranking workflow multiplies the base rank by specific factors when frontmatter fields align with query intent.

How Frontmatter Tags Boost Relevance

The tags field in page frontmatter directly influences final ranking through the apply_tag_boost function in crates/ai-memory-store/src/fts_query.rs. When a page contains tags that match query terms, the system increases the relevance score proportionally.

The implementation works as follows:

1. The system parses the frontmatter["tags"] array from each result
2. For every tag matching a query term, it applies the tuning.tag_boost multiplier
3. Pages with multiple matching tags receive compounded boosts

This means a page with a lower raw FTS5 score but containing a matching "security" tag can outrank a page with higher text similarity but no relevant tags. The boost is applied as a multiplicative factor to the base rank:

let mut rank = base_rank;
rank *= tuning.tag_boost * (matched_tags as f32);

How Page Kind Influences Ranking

The kind frontmatter field categorizes pages into semantic types such as "fact", "rule", or "decision". The apply_kind_boost routine in crates/ai-memory-store/src/fts_query.rs compares frontmatter["kind"] against the query-derived kind, scaling the rank by tuning.kind_boost when they match.

Queries can specify kind explicitly (e.g., kind:rule) or implicitly through path inference. This ensures that when users search for procedural documentation, "rule" pages surface ahead of generic "fact" pages regardless of raw text similarity.

Derived Kind from File Paths

When pages lack explicit kind declarations, ai-memory derives the kind from the file path. The derive_kind_from_path helper in crates/ai-memory-wiki/src/wiki.rs analyzes directory structures—for example, files under decisions/ automatically receive kind = "decision" before storage. This derived kind participates in the same boost logic, maintaining consistent ranking behavior across legacy content.

The Complete Relevance Calculation Workflow

The relevance adjustment follows a deterministic four-stage pipeline:

1. FTS5 ranking – SQLite's FTS5 engine computes the initial full-text match score based on term frequency and proximity.

2. Retrieval tuning initialization – The ReaderPool::set_retrieval_tuning method stores the RetrievalTuning configuration, typically loaded from CLI arguments in crates/ai-memory-cli/src/commands/serve.rs:

   // crates/ai-memory-cli/src/commands/serve.rs
   store.reader.set_retrieval_tuning(config.retrieval.tuning());

3. Boost application – In fts_query.rs, after calculating the base rank, the system sequentially applies tag and kind multipliers:

   if page_kind == query_kind { 
       rank *= tuning.kind_boost; 
   }

4. Final ordering – The boosted scores sort in descending order before returning to API endpoints in crates/ai-memory-mcp/src/server.rs and the web interface.

Configuring and Using Relevance Tuning

To leverage these relevance adjustments, developers must configure the tuning parameters and structure content with appropriate frontmatter.

Setting Retrieval Tuning via CLI

When starting the server, pass retrieval tuning configuration to the store:

// crates/ai-memory-cli/src/commands/serve.rs
store.reader.set_retrieval_tuning(RetrievalTuning {
    tag_boost: 1.5,
    kind_boost: 2.0,
    title_boost: 1.2,
    abstract_vectors: true,
    recency_decay: Some(0.95),
});

Creating Boosted Content

Structure pages with explicit tags and kind declarations to maximize discoverability:

{
  "path": "knowledge/rules/example.md",
  "frontmatter": {
    "title": "Example Rule",
    "kind": "rule",
    "tags": ["security", "policy"]
  },
  "body": "All users must authenticate using two-factor authentication..."
}

Executing Optimized Queries

Client code benefits automatically from these adjustments when calling the search API:

let query = "security policy";
let results = reader.search(query, None).await?;

The search implementation in reader.rs invokes the FTS query followed by the tag and kind boost logic, ensuring pages containing the "security" tag or with kind = "rule" receive appropriate rank elevation.

Summary

• ai-memory combines FTS5 full-text scoring with multiplicative boosts derived from frontmatter metadata to adjust search relevance.
• The tags array in frontmatter triggers score multiplication via apply_tag_boost in crates/ai-memory-store/src/fts_query.rs, with each matching tag increasing the final rank.
• The kind field (or path-derived kind via derive_kind_from_path in crates/ai-memory-wiki/src/wiki.rs) enables categorical prioritization through apply_kind_boost.
• The RetrievalTuning struct in crates/ai-memory-store/src/retrieval_tuning.rs provides configurable multipliers (tag_boost, kind_boost) that operators adjust at runtime.
• This architecture ensures semantic relevance takes precedence over simple term frequency, surfacing contextually appropriate results.

Frequently Asked Questions

How does ai-memory handle pages without explicit kind declarations?

When a page lacks a kind field in its frontmatter, the system invokes derive_kind_from_path in crates/ai-memory-wiki/src/wiki.rs to infer the kind from the directory structure. For instance, files stored under decisions/ automatically receive kind = "decision". This derived kind then participates in the same boost calculations as explicitly declared kinds, ensuring consistent ranking across the entire document corpus.

Can tag boosts override FTS5 relevance completely?

Tag boosts serve as multipliers rather than replacement scores, but with high tag_boost values (e.g., 2.0 or greater), pages with matching tags can significantly outrank pages with higher raw FTS5 scores but no tag matches. The system applies the boost after FTS5 ranking in crates/ai-memory-store/src/fts_query.rs, creating a hybrid score that balances textual similarity with metadata relevance.

Where is the retrieval tuning configuration stored and modified?

The RetrievalTuning struct instances are stored in the ReaderPool and set via set_retrieval_tuning methods. Configuration typically originates in the CLI layer (crates/ai-memory-cli/src/commands/serve.rs) where command-line arguments or config files populate the struct fields, then propagate to the storage layer at runtime. This allows operators to adjust relevance weightings without modifying source code or reindexing content.

Does ai-memory support decaying relevance based on content age?

Yes, the RetrievalTuning struct includes an optional recency_decay field of type Option<f32>. When configured, this factor applies a time-based decay multiplier to the relevance score, causing newer pages to rank higher than older ones. This decay applies alongside tag and kind boosts in the final ranking calculation, allowing temporal relevance to influence results alongside semantic metadata.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}