How ai‑memory Handles Authority Adjustment in Retrieval for Different Content Types

ai‑memory applies a bounded page‑authority multiplier to raw SQLite FTS5 relevance scores, allowing high‑authority content types to outrank lower‑credibility matches even when their textual similarity is slightly lower.

The akitaonrails/ai-memory repository implements a hybrid retrieval system that fuses full‑text search with source credibility metrics. By storing authority metadata in the database schema and applying multipliers during query execution, the system can privilege maintained knowledge bases over transient session logs or auto‑generated snippets.

The Authority Adjustment Mechanism

ai‑memory stores an authority column directly in the FTS5 index, populated from each page’s front‑matter metadata. Tags such as author:high or author:low in YAML front‑matter translate into numeric multipliers (stored as Option<f64>) that scale the raw FTS rank.

When a page lacks authority metadata, the system defaults to a neutral factor of 1.0, leaving the original rank unchanged (line 412 of reader.rs). This ensures backward compatibility with unannotated content while allowing opt‑in credibility weighting.

The Five‑Step Retrieval Pipeline

The authority adjustment logic operates inside the search_fused method (around line 1287 in crates/ai-memory-store/src/reader.rs). The pipeline executes as follows:

1. Full‑text query – SQLite’s FTS5 engine produces an initial ranked list based on textual relevance.

2. Candidate expansion – The helper authority_candidate_limit(limit) expands the result window by up to 1,300 extra rows (lines 7629‑7635). This bounded expansion ensures that high‑authority pages sitting just outside the top‑N textual matches are still considered, preventing the authority boost from sacrificing recall.

3. Authority lookup – For each candidate hit, the system retrieves the corresponding authority row from the index (lines 3760‑3762).

4. Rank fusion – The raw hit.rank is multiplied by the authority factor via authority.adjust_rank(hit.rank) (lines 191‑211). This produces a fused score combining lexical similarity and source credibility.

5. Final ordering – Results are re‑sorted by the adjusted rank, surfacing high‑authority pages that might otherwise be buried.

Configuring Authority for Different Content Types

Different content types inject distinct authority tags via their front‑matter definitions. The crates/ai-memory-wiki/src/wiki.rs module persists these tags into the FTS authority column during indexing.

• Maintained documentation might carry author:high or version:stable tags, receiving multipliers such as 2.0.
• Transient session evidence or LLM‑generated snippets might default to author:low or omit tags entirely, retaining the 1.0 neutral weight.
• JSON API responses can inject computed authority scores based on endpoint reliability.

Although the multiplier values differ by content type, the adjustment logic itself is uniform: every candidate passes through the same adjust_rank() function regardless of source format.

Implementation Details in reader.rs

The core retrieval logic resides in crates/ai-memory-store/src/reader.rs. Key implementation characteristics include:

• Bounded overhead – The expansion window caps at 1,300 additional rows for requests with limit = usize::MAX, preventing unbounded memory growth during authority lookups.
• Null safety – Authority is modeled as Option<f64>, with None mapping to the identity multiplier 1.0.
• Normalization helpers – The authority_tags and normalize_authority_tag utilities (lines 162‑173) validate and sanitize front‑matter values before they reach the retrieval layer.

The crates/ai-memory-mcp/src/server.rs module (line 386) exposes these authority‑adjusted rankings through the MCP API, ensuring that downstream consumers receive results ordered by credibility‑weighted relevance.

Practical Example: Performing Authority‑Adjusted Searches

The following Rust example demonstrates how to execute a search that automatically applies authority multipliers:

use ai_memory_store::Reader;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let reader = Reader::new().await?;
    // Returns top 10 hits with authority-adjusted rankings
    let hits = reader.search_fused("rust", 10).await?;
    
    for hit in hits {
        println!(
            "Page: {} | Adjusted rank: {:.3} | Authority: {:.2}",
            hit.page_path,
            hit.rank,
            hit.authority.unwrap_or(1.0)
        );
    }
    Ok(())
}

For manual authority application outside the standard pipeline:

let authority = Authority::new(2.0); // Boost weight by 2×
let raw_rank = 0.75;
let adjusted = authority.adjust_rank(raw_rank); // 0.375 after inverse weighting

Summary

• Authority column – Stored in the FTS5 index and populated from YAML front‑matter tags.
• Bounded expansion – authority_candidate_limit() prevents recall loss by buffering up to 1,300 extra candidates.
• Uniform application – The adjust_rank() multiplier (lines 191‑211) applies consistently across markdown, JSON, and snippet content types.
• Default neutrality – Pages without authority metadata receive a 1.0 multiplier, preserving raw FTS ordering.

Frequently Asked Questions

What happens if a page has no authority metadata?

The system assigns a default authority factor of 1.0, meaning the original FTS rank remains unchanged (as implemented at line 412 of reader.rs). This design ensures that legacy or unannotated content continues to function without schema migration.

How does ai‑memory prevent authority boosts from degrading search recall?

The retrieval layer uses a bounded candidate window. The authority_candidate_limit() helper expands the initial result set by a maximum of 1,300 rows (lines 7629‑7635) before applying multipliers. This ceiling guarantees that high‑authority pages slightly outside the top‑N textual matches are considered, but the system does not retrieve the entire corpus for every query.

Can different content types receive different authority multipliers?

Yes. While the adjustment logic is uniform, the actual multiplier values originate from content‑specific front‑matter tags managed in crates/ai-memory-wiki/src/wiki.rs. Markdown documentation, JSON API responses, and LLM‑generated snippets can each declare distinct author: or priority: tags, which normalize into different numeric factors stored in the FTS authority column.

Where is the authority adjustment logic implemented?

The primary implementation resides in crates/ai-memory-store/src/reader.rs, specifically within the search_fused() method (around line 1287) and the Authority::adjust_rank() helper (lines 191‑211). Candidate expansion logic appears at lines 162‑173 and lines 7629‑7635.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}