How to Audit Contamination and Resolve Duplicate or Conflicting Page Entries in ai-memory

Run ai-memory audit-contamination to generate a read-only AuditReport identifying stale content, duplicate titles, and orphan pages, then resolve issues by deleting duplicates via delete-page <path> and re-running the audit to verify cleanup.

The ai-memory system persists observations, pages, and metadata in a SQLite-backed store. Over time, concurrent writes, bulk imports, and copy operations can introduce structural inconsistencies that corrupt retrieval quality. Understanding how to audit contamination and resolve duplicate or conflicting page entries in ai-memory is essential for maintaining a reliable knowledge base that supports accurate LLM-driven processing.

Understanding Contamination Categories

The contamination audit classifies structural issues into three distinct categories. In crates/ai-memory-store/src/reader.rs, the ReaderPool::audit_contamination method returns an AuditReport containing these contamination types:

• Stale pages: Content whose latest generation is older than the newest generation in the project. These pages may contain outdated information that no longer reflects the current state of your knowledge base.

• Duplicates: Multiple latest pages sharing the same title front-matter. Only one page per title is considered canonical; the rest are flagged as contamination.

• Orphans: Pages with no incoming links from any other page in the graph. These unreachable nodes often result from failed ingestion jobs or deleted references.

The audit is advisory only and performs no mutations. This read-only safety guarantee prevents accidental data loss during inspection.

Running a Contamination Audit

You can execute the audit through three entry points, all of which forward requests to ReaderPool::audit_contamination with optional scope and home directory filtering.

Programmatic Access via ReaderPool

For Rust applications, access the store directly:

use ai_memory_store::ReaderPool;
use ai_memory_store::ScopeResolver;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let resolver = ScopeResolver::new("my_workspace", "my_project");
    let reader = ReaderPool::new().await?;
    
    // Run audit with optional scope and home_dir filtering
    let report = reader.audit_contamination(Some(resolver), None).await?;
    
    println!("Stale pages: {}", report.stale.len());
    println!("Duplicate titles: {}", report.duplicates.len());
    
    for dup in report.duplicates {
        println!("Title: {}", dup.title);
        for page in dup.pages {
            println!("  - {} (id={})", page.path, page.id);
        }
    }
    Ok(())
}

CLI Command

The crates/ai-memory-cli/src/commands/audit_contamination.rs implementation provides the primary interface:

# Audit entire workspace

ai-memory audit-contamination

# Audit specific project with scope

ai-memory audit-contamination --scope my_workspace/my_project --home-dir /home/alice

MCP Admin API

The handle_audit_contamination handler in crates/ai-memory-mcp/src/admin.rs exposes an HTTP endpoint:

POST /admin/audit_contamination HTTP/1.1
Content-Type: application/json

{
  "scope": { "workspace": "my_workspace", "project": "my_project" },
  "home_dir": "/home/alice"
}

The response returns JSON with stale, duplicates, and orphans arrays.

Web Health Endpoint

The web interface in crates/ai-memory-web/src/routes/api.rs exposes contamination data via GET /api/v1/health, which includes a duplicate_pages array in the health payload for monitoring purposes.

Resolving Duplicate and Conflicting Entries

After obtaining the AuditReport, follow this resolution workflow:

1. Identify the Canonical Page

Review each duplicate set in the report, which lists page IDs and titles. Select the canonical page based on recency, content richness, or manual relevance assessment.

2. Delete Duplicate Pages

Remove non-canonical entries using the command implemented in crates/ai-memory-cli/src/commands/delete_page.rs:

ai-memory delete-page notes/old-duplicate.md

Alternatively, use the MCP endpoint:

DELETE /api/v1/page/notes/old-duplicate.md

Deletion automatically updates the link graph, removing inbound and outbound edges.

3. Merge Content if Necessary

If duplicates contain unique valuable information, open the canonical page using ai-memory wiki edit <path> and manually incorporate content from the removed duplicates before deletion.

4. Verify Cleanup

Re-run the audit to confirm the duplicates count is zero:

ai-memory audit-contamination

5. Address Stale Pages

Resolve stale content by triggering an auto-improve run:

ai-memory auto-improve

This creates a new generation for stale pages. Alternatively, manually edit the page to increment its generation.

6. Handle Orphan Pages

Either delete orphans if they are truly unused, or integrate them by adding inbound links from relevant pages. This restores their position in the graph and removes them from the orphan category.

Root Causes of Duplicate Entries

Understanding why contamination occurs helps prevent future issues:

• Concurrent writes: When two agents write to the same title simultaneously, each receives a unique page ID, creating duplicates.

• Importer behavior: Bulk import tools such as ai-memory-importer may not deduplicate titles before insertion.

• On-conflict policies: Using MCP copy or move endpoints with on_conflict=duplicate creates new page slots to preserve both contents, potentially leaving duplicate titles when the target already contains a page with the same title.

Summary

• Audit contamination using ReaderPool::audit_contamination in crates/ai-memory-store/src/reader.rs to identify stale, duplicate, and orphan pages without modifying data.
• Resolve duplicates by identifying canonical pages, deleting extras via ai-memory delete-page, and merging content when necessary.
• Clear stale pages by running ai-memory auto-improve to generate fresh content.
• Fix orphans by deleting dead pages or adding inbound links to reintegrate them into the graph.
• Re-verify by running the audit again to ensure contamination counts reach zero.

Frequently Asked Questions

How does ai-memory detect duplicate pages?

The audit queries the SQLite store for pages sharing identical title front-matter values. In crates/ai-memory-store/src/reader.rs, the audit_contamination method compares generation timestamps and link graphs to classify one page per title as canonical and flag the rest as duplicates.

Is the contamination audit safe to run on production stores?

Yes. The audit is strictly read-only and never mutates the underlying store. It only generates an AuditReport describing stale, duplicate, and orphan conditions. All destructive actions require explicit commands such as delete-page or API delete requests.

What is the difference between stale and orphan pages?

Stale pages contain content older than the project's newest generation but remain linked within the graph. Orphan pages have no incoming links from other pages, making them unreachable during graph traversal, regardless of their generation age. A page can theoretically be both stale and orphaned.

Can I automate duplicate resolution?

Currently, resolution requires manual review to identify the canonical page and decide whether to merge content. While the CLI and API provide the deletion mechanisms in crates/ai-memory-cli/src/commands/delete_page.rs, the decision logic for canonical selection and content merging remains a human-in-the-loop process to prevent accidental data loss.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}