How Embedding Backfill Works for Existing Wiki Pages in ai-memory

The embedding backfill process scans wiki pages for missing or stale vector embeddings, deduplicates against existing vectors, generates embeddings in batches of 100, and writes them atomically via a single-writer handle.

The ai-memory repository maintains semantic search capabilities by ensuring every wiki page has an up-to-date vector embedding. When content changes or embeddings are missing entirely, the embedding backfill process automatically generates these vectors while providing granular control over re-embedding, dry runs, and error handling.

The Core Backfill Workflow

The implementation resides in crates/ai-memory-consolidate/src/embed.rs. The run_embedding_backfill function executes an eight-step pipeline that processes candidate pages efficiently and safely.

Candidate Discovery with Decay Detection

The process begins by identifying "decayed" content—pages modified since their last embedding. The ReaderPool::decay_candidates method queries the target workspace and project (lines 93-94):

let candidates = reader.decay_candidates(workspace_id, project_id).await?;

This returns all pages requiring updated vector representations.

Deduplication and Option Handling

Before processing, the system checks for existing embeddings to avoid redundant work. Unless options.reembed is true, it builds a HashSet of page IDs already embedded for the specific (provider, model, dim) configuration (lines 94-108):

let already: HashSet<_> = if options.reembed {
    HashSet::new()
} else {
    reader.embedded_page_ids(...).await?.into_iter().collect()
};

During iteration, the logic at lines 14-17 skips any page ID present in this set, incrementing the skipped counter.

Content Validation and Dry-Run Mode

The backfill iterates through candidates with three validation layers:

1. Dry-run protection: If options.dry_run is enabled, the system increments would_embed and continues without generating vectors (lines 18-21).
2. Content retrieval: The system fetches markdown via Wiki::read_page. Read failures increment the failed counter but do not abort the batch (lines 22-28).
3. Empty content filter: Pages with trimmed empty bodies are skipped (lines 30-33).

Vector Generation and Batched Persistence

Valid pages proceed to the configured Embedder. Provider errors during generation are logged and counted as failed without terminating the loop (lines 34-41).

Successful embeddings queue in a pending vector. Once the batch reaches EMBEDDING_WRITE_BATCH (100 items) or the candidate list exhausts, flush_embedding_batch commits the data via the single-writer WriterHandle (lines 42-53):

pending.push(EmbeddingWrite { … });
if pending.len() >= EMBEDDING_WRITE_BATCH {
    flush_embedding_batch(writer, &mut pending, &mut counts).await;
}

Triggering the Backfill Process

The system exposes the backfill through three distinct interfaces:

Administrative HTTP API

The POST /admin/embed endpoint defined in crates/ai-memory-mcp/src/admin.rs (lines 19-52) allows external systems to initiate backfills on demand.

Command-Line Interface

The ai-memory embed command in crates/ai-memory-cli/src/commands/embed.rs supports interactive operation:

# Generate embeddings only for pages lacking vectors

ai-memory embed

# Force regeneration of existing embeddings

ai-memory embed --force

# Preview candidate count without writing

ai-memory embed --dry-run

Scheduled Maintenance and Startup

Two automated mechanisms ensure continuous consistency:

• Server startup: Wiki::backfill_scope_manifests in crates/ai-memory-wiki/src/wiki.rs (lines 1403-1405) runs once to verify scope-manifest files exist.
• Maintenance ticks: run_scheduled_embedding_backfill_tick in crates/ai-memory-cli/src/commands/serve.rs (lines 1306-1319) executes periodic backfills according to the configured schedule.

Error Handling and Statistics Collection

The EmbedBackfillCounts struct (lines 25-38) tracks five metrics: embedded, skipped, failed, and would_embed. This allows callers to report precise summary statistics after completion.

Error handling distinguishes between transient and fatal failures:

• Non-fatal errors: Read failures and provider API errors are logged locally and increment the failed counter. The process continues with remaining candidates.
• Fatal errors: Only store-level failures (e.g., SQLite write errors) propagate as EmbedBackfillError and abort the entire operation (lines 77-80).

Programmatic Integration

Embed the backfill in Rust applications using the ai-memory-consolidate crate:

use ai_memory_consolidate::{run_embedding_backfill, EmbedBackfillOptions};
use std::sync::Arc;

let opts = EmbedBackfillOptions { reembed: false, dry_run: false };
let counts = run_embedding_backfill(
    &reader,
    &writer,
    &wiki,
    &Arc::new(my_embedder),
    workspace_id,
    project_id,
    opts,
).await?;

println!("Results: {} embedded, {} skipped, {} failed", 
    counts.embedded, counts.skipped, counts.failed);

Summary

• The backfill identifies stale content via ReaderPool::decay_candidates in crates/ai-memory-consolidate/src/embed.rs, targeting pages changed since their last embedding.
• Deduplication against existing (provider, model, dim) tuples prevents redundant work unless the reembed option overrides this behavior.
• Embeddings commit in batches of 100 (EMBEDDING_WRITE_BATCH) through a single-writer WriterHandle to prevent database contention.
• Three triggers exist: the POST /admin/embed HTTP endpoint, the ai-memory embed CLI command, and the run_scheduled_embedding_backfill_tick maintenance scheduler.
• The error handling strategy isolates storage failures (which abort) from provider or read errors (which log and continue), ensuring robust batch processing.

Frequently Asked Questions

What defines a "decayed" wiki page candidate?

A decayed page is any wiki page whose content modification timestamp exceeds the timestamp of its last vector embedding. The decay_candidates method in crates/ai-memory-consolidate/src/embed.rs (lines 93-94) queries precisely these records, ensuring the backfill targets only genuinely stale content rather than reprocessing unchanged pages.

How does the system prevent duplicate embeddings?

By default, the backfill constructs a HashSet of embedded_page_ids for the current embedding configuration (provider, model, dim) and skips any candidate appearing in this set. To force regeneration—useful when switching embedding models or correcting corrupted vectors—set the reembed option to true or use the CLI --force flag.

What happens when the embedding provider returns an error?

The system treats provider errors as localized failures. The error is logged, the failed counter increments, and the loop continues to the next candidate. Only SQLite storage errors propagate as EmbedBackfillError and terminate the batch. This design prevents transient API outages from stalling the entire backfill queue.

Can I estimate processing time without generating embeddings?

Yes. Enable dry_run mode in EmbedBackfillOptions or pass --dry-run to the CLI. This mode traverses all candidates, performs deduplication checks, and counts potential work in the would_embed field without invoking the embedder or writing to the database, providing accurate capacity planning metrics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}