# How Embedding Backfill Works for Existing Wiki Pages in ai-memory

> Discover how ai-memory's embedding backfill efficiently updates wiki pages. Learn about scanning, deduplication, batch generation, and atomic writes for seamless vector management.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-25

---

**The embedding backfill process scans wiki pages for missing or stale vector embeddings, deduplicates against existing vectors, generates embeddings in batches of 100, and writes them atomically via a single-writer handle.**

The ai-memory repository maintains semantic search capabilities by ensuring every wiki page has an up-to-date vector embedding. When content changes or embeddings are missing entirely, the embedding backfill process automatically generates these vectors while providing granular control over re-embedding, dry runs, and error handling.

## The Core Backfill Workflow

The implementation resides in [`crates/ai-memory-consolidate/src/embed.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/embed.rs). The `run_embedding_backfill` function executes an eight-step pipeline that processes candidate pages efficiently and safely.

### Candidate Discovery with Decay Detection

The process begins by identifying "decayed" content—pages modified since their last embedding. The `ReaderPool::decay_candidates` method queries the target workspace and project (lines 93-94):

```rust
let candidates = reader.decay_candidates(workspace_id, project_id).await?;

```

This returns all pages requiring updated vector representations.

### Deduplication and Option Handling

Before processing, the system checks for existing embeddings to avoid redundant work. Unless `options.reembed` is `true`, it builds a `HashSet` of page IDs already embedded for the specific `(provider, model, dim)` configuration (lines 94-108):

```rust
let already: HashSet<_> = if options.reembed {
    HashSet::new()
} else {
    reader.embedded_page_ids(...).await?.into_iter().collect()
};

```

During iteration, the logic at lines 14-17 skips any page ID present in this set, incrementing the *skipped* counter.

### Content Validation and Dry-Run Mode

The backfill iterates through candidates with three validation layers:

1.  **Dry-run protection**: If `options.dry_run` is enabled, the system increments `would_embed` and continues without generating vectors (lines 18-21).
2.  **Content retrieval**: The system fetches markdown via `Wiki::read_page`. Read failures increment the *failed* counter but do not abort the batch (lines 22-28).
3.  **Empty content filter**: Pages with trimmed empty bodies are skipped (lines 30-33).

### Vector Generation and Batched Persistence

Valid pages proceed to the configured `Embedder`. Provider errors during generation are logged and counted as *failed* without terminating the loop (lines 34-41).

Successful embeddings queue in a `pending` vector. Once the batch reaches `EMBEDDING_WRITE_BATCH` (100 items) or the candidate list exhausts, `flush_embedding_batch` commits the data via the single-writer `WriterHandle` (lines 42-53):

```rust
pending.push(EmbeddingWrite { … });
if pending.len() >= EMBEDDING_WRITE_BATCH {
    flush_embedding_batch(writer, &mut pending, &mut counts).await;
}

```

## Triggering the Backfill Process

The system exposes the backfill through three distinct interfaces:

### Administrative HTTP API

The `POST /admin/embed` endpoint defined in [`crates/ai-memory-mcp/src/admin.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/admin.rs) (lines 19-52) allows external systems to initiate backfills on demand.

### Command-Line Interface

The `ai-memory embed` command in [`crates/ai-memory-cli/src/commands/embed.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/embed.rs) supports interactive operation:

```bash

# Generate embeddings only for pages lacking vectors

ai-memory embed

# Force regeneration of existing embeddings

ai-memory embed --force

# Preview candidate count without writing

ai-memory embed --dry-run

```

### Scheduled Maintenance and Startup

Two automated mechanisms ensure continuous consistency:

-   **Server startup**: `Wiki::backfill_scope_manifests` in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) (lines 1403-1405) runs once to verify scope-manifest files exist.
-   **Maintenance ticks**: `run_scheduled_embedding_backfill_tick` in [`crates/ai-memory-cli/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/serve.rs) (lines 1306-1319) executes periodic backfills according to the configured schedule.

## Error Handling and Statistics Collection

The `EmbedBackfillCounts` struct (lines 25-38) tracks five metrics: `embedded`, `skipped`, `failed`, and `would_embed`. This allows callers to report precise summary statistics after completion.

Error handling distinguishes between transient and fatal failures:

-   **Non-fatal errors**: Read failures and provider API errors are logged locally and increment the `failed` counter. The process continues with remaining candidates.
-   **Fatal errors**: Only store-level failures (e.g., SQLite write errors) propagate as `EmbedBackfillError` and abort the entire operation (lines 77-80).

## Programmatic Integration

Embed the backfill in Rust applications using the `ai-memory-consolidate` crate:

```rust
use ai_memory_consolidate::{run_embedding_backfill, EmbedBackfillOptions};
use std::sync::Arc;

let opts = EmbedBackfillOptions { reembed: false, dry_run: false };
let counts = run_embedding_backfill(
    &reader,
    &writer,
    &wiki,
    &Arc::new(my_embedder),
    workspace_id,
    project_id,
    opts,
).await?;

println!("Results: {} embedded, {} skipped, {} failed", 
    counts.embedded, counts.skipped, counts.failed);

```

## Summary

-   The backfill identifies stale content via `ReaderPool::decay_candidates` in [`crates/ai-memory-consolidate/src/embed.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/embed.rs), targeting pages changed since their last embedding.
-   Deduplication against existing `(provider, model, dim)` tuples prevents redundant work unless the `reembed` option overrides this behavior.
-   Embeddings commit in batches of 100 (`EMBEDDING_WRITE_BATCH`) through a single-writer `WriterHandle` to prevent database contention.
-   Three triggers exist: the `POST /admin/embed` HTTP endpoint, the `ai-memory embed` CLI command, and the `run_scheduled_embedding_backfill_tick` maintenance scheduler.
-   The error handling strategy isolates storage failures (which abort) from provider or read errors (which log and continue), ensuring robust batch processing.

## Frequently Asked Questions

### What defines a "decayed" wiki page candidate?

A decayed page is any wiki page whose content modification timestamp exceeds the timestamp of its last vector embedding. The `decay_candidates` method in [`crates/ai-memory-consolidate/src/embed.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/embed.rs) (lines 93-94) queries precisely these records, ensuring the backfill targets only genuinely stale content rather than reprocessing unchanged pages.

### How does the system prevent duplicate embeddings?

By default, the backfill constructs a `HashSet` of `embedded_page_ids` for the current embedding configuration `(provider, model, dim)` and skips any candidate appearing in this set. To force regeneration—useful when switching embedding models or correcting corrupted vectors—set the `reembed` option to `true` or use the CLI `--force` flag.

### What happens when the embedding provider returns an error?

The system treats provider errors as localized failures. The error is logged, the `failed` counter increments, and the loop continues to the next candidate. Only SQLite storage errors propagate as `EmbedBackfillError` and terminate the batch. This design prevents transient API outages from stalling the entire backfill queue.

### Can I estimate processing time without generating embeddings?

Yes. Enable `dry_run` mode in `EmbedBackfillOptions` or pass `--dry-run` to the CLI. This mode traverses all candidates, performs deduplication checks, and counts potential work in the `would_embed` field without invoking the embedder or writing to the database, providing accurate capacity planning metrics.