# How the Forget Sweep Mechanism Works in ai-memory for Expiring Pages

> Understand the forget sweep mechanism in ai-memory for expiring pages. Learn how TTL expiration, decay eviction, and tombstone deletion remove obsolete wiki content.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: internals
- Published: 2026-08-21

---

**The forget sweep is a three-stage retention pass that removes obsolete wiki pages through TTL expiration, decay-based eviction, and hard-deletion of tombstones.**

The **forget sweep mechanism** in the `akitaonrails/ai-memory` repository provides automatic lifecycle management for AI-generated wiki pages. Implemented primarily in the consolidation crate, this mechanism periodically evaluates page freshness, recency, and access patterns to decide what content should be expired, evicted, or permanently purged from the store.

## The Three-Stage Architecture

The sweep operates as a coordinated pipeline across three distinct stages, each targeting different categories of obsolete content.

### Stage 1: TTL Pass (Time-To-Live Expiration)

The first stage performs an immediate **TTL check** on every *latest* page candidate. In [`crates/ai-memory-consolidate/src/sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/sweep.rs) (lines 11-15), the algorithm scans rows where `is_latest = 1` and inspects the optional front-matter key `expires_at`. 

If the current timestamp exceeds the page's expiration time (`c.expires_at_us ≤ now_us`), the page is immediately marked for deletion regardless of its tier or pin status. This check appears at lines 61-73, where detected expired pages are collected into the `expired` vector for subsequent hard deletion.

### Stage 2: Decay Pass (Retention Score Eviction)

For pages that survive the TTL check, the mechanism evaluates **decay eligibility**. This stage only considers *episodic* pages that are not pinned, as determined by the `is_decayable` function (lines 30-43 in [`sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/sweep.rs)).

Eligible pages receive a **retention score** computed by `retention_score_with_breadth` (lines 78-89), which weights age, access count, last-access gap, salience, and optional access-breadth coefficients. Pages scoring below `params.cold_threshold` are added to the eviction list. When `dry_run` is disabled, these pages are removed via `wiki.evict_page_if_latest` (lines 75-99), which deletes the Markdown file and writes a decay tombstone record.

### Stage 3: Hard-Delete Pass (Tombstone Cleanup)

The final stage performs **permanent cleanup** of decay tombstones older than `params.hard_delete_after_days`. The cutoff calculation occurs at lines 60-64, followed by retrieval via `reader.decay_tombstones_before`. Each qualifying tombstone is permanently removed through `wiki.hard_delete_decay_tombstone` (lines 101-119), ensuring storage reclamation while maintaining an audit trail during the grace period.

## Key Implementation Details

### Candidate Collection and Scoring

The sweep begins with `reader.decay_candidates(workspace_id, project_id)` (line 53), which returns all latest rows for the specified project. For each candidate, the mechanism optionally calculates access breadth weighting through `access_breadth_for_scoring` (lines 98-103), skipping the extra query when the coefficient is zero to optimize performance.

### Wiki Layer Integration

All destructive operations delegate to the wiki crate for atomic file system consistency. TTL deletions use `wiki.delete_page_if_latest` (lines 102-119), while decay evictions use `wiki.evict_page_if_latest` (lines 138-146). These methods ensure that SQLite row deletions remain synchronized with the underlying Markdown file removals.

### Idempotent Reporting

Every sweep operation produces a **SweepReport** struct that tracks evaluated candidates, evicted pages, expired pages, and hard-deleted tombstones. This design ensures operations are idempotent—re-running the sweep with identical parameters produces the same result set without double-deleting content.

## Code Examples

### Running a Sweep Programmatically

Implement the sweep in Rust using the `ai-memory-consolidate` crate:

```rust
use ai_memory_store::{ReaderPool, WriterHandle, DecayParams};
use ai_memory_wiki::Wiki;
use ai_memory_core::{WorkspaceId, ProjectId};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize store connections
    let reader: ReaderPool = /* ... */;
    let writer: WriterHandle = /* ... */;
    let wiki: Wiki = /* ... */;

    // Configure retention parameters
    let params = DecayParams {
        cold_threshold: 0.2,
        decay_rate: 0.01,
        hard_delete_after_days: 30,
        ..Default::default()
    };

    // Execute dry-run to preview deletions
    let dry_report = ai_memory_consolidate::run_sweep(
        &reader,
        &writer,
        Some(&wiki),
        WorkspaceId::new(),
        ProjectId::new(),
        &params,
        true,  // dry_run enabled
    ).await?;

    println!("Preview: {} expired, {} evicted", 
             dry_report.expired.len(), 
             dry_report.evicted.len());

    // Execute actual sweep
    let live_report = ai_memory_consolidate::run_sweep(
        &reader,
        &writer,
        Some(&wiki),
        WorkspaceId::new(),
        ProjectId::new(),
        &params,
        false,  // dry_run disabled
    ).await?;

    println!("Deleted {} pages total", 
             live_report.expired.len() + live_report.evicted.len());
    Ok(())
}

```

### Command-Line Interface

Execute sweeps directly from the terminal using the `ai-memory` binary:

```bash

# Dry-run sweep for specific workspace and project

ai-memory sweep --workspace 1 --project 42 --dry-run

# Live execution with default retention parameters

ai-memory sweep --workspace 1 --project 42

```

## Summary

- The forget sweep operates in three sequential stages: **TTL expiration**, **decay-based eviction**, and **hard-delete cleanup**.
- **TTL checks** occur at lines 61-73 in [`sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/sweep.rs), immediately identifying pages with expired `expires_at` metadata.
- **Decay scoring** at lines 78-89 evaluates episodic, non-pinned pages using `retention_score_with_breadth` and the `cold_threshold` parameter.
- **Hard-deletion** of tombstones follows the grace period defined by `hard_delete_after_days`, implemented at lines 101-119.
- All operations are **idempotent** and report results through the `SweepReport` struct, with file system operations delegated to the wiki crate for consistency.

## Frequently Asked Questions

### What is the difference between TTL expiration and decay eviction?

**TTL expiration** removes pages with explicit expiration timestamps set in their front-matter, regardless of access patterns or pin status. **Decay eviction** removes *episodic* (non-pinned) pages that have fallen below the retention score threshold due to age and low access frequency. TTL is deterministic based on wall-clock time, while decay is heuristic based on usage patterns.

### How does the `dry_run` parameter affect sweep execution?

When `dry_run` is set to `true`, the sweep executes all candidate detection and scoring logic but skips the actual file deletions and database mutations. The returned `SweepReport` contains the lists of pages that *would* be deleted, allowing administrators to preview retention impacts before committing changes.

### When are decay tombstones permanently hard-deleted?

Tombstones remain in the database until they exceed the `hard_delete_after_days` threshold defined in `DecayParams`. The sweep calculates a cutoff timestamp (lines 60-64) and removes only tombstones created before that date through `wiki.hard_delete_decay_tombstone`, preserving a grace period for potential data recovery.

### Can pinned pages be removed by the forget sweep?

No. The `is_decayable` function (lines 30-43) explicitly filters out pages where `is_pinned = 1`. Pinned pages bypass the decay scoring stage entirely and can only be removed through TTL expiration (if they have an explicit `expires_at` date) or manual deletion outside the sweep mechanism.