How the Forget Sweep Mechanism Works in ai-memory for Expiring Pages

The forget sweep is a three-stage retention pass that removes obsolete wiki pages through TTL expiration, decay-based eviction, and hard-deletion of tombstones.

The forget sweep mechanism in the akitaonrails/ai-memory repository provides automatic lifecycle management for AI-generated wiki pages. Implemented primarily in the consolidation crate, this mechanism periodically evaluates page freshness, recency, and access patterns to decide what content should be expired, evicted, or permanently purged from the store.

The Three-Stage Architecture

The sweep operates as a coordinated pipeline across three distinct stages, each targeting different categories of obsolete content.

Stage 1: TTL Pass (Time-To-Live Expiration)

The first stage performs an immediate TTL check on every latest page candidate. In crates/ai-memory-consolidate/src/sweep.rs (lines 11-15), the algorithm scans rows where is_latest = 1 and inspects the optional front-matter key expires_at.

If the current timestamp exceeds the page's expiration time (c.expires_at_us ≤ now_us), the page is immediately marked for deletion regardless of its tier or pin status. This check appears at lines 61-73, where detected expired pages are collected into the expired vector for subsequent hard deletion.

Stage 2: Decay Pass (Retention Score Eviction)

For pages that survive the TTL check, the mechanism evaluates decay eligibility. This stage only considers episodic pages that are not pinned, as determined by the is_decayable function (lines 30-43 in sweep.rs).

Eligible pages receive a retention score computed by retention_score_with_breadth (lines 78-89), which weights age, access count, last-access gap, salience, and optional access-breadth coefficients. Pages scoring below params.cold_threshold are added to the eviction list. When dry_run is disabled, these pages are removed via wiki.evict_page_if_latest (lines 75-99), which deletes the Markdown file and writes a decay tombstone record.

Stage 3: Hard-Delete Pass (Tombstone Cleanup)

The final stage performs permanent cleanup of decay tombstones older than params.hard_delete_after_days. The cutoff calculation occurs at lines 60-64, followed by retrieval via reader.decay_tombstones_before. Each qualifying tombstone is permanently removed through wiki.hard_delete_decay_tombstone (lines 101-119), ensuring storage reclamation while maintaining an audit trail during the grace period.

Key Implementation Details

Candidate Collection and Scoring

The sweep begins with reader.decay_candidates(workspace_id, project_id) (line 53), which returns all latest rows for the specified project. For each candidate, the mechanism optionally calculates access breadth weighting through access_breadth_for_scoring (lines 98-103), skipping the extra query when the coefficient is zero to optimize performance.

Wiki Layer Integration

All destructive operations delegate to the wiki crate for atomic file system consistency. TTL deletions use wiki.delete_page_if_latest (lines 102-119), while decay evictions use wiki.evict_page_if_latest (lines 138-146). These methods ensure that SQLite row deletions remain synchronized with the underlying Markdown file removals.

Idempotent Reporting

Every sweep operation produces a SweepReport struct that tracks evaluated candidates, evicted pages, expired pages, and hard-deleted tombstones. This design ensures operations are idempotent—re-running the sweep with identical parameters produces the same result set without double-deleting content.

Code Examples

Running a Sweep Programmatically

Implement the sweep in Rust using the ai-memory-consolidate crate:

use ai_memory_store::{ReaderPool, WriterHandle, DecayParams};
use ai_memory_wiki::Wiki;
use ai_memory_core::{WorkspaceId, ProjectId};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize store connections
    let reader: ReaderPool = /* ... */;
    let writer: WriterHandle = /* ... */;
    let wiki: Wiki = /* ... */;

    // Configure retention parameters
    let params = DecayParams {
        cold_threshold: 0.2,
        decay_rate: 0.01,
        hard_delete_after_days: 30,
        ..Default::default()
    };

    // Execute dry-run to preview deletions
    let dry_report = ai_memory_consolidate::run_sweep(
        &reader,
        &writer,
        Some(&wiki),
        WorkspaceId::new(),
        ProjectId::new(),
        &params,
        true,  // dry_run enabled
    ).await?;

    println!("Preview: {} expired, {} evicted", 
             dry_report.expired.len(), 
             dry_report.evicted.len());

    // Execute actual sweep
    let live_report = ai_memory_consolidate::run_sweep(
        &reader,
        &writer,
        Some(&wiki),
        WorkspaceId::new(),
        ProjectId::new(),
        &params,
        false,  // dry_run disabled
    ).await?;

    println!("Deleted {} pages total", 
             live_report.expired.len() + live_report.evicted.len());
    Ok(())
}

Command-Line Interface

Execute sweeps directly from the terminal using the ai-memory binary:

# Dry-run sweep for specific workspace and project

ai-memory sweep --workspace 1 --project 42 --dry-run

# Live execution with default retention parameters

ai-memory sweep --workspace 1 --project 42

Summary

• The forget sweep operates in three sequential stages: TTL expiration, decay-based eviction, and hard-delete cleanup.
• TTL checks occur at lines 61-73 in sweep.rs, immediately identifying pages with expired expires_at metadata.
• Decay scoring at lines 78-89 evaluates episodic, non-pinned pages using retention_score_with_breadth and the cold_threshold parameter.
• Hard-deletion of tombstones follows the grace period defined by hard_delete_after_days, implemented at lines 101-119.
• All operations are idempotent and report results through the SweepReport struct, with file system operations delegated to the wiki crate for consistency.

Frequently Asked Questions

What is the difference between TTL expiration and decay eviction?

TTL expiration removes pages with explicit expiration timestamps set in their front-matter, regardless of access patterns or pin status. Decay eviction removes episodic (non-pinned) pages that have fallen below the retention score threshold due to age and low access frequency. TTL is deterministic based on wall-clock time, while decay is heuristic based on usage patterns.

How does the dry_run parameter affect sweep execution?

When dry_run is set to true, the sweep executes all candidate detection and scoring logic but skips the actual file deletions and database mutations. The returned SweepReport contains the lists of pages that would be deleted, allowing administrators to preview retention impacts before committing changes.

When are decay tombstones permanently hard-deleted?

Tombstones remain in the database until they exceed the hard_delete_after_days threshold defined in DecayParams. The sweep calculates a cutoff timestamp (lines 60-64) and removes only tombstones created before that date through wiki.hard_delete_decay_tombstone, preserving a grace period for potential data recovery.

Can pinned pages be removed by the forget sweep?

No. The is_decayable function (lines 30-43) explicitly filters out pages where is_pinned = 1. Pinned pages bypass the decay scoring stage entirely and can only be removed through TTL expiration (if they have an explicit expires_at date) or manual deletion outside the sweep mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}