How the Forget Sweep Handles TTL Expiration Versus Cold Threshold Eviction in ai-memory

The forget sweep processes TTL expiration as unconditional hard deletes that override pin status, while cold-threshold eviction performs decay-based soft deletes that preserve tombstones and respect pin constraints.

The akitaonrails/ai-memory repository implements a daily consolidation process that reclaims storage by evaluating every latest page against two distinct removal criteria. Located in crates/ai-memory-consolidate/src/sweep.rs, the sweep orchestrates TTL expiration (time-based hard deletion) and cold-threshold eviction (retention-based soft deletion) through separate code paths that handle pinning, history preservation, and storage cleanup differently.

TTL Expiration: Unconditional Hard Deletion

The sweep treats TTL expiration as an explicit user directive to destroy content. When a page's front-matter contains an expires_at: timestamp that has passed, the sweep immediately flags the page for deletion regardless of its access patterns or pin status.

In crates/ai-memory-consolidate/src/sweep.rs (lines 61–73), the sweep checks c.expires_at_us <= now_us during candidate iteration. If true, the page is added to the expired vector and the loop continues to the next candidate. This short-circuit evaluation ensures TTL checks take precedence over retention scoring.

During the execution phase, TTL-expired pages trigger wiki.delete_page_if_latest, which:

• Removes the Markdown file from the wiki layer
• Erases corresponding rows from the SQLite store
• Proceeds even if the page is pinned (expiry outranks pins)

Failures are logged non-fatally, allowing retry on subsequent sweep runs.

Cold-Threshold Eviction: Decay-Based Soft Deletion

Cold-threshold eviction targets low-value episodic memory through a retention scoring system. Pages with scores below params.cold_threshold are evicted only if they pass decayability filters, creating a tombstone record instead of immediate obliteration.

Retention Scoring Mechanics

After filtering out non-decayable pages, the sweep calculates a retention score via retention_score_with_breadth. This function weights:

• Age of the page
• Access count and last-access gap
• Salience metadata
• Optional breadth coefficient (actor count data from access_breadth_for_scoring)

The resulting score determines whether the page merits retention or decay.

Decayability Constraints

Before scoring, the sweep invokes is_decayable (lines 30–43 in sweep.rs) to filter candidates. A page is not decayable if it is:

• A semantic page (as opposed to episodic)
• Marked with front-matter pinned: true
• Already pinned through other mechanisms

Only episodic, unpinned pages proceed to cold-threshold evaluation.

The Sweep Execution Flow

The forget sweep processes pages through a strict pipeline:

1. Candidate Collection – reader.decay_candidates returns all pages where is_latest = 1
2. TTL Precedence Check – If expires_at_us exists and is past due, the page joins the expired list immediately
3. Decayability Filter – is_decayable removes semantic and pinned pages from cold eviction consideration
4. Score Calculation – Eligible pages receive retention scores with optional breadth weighting
5. Threshold Test – Scores below params.cold_threshold trigger eviction (lines 89–99)

This sequence ensures TTL expiration always takes precedence, while cold eviction only applies to decayable, low-scoring content.

Post-Processing: Divergent Cleanup Actions

The sweep applies different storage operations based on the eviction reason:

TTL-Expired Pages:

• Invokes wiki.delete_page_if_latest to permanently remove files and database rows
• No tombstone is preserved; history is lost immediately
• Pinned status is ignored

Evicted (Cold) Pages:

• Invokes wiki.evict_page_if_latest to write a decay tombstone row marking the page obsolete
• Deletes the Markdown source but preserves history metadata
• Respects pin status (pinned pages never reach this stage)

Hard-Delete Pass: After processing TTL and cold evictions, the sweep optionally purges decay tombstones older than params.hard_delete_after_days, cleaning up auxiliary rows and superseded ancestry.

Configuring Decay Parameters in Practice

The sweep accepts DecayParams to tune both mechanisms:

use ai_memory_store::{DecayParams, ReaderPool, WriterHandle};
use ai_memory_wiki::Wiki;
use ai_memory_core::{WorkspaceId, ProjectId};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize storage and wiki interfaces
    let reader: ReaderPool = /* ... */;
    let writer: WriterHandle = /* ... */;
    let wiki: Wiki = /* ... */;

    // Configure sweep behavior
    let params = DecayParams {
        cold_threshold: 0.25,          // Retention score floor for eviction
        hard_delete_after_days: 30,    // Tombstone preservation window
        ..Default::default()
    };

    // Execute sweep (set dry_run false to apply changes)
    let report = ai_memory_consolidate::run_sweep(
        &reader,
        &writer,
        Some(&wiki),
        WorkspaceId::new(),
        ProjectId::new(),
        &params,
        false,
    )
    .await?;

    println!(
        "Sweep complete: {} evicted (cold), {} expired (TTL)",
        report.evicted.len(),
        report.expired.len()
    );
    Ok(())
}

The SweepReport returned by run_sweep separates results into expired (TTL) and evicted (cold) vectors, allowing downstream systems to distinguish between time-based and score-based removals.

Summary

• TTL expiration triggers hard deletes via wiki.delete_page_if_latest, removing files and database rows unconditionally, even for pinned pages.
• Cold-threshold eviction creates decay tombstones via wiki.evict_page_if_latest, preserving history while removing source files from episodic, unpinned pages with retention scores below cold_threshold.
• The sweep evaluates TTL first (lines 61–73), then applies decayability filters before retention scoring (lines 30–43, 89–99).
• Hard-delete cleanup runs after primary eviction to remove stale tombstones based on hard_delete_after_days.

Frequently Asked Questions

Does pinning a page prevent TTL expiration?

No. According to the source code in crates/ai-memory-consolidate/src/sweep.rs, an explicit expires_at timestamp outranks pin status. TTL checks occur before decayability filters, meaning even pinned pages are hard-deleted once their expiration timestamp passes.

What happens to the history of a cold-evicted page?

Cold-threshold eviction preserves history through a decay tombstone written by wiki.evict_page_if_latest. This tombstone remains in the SQLite store until the hard-delete pass removes it after hard_delete_after_days, allowing optional recovery or auditing of evicted content.

Can semantic pages be evicted via the cold threshold?

No. The is_decayable function (lines 30–43) explicitly filters out semantic pages, allowing only episodic pages to proceed to retention scoring. Semantic content must be deleted manually or via TTL expiration; it is never subject to automated cold eviction.

How often does the forget sweep run?

The sweep runs daily by default or on-demand via the run_sweep function. Each execution processes all latest pages (is_latest = 1) in a project, applying TTL and cold-threshold logic idempotently with non-fatal error handling for retry safety.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}