How ai-memory Handles Data Retention and Page Decay: Exponential Decay in a SQLite Wiki

ai-memory implements a tunable exponential decay algorithm that scores pages based on age and access patterns, soft-deletes cold content below a configurable threshold, and permanently purges tombstones after a 180-day retention period.

The ai-memory project provides a SQLite-backed wiki system for AI observations, but without boundaries, knowledge bases grow indefinitely. To solve this, the repository implements a sophisticated data retention and page decay mechanism that automatically evaluates content freshness, removes stale pages, and preserves important information through user feedback. This self-pruning architecture ensures the system remains performant while retaining valuable context.

The Retention Score Algorithm

At the core of data retention and page decay lies the scoring engine in crates/ai-memory-store/src/decay.rs. The retention_score function calculates a floating-point value for each page by combining temporal decay with access frequency signals.

Tunable Decay Parameters

The DecayParams struct defines the coefficients governing decay behavior:

• lambda: Controls exponential decay of page age (default 0.02, yielding a ~35-day half-life)
• sigma: Boost factor for access reinforcement
• mu: Exponential decay applied to days since last access (default 0.04, ~2-day half-life)
• salience_default: Baseline weight for pages without explicit feedback
• cold_threshold: Score boundary below which pages become eviction candidates
• hard_delete_after_days: Tombstone survival period before permanent removal (default 180 days)

Computing the Score

The algorithm combines three mathematical terms in the retention_score function (lines 48-75 in decay.rs):

let time_term = salience * (-params.lambda * age_days).exp();
let access_term = days_since_access.map_or(0.0, |d| {
    params.sigma * (1.0 + access_count as f64).ln() *
    (-params.mu * d).exp() * breadth
});
let score = time_term + access_term;

Age decay reduces scores exponentially through lambda, while recent accesses add a reinforcement boost via sigma that itself decays with mu. The optional breadth parameter (from retention_score_with_breadth) weights distinct actors accessing the content.

Identifying Decay Candidates

The store reader exposes decay_candidates in crates/ai-memory-store/src/reader.rs (line 3193) to surface pages eligible for removal:

pub async fn decay_candidates(
    &self,
    ws: WorkspaceId,
    proj: ProjectId,
) -> Result<Vec<DecayCandidate>, StoreError>

This function filters for pages that are not pinned and whose retention score falls below cold_threshold. It returns metadata including page ID, path, and pinned status to the sweep orchestrator.

The Decay Sweep and Soft Deletion

The periodic decay sweep lives in crates/ai-memory-consolidate/src/sweep.rs. The system fetches candidates via reader.decay_candidates, then invokes soft_delete_for_decay_if_latest in crates/ai-memory-wiki/src/wiki.rs (lines 66-74):

self.writer.soft_delete_for_decay_if_latest(
    workspace_id,
    project_id,
    path.clone(),
    expected_latest_id,
).await?;

Soft deletion sets the superseded_at timestamp on the page, creating a decay tombstone while preserving the original Markdown file on disk. This approach maintains audit trails and allows potential recovery while removing content from active queries.

Hard Deletion of Aged Tombstones

After hard_delete_after_days (default 180 days), the system permanently purges decay tombstones. The hard_delete_decayed_page_chain function in crates/ai-memory-wiki/src/wiki.rs (lines 98-108) executes this operation:

self.writer.hard_delete_decayed_page_chain(
    workspace_id,
    project_id,
    path.clone(),
    tombstone_id,
    current_latest,
    cutoff_us,
).await?;

This irreversible action removes both the SQLite records and the on-disk markdown hierarchy, reclaiming storage completely.

Feedback-Driven Salience Adjustments

User or agent feedback can override decay through explicit salience adjustments. The salience_after_feedback function in decay.rs (lines 32-55) updates page weights based on FeedbackKind enums (e.g., Helpful or NotHelpful).

Salience values are clamped between SALIENCE_MIN (0.25) and SALIENCE_MAX (2.0), allowing reinforced pages to survive significantly longer than default content. This creates a hybrid retention model combining algorithmic decay with explicit human curation.

Configuration and Default Values

All coefficients are configurable through the DecayParams struct, overridable via ai_memory_core::Config. The defaults provide aggressive cleanup of abandoned content:

• Untouched pages: ~35-day half-life (lambda = 0.02)
• Access reinforcement: ~2-day half-life (mu = 0.04)

Pages scoring below cold_threshold enter the decay queue immediately, while tombstones survive 180 days before hard deletion.

Summary

• ai-memory implements data retention and page decay through a tunable exponential scoring algorithm in crates/ai-memory-store/src/decay.rs.
• The retention_score function combines age decay (lambda) with access reinforcement (sigma, mu) to calculate page vitality.
• The decay_candidates function in reader.rs identifies non-pinned pages below the cold_threshold for removal.
• Soft deletion via soft_delete_for_decay_if_latest creates tombstones preserving audit trails, while hard_delete_decayed_page_chain permanently removes content after 180 days.
• Feedback mechanisms allow explicit salience adjustments between 0.25 and 2.0 to protect valuable pages from automatic decay.
• Default parameters enforce a ~35-day half-life for untouched content and ~2-day half-life for access-based reinforcement signals.

Frequently Asked Questions

What happens when a page's retention score drops below the cold_threshold?

When a page's calculated score falls below the configured cold_threshold, it becomes eligible for the decay sweep. The decay_candidates function surfaces these pages to the sweep orchestrator, which invokes soft_delete_for_decay_if_latest to mark them as tombstones. This soft-deleted state removes the content from active queries while preserving the underlying Markdown file for 180 days.

How does ai-memory distinguish between soft deletion and hard deletion?

Soft deletion sets the superseded_at timestamp on a page record, creating a decay tombstone that retains the original file on disk for potential recovery. Hard deletion occurs when a tombstone exceeds hard_delete_after_days (default 180 days), triggering hard_delete_decayed_page_chain to permanently purge both the SQLite records and the on-disk markdown hierarchy.

Can users prevent specific pages from being decayed?

Yes. The decay_candidates function specifically excludes pinned pages from eviction consideration. Additionally, users can provide feedback signals that adjust a page's salience via salience_after_feedback. Since salience multiplies the time decay term, explicitly boosting important content above the cold_threshold prevents automatic removal regardless of age.

What is the default retention period before permanent removal?

By default, decay tombstones are retained for 180 days (hard_delete_after_days = 180) before the system invokes hard_delete_decayed_page_chain for permanent removal. This window provides nearly six months for potential recovery or audit of soft-deleted content before storage reclamation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}