# How ai-memory Handles Data Retention and Page Decay: Exponential Decay in a SQLite Wiki

> Discover how ai-memory handles data retention and page decay using exponential decay in SQLite. Learn about soft-deletion and purging for efficient data management.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-31

---

**ai-memory implements a tunable exponential decay algorithm that scores pages based on age and access patterns, soft-deletes cold content below a configurable threshold, and permanently purges tombstones after a 180-day retention period.**

The ai-memory project provides a SQLite-backed wiki system for AI observations, but without boundaries, knowledge bases grow indefinitely. To solve this, the repository implements a sophisticated **data retention and page decay** mechanism that automatically evaluates content freshness, removes stale pages, and preserves important information through user feedback. This self-pruning architecture ensures the system remains performant while retaining valuable context.

## The Retention Score Algorithm

At the core of **data retention and page decay** lies the scoring engine in [`crates/ai-memory-store/src/decay.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/decay.rs). The `retention_score` function calculates a floating-point value for each page by combining temporal decay with access frequency signals.

### Tunable Decay Parameters

The `DecayParams` struct defines the coefficients governing decay behavior:

- **`lambda`**: Controls exponential decay of page age (default 0.02, yielding a ~35-day half-life)
- **`sigma`**: Boost factor for access reinforcement
- **`mu`**: Exponential decay applied to days since last access (default 0.04, ~2-day half-life)
- **`salience_default`**: Baseline weight for pages without explicit feedback
- **`cold_threshold`**: Score boundary below which pages become eviction candidates
- **`hard_delete_after_days`**: Tombstone survival period before permanent removal (default 180 days)

### Computing the Score

The algorithm combines three mathematical terms in the `retention_score` function (lines 48-75 in [`decay.rs`](https://github.com/akitaonrails/ai-memory/blob/main/decay.rs)):

```rust
let time_term = salience * (-params.lambda * age_days).exp();
let access_term = days_since_access.map_or(0.0, |d| {
    params.sigma * (1.0 + access_count as f64).ln() *
    (-params.mu * d).exp() * breadth
});
let score = time_term + access_term;

```

**Age decay** reduces scores exponentially through `lambda`, while **recent accesses** add a reinforcement boost via `sigma` that itself decays with `mu`. The optional **breadth** parameter (from `retention_score_with_breadth`) weights distinct actors accessing the content.

## Identifying Decay Candidates

The store reader exposes `decay_candidates` in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (line 3193) to surface pages eligible for removal:

```rust
pub async fn decay_candidates(
    &self,
    ws: WorkspaceId,
    proj: ProjectId,
) -> Result<Vec<DecayCandidate>, StoreError>

```

This function filters for pages that are **not pinned** and whose retention score falls below `cold_threshold`. It returns metadata including page ID, path, and pinned status to the sweep orchestrator.

## The Decay Sweep and Soft Deletion

The periodic decay sweep lives in [`crates/ai-memory-consolidate/src/sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/sweep.rs). The system fetches candidates via `reader.decay_candidates`, then invokes `soft_delete_for_decay_if_latest` in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) (lines 66-74):

```rust
self.writer.soft_delete_for_decay_if_latest(
    workspace_id,
    project_id,
    path.clone(),
    expected_latest_id,
).await?;

```

**Soft deletion** sets the `superseded_at` timestamp on the page, creating a *decay tombstone* while preserving the original Markdown file on disk. This approach maintains audit trails and allows potential recovery while removing content from active queries.

## Hard Deletion of Aged Tombstones

After `hard_delete_after_days` (default 180 days), the system permanently purges decay tombstones. The `hard_delete_decayed_page_chain` function in [`crates/ai-memory-wiki/src/wiki.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) (lines 98-108) executes this operation:

```rust
self.writer.hard_delete_decayed_page_chain(
    workspace_id,
    project_id,
    path.clone(),
    tombstone_id,
    current_latest,
    cutoff_us,
).await?;

```

This irreversible action removes both the SQLite records and the on-disk markdown hierarchy, reclaiming storage completely.

## Feedback-Driven Salience Adjustments

User or agent feedback can override decay through explicit **salience** adjustments. The `salience_after_feedback` function in [`decay.rs`](https://github.com/akitaonrails/ai-memory/blob/main/decay.rs) (lines 32-55) updates page weights based on `FeedbackKind` enums (e.g., `Helpful` or `NotHelpful`).

Salience values are clamped between `SALIENCE_MIN` (0.25) and `SALIENCE_MAX` (2.0), allowing reinforced pages to survive significantly longer than default content. This creates a hybrid retention model combining algorithmic decay with explicit human curation.

## Configuration and Default Values

All coefficients are configurable through the `DecayParams` struct, overridable via `ai_memory_core::Config`. The defaults provide aggressive cleanup of abandoned content:

- **Untouched pages**: ~35-day half-life (`lambda = 0.02`)
- **Access reinforcement**: ~2-day half-life (`mu = 0.04`)

Pages scoring below `cold_threshold` enter the decay queue immediately, while tombstones survive 180 days before hard deletion.

## Summary

- **ai-memory** implements **data retention and page decay** through a tunable exponential scoring algorithm in [`crates/ai-memory-store/src/decay.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/decay.rs).
- The `retention_score` function combines age decay (`lambda`) with access reinforcement (`sigma`, `mu`) to calculate page vitality.
- The `decay_candidates` function in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) identifies non-pinned pages below the `cold_threshold` for removal.
- Soft deletion via `soft_delete_for_decay_if_latest` creates tombstones preserving audit trails, while `hard_delete_decayed_page_chain` permanently removes content after 180 days.
- Feedback mechanisms allow explicit salience adjustments between 0.25 and 2.0 to protect valuable pages from automatic decay.
- Default parameters enforce a ~35-day half-life for untouched content and ~2-day half-life for access-based reinforcement signals.

## Frequently Asked Questions

### What happens when a page's retention score drops below the cold_threshold?

When a page's calculated score falls below the configured `cold_threshold`, it becomes eligible for the decay sweep. The `decay_candidates` function surfaces these pages to the sweep orchestrator, which invokes `soft_delete_for_decay_if_latest` to mark them as tombstones. This soft-deleted state removes the content from active queries while preserving the underlying Markdown file for 180 days.

### How does ai-memory distinguish between soft deletion and hard deletion?

Soft deletion sets the `superseded_at` timestamp on a page record, creating a decay tombstone that retains the original file on disk for potential recovery. Hard deletion occurs when a tombstone exceeds `hard_delete_after_days` (default 180 days), triggering `hard_delete_decayed_page_chain` to permanently purge both the SQLite records and the on-disk markdown hierarchy.

### Can users prevent specific pages from being decayed?

Yes. The `decay_candidates` function specifically excludes **pinned** pages from eviction consideration. Additionally, users can provide feedback signals that adjust a page's **salience** via `salience_after_feedback`. Since salience multiplies the time decay term, explicitly boosting important content above the `cold_threshold` prevents automatic removal regardless of age.

### What is the default retention period before permanent removal?

By default, decay tombstones are retained for **180 days** (`hard_delete_after_days = 180`) before the system invokes `hard_delete_decayed_page_chain` for permanent removal. This window provides nearly six months for potential recovery or audit of soft-deleted content before storage reclamation.