Understanding the 4 Retention Tiers for Data in ai-memory

ai-memory organizes information into four distinct retention tiers—Working, Episodic, Semantic, and Procedural—that govern how long pages persist and how their relevance decays over time.

The ai-memory project implements a biologically inspired memory architecture for AI systems, using specific retention tiers for data to optimize between volatile session context and permanent knowledge storage. These tiers are formally defined in docs/ARCHITECTURE.md and control everything from immediate working memory to indefinitely stored procedural patterns.

The Four Retention Tiers Explained

Working Tier: Session-Only Storage

The Working tier acts as short-term memory for the current session only. When the session terminates, pages in this tier undergo a hard-drop and are no longer accessible through standard queries. According to the architecture documentation, raw observations are preserved in the observations table for forensic purposes, but the structured page data disappears. This tier is ideal for transient contexts, temporary calculations, and immediate conversational state that should not persist beyond the current interaction.

Episodic Tier: Time-Bounded Decay

The Episodic tier implements a graduated retention policy: pages remain "hot" for 30 days, transition to "cold" status for 180 days, and face eviction thereafter. This tier uses the most complex decay mechanics in the system, scoring relevance through a multi-factor formula implemented in crates/ai-memory-store/src/decay.rs:

salience·exp(−λΔt) + σ·log(1+access_count)·exp(−μ·days_since_access)·(1 + breadth_weight·ln(1+max(distinct_actors−1,0)))

The formula blends salience (base importance), exponential time decay (controlled by λ), logarithmic access counting with temporal decay (μ), and a breadth term that rewards pages accessed by multiple distinct actors. This tier suits time-relevant events, recent conversations, and context that should fade naturally unless frequently reinforced.

Semantic Tier: Permanent Knowledge Storage

The Semantic tier provides indefinite persistence with no automatic decay. Pages survive forever unless explicitly superseded by later LLM-driven rewrites (specifically the M7 rewrite step mentioned in the architecture). This tier stores facts, concepts, and refined knowledge that should remain accessible regardless of temporal distance. Unlike Episodic data, Semantic pages do not lose relevance due to age or disuse.

Procedural Tier: Pattern-Based Retention

The Procedural tier also persists indefinitely but implements frequency-based decay rather than time-based decay. If a procedural pattern (such as a workflow or skill) is not re-observed or reinforced through repeated use, its relevance slowly diminishes. This mechanic ensures that outdated procedures naturally sink below newer, more active patterns while remaining recoverable if the behavior resumes.

Creating and Managing Pages by Tier

Assigning Tiers in Rust

The Tier enum defined in crates/ai-memory-core/src/page.rs allows explicit tier assignment during page creation:

use ai_memory_core::{Page, Tier};

let mut page = Page::new("example.md", "example body")
    .with_kind("note")
    .with_tier(Tier::Episodic);   // Options: Working, Episodic, Semantic, or Procedural
store.write_page(page).await?;

Running Retention Sweeps

The system provides CLI commands to manually trigger retention evaluation. The implementation resides in crates/ai-memory-mcp/src/admin.rs (lines 91-98) and crates/ai-memory-cli/src/commands/forget_sweep.rs:

# Preview which pages would be evicted without deleting them

ai-memory forget-sweep --dry-run

# Execute the actual retention pass

ai-memory forget-sweep

Filtering Expired Data

When querying the store, you can exclude expired Episodic pages using the --include-expired flag:

ai-memory query "project:myproj query:search term" --include-expired=false

Exemptions from Decay

Certain pages bypass all decay mechanisms regardless of their assigned tier. Pages marked with pinned: true in their front-matter are exempt from all retention sweeps. Additionally, any page stored under the _slots/ namespace receives automatic pinning, ensuring critical structural data persists indefinitely without manual intervention (as documented in docs/ARCHITECTURE.md lines 102-107).

Summary

• Four distinct tiers govern data lifecycle: Working (session-only), Episodic (time-decayed), Semantic (permanent), and Procedural (frequency-decayed).
• Episodic tier uses the most sophisticated scoring, combining salience, exponential time decay, access logarithms, and actor breadth in crates/ai-memory-store/src/decay.rs.
• Semantic and Procedural tiers persist indefinitely, differing in that Semantic pages never decay while Procedural pages fade through non-use.
• Pinned pages and the _slots/ namespace are immune to all retention policies.
• CLI tools like forget-sweep allow manual triggering of retention passes with dry-run support.

Frequently Asked Questions

How do I create a page in a specific retention tier using the Rust API?

Use the Tier enum from ai-memory-core and chain the .with_tier() method when constructing a Page object. Pass Tier::Working, Tier::Episodic, Tier::Semantic, or Tier::Procedural to place the page in the appropriate retention tier before calling store.write_page().

What is the difference between the Episodic and Semantic retention tiers?

Episodic pages decay over time using the scoring formula in decay.rs, transitioning from hot (30 days) to cold (180 days) before eventual eviction. Semantic pages persist indefinitely with no automatic decay and are only updated through explicit LLM-driven rewrites (the M7 step).

How can I prevent important pages from being deleted by the retention system?

Add pinned: true to the page's front-matter, or store the page under the _slots/ namespace, which automatically applies pinning. Pinned pages are exempt from all decay paths and retention sweeps regardless of their assigned tier.

Where is the decay scoring formula implemented for the Episodic tier?

The mathematical scoring logic resides in crates/ai-memory-store/src/decay.rs, which implements the blended formula combining salience, time-based exponential decay, access-count logarithms, and the breadth term for distinct actors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}