# How Raw Observations are Handled by the Memory Decay System in ai-memory

> Discover how ai-memory's memory decay system handles raw observations. Learn about the pruning process and observation retention for efficient AI memory management.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: internals
- Published: 2026-09-06

---

**Raw observations are never decayed—they are only pruned via an optional, disabled-by-default observation retention pass that runs after page consolidation.**

The ai-memory system treats raw session observations differently from distilled wiki pages. While pages undergo salience-based decay and TTL expiry, raw observations persist as unmodified capture data until explicitly removed through a separate cleanup mechanism. This design preserves original session data as a searchable fallback while allowing operators to optionally reclaim storage space.

---

## Understanding the Three-Stage Sweep Pipeline

The memory decay system runs a coordinated sweep across three distinct cleanup stages. Raw observations are isolated to the final stage, ensuring they survive any page-level eviction.

In [`crates/ai-memory-consolidate/src/sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/sweep.rs), the `run_sweep` function orchestrates these stages:

1. **Page decay** — Episodic pages with salience below `DecayParams::threshold` are removed
2. **TTL expiry** — Pages with expired `expires_at` timestamps are hard-deleted
3. **Observation prune** — Raw observations are processed only here, and only if enabled

```rust
// From crates/ai-memory-consolidate/src/sweep.rs lines 22-30
// The sweep executes stages sequentially; observation pruning runs last

```

Running observation pruning last guarantees that if a session's consolidated page is removed in earlier stages, its raw observations remain intact. This prevents accidental data loss where a session would have neither summary nor original capture.

---

## The ObservationRetention Configuration

Raw observation pruning is controlled by the `ObservationRetention` struct, defined in the sweep module:

```rust
pub struct ObservationRetention {
    /// Age in days past which a consolidated session's observations may be pruned.
    /// `0` disables the pass entirely — the default.
    pub days: i64,
    /// Rows deleted per transaction.
    pub batch: usize,
}

```

**Key behavior:** When `days` is `0` (the default), the entire observation prune pass is skipped. No raw observations are examined or deleted regardless of age.

When configured with `days > 0`, the sweep calculates an `observation_cutoff_us` timestamp and proceeds to identify prunable observations. Only sessions that already have a live distilled page are eligible—ensuring raw data serves as a fallback for its summary page, not as orphaned storage.

From [`crates/ai-memory-consolidate/src/sweep.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/sweep.rs) lines 103-118 and 122-130, the sweep:
- Queries the store for prunable observation count per workspace/project
- Validates against the retention policy
- Invokes the writer's prune operation in batches

---

## The Writer's Prune Implementation

The actual deletion is performed by `prune_consolidated_observations` in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs). This function receives:

- Workspace and project identifiers
- The cutoff timestamp (`observation_cutoff_us`)
- Batch size limit

It executes deletions within a single transaction, returning the count of affected rows. The sweep aggregates these results across all workspace/project pairs, tracking total rows pruned, batch counts, and distinct sessions affected for the final report.

From [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) lines 1365-1380, the writer constructs and executes the deletion query, respecting the configured batch size to limit transaction scope.

---

## Raw Observations as Search Fallback

Raw observations serve a critical secondary purpose: **full-text search fallback**. When compiled wiki pages return no matches, the system queries raw observation content through a separate FTS5 index.

In [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) lines 382-386, API responses include `raw_hits` containing matching raw observations. This fallback operates independently of decay—raw observations remain searchable until the optional prune pass removes them.

---

## Practical Configuration Examples

### Enable 30-day observation retention

```rust
use ai_memory_consolidate::{DecayParams, ObservationRetention, run_sweep};

let decay_params = DecayParams::default();
let observation_retention = ObservationRetention {
    days: 30,      // Prune observations older than 30 days
    batch: 5_000,  // Delete up to 5,000 rows per transaction
};

let report = run_sweep(
    &reader_pool,
    &writer_handle,
    Some(&wiki),
    workspace_id,
    project_id,
    &decay_params,
    false, // dry_run = false for actual deletion
    observation_retention,
).await?;

println!("Pages decayed: {}", report.pages_decayed);
println!("Observations pruned: {}", report.observations_pruned);
println!("Sessions affected: {}", report.sessions_affected);

```

### Query raw observation fallback via API

```bash

# Include raw_hits=true to search unconsolidated session data

curl "http://localhost:8080/api/v1/search?project=myproj&query=unresolved%20bug&raw_hits=true"

```

### Verify current retention settings

```bash

# CLI configuration check

ai-memory config get observation_retention

```

---

## Summary

- **Raw observations are excluded from decay** — The salience-based decay system only processes distilled wiki pages, never original session captures
- **Pruning is opt-in** — `ObservationRetention::days` defaults to `0`, disabling all raw observation cleanup
- **Pruning runs last** — The observation prune pass executes after page decay and TTL expiry, preserving raw data for sessions that lost their consolidated pages
- **Fallback preservation** — Raw observations remain searchable via FTS5 until explicitly pruned, serving as insurance against page loss
- **Batch deletion** — The writer's `prune_consolidated_observations` removes eligible observations in configurable transactions to control database load

---

## Frequently Asked Questions

### What happens to raw observations if I never enable observation retention?

Raw observations persist indefinitely. With `ObservationRetention::days` set to `0` (the default), the prune pass is skipped entirely. Your raw session data will continue to accumulate and remain available for search fallback regardless of how old the sessions become.

### Can raw observations be deleted even if their consolidated page still exists?

No. The prune logic specifically filters for sessions that have a live distilled page. If a page has been removed through decay or TTL expiry, its raw observations are protected from pruning. This ensures you never lose both the summary and the original capture simultaneously.

### How does observation pruning affect search performance?

Pruning reclaimed storage and reduces FTS5 index size, which can improve search performance over time. However, once pruned, raw observations are no longer available as fallbacks—searches will return empty if no compiled pages match and raw hits have been removed.

### What batch size should I use for large workloads?

The default `batch: 5_000` handles most deployments well. For high-volume installations with millions of observations, consider smaller batches (1,000-2,500) to reduce lock contention and transaction duration. Monitor `report.batches_executed` relative to total observations to tune appropriately.