# The Lifecycle of an Observation in ai-memory: 8 Stages from Capture to Retrieval

> Explore the 8 stages of an observation lifecycle in ai-memory. Understand capture, ingestion, consolidation, pruning, and retrieval for effective AI data management.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: internals
- Published: 2026-08-31

---

**An observation in ai-memory moves through eight distinct stages—from hook emission and HTTP capture, through SQLite insertion and optional ingestion, to consolidation, pruning, and retrieval—enforced by privacy sanitization and single-writer atomicity.**

The ai-memory system captures discrete events from AI agent lifecycles and persists them as structured observations. Understanding the lifecycle of an observation in ai-memory is essential for developers optimizing storage, implementing custom hooks, or debugging session persistence. This guide traces the complete journey using actual source paths from the akitaonrails/ai-memory repository.

## Stage 1: Hook Emission and HTTP Capture

The lifecycle begins when an agent CLI emits a lifecycle-hook event. These events include `SessionStart`, `UserPromptSubmit`, `PostToolUse`, or `Stop` hooks.

### Lifecycle Hook Events

The hook scripts automatically post a JSON envelope to the server's `/hook` endpoint. This payload contains the raw observation data including kind, title, body, and session metadata.

### Privacy Sanitization at the Boundary

Upon receipt, the server wraps the payload in a `Sanitized<NewObservation>` type. According to the ai-memory source code, this enforces privacy stripping and size constraints—16 KiB for body content and 2 KiB for notifications—before any disk write occurs. This guarantees that sensitive data never reaches the storage layer.

## Stage 2: Atomic Insertion via the Writer Actor

All writes funnel through a single writer actor to prevent race conditions. In [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) at lines 124-130, the `InsertObservation { obs, reply }` command persists the raw observation to the SQLite `observations` table.

### The InsertObservation Command

The following Rust code demonstrates the sanitized capture and insertion:

```rust
// Capture a user-prompt observation from a hook
let payload = NewObservation {
    kind: ObservationKind::UserPrompt,
    title: "Ask about observation lifecycle".into(),
    body: "What is the lifecycle of an observation in ai-memory?".into(),
    // additional fields omitted
};

let sanitized = Sanitizer::sanitize(payload);          // privacy strip
store.insert_observation(sanitized).await?;            // InsertObservation

```

This single-writer guarantee ensures atomicity. The `obs` field contains the bounded observation data, while the `reply` channel returns the operation result to the caller.

### Optional Ingestion Processing

Some observations require additional processing. The writer issues an `InsertObservationIngest` command (defined around lines 127-134 in [`writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/writer.rs)) to trigger background jobs. These asynchronous tasks compute vector embeddings or stage auto-improvement runs without blocking the main insertion path.

## Stage 3: Session Consolidation

When a session ends via a `SessionEnd` hook, the system runs consolidation logic defined in [`crates/ai-memory-store/src/session_consolidation.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/session_consolidation.rs). This process creates a deterministic page summarizing the session and may generate a handoff document.

During consolidation, the observation is marked as *consolidated*, establishing it as immutable historical data eligible for later pruning while preserving the complete audit trail of raw session events.

## Stage 4: Lifecycle-Aware Pruning

Consolidated observations undergo pruning to maintain store compactness. The `PruneConsolidatedObservations` command (found around line 249 in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs)) removes obsolete rows while retaining the latest version of each page.

Only observations that have been fully consolidated are eligible for deletion. This preserves raw session events until they have been summarized. The operation is typically invoked by a cron job:

```rust
// Prune consolidated observations
store.prune_consolidated_observations().await?;

```

## Stage 5: Retrieval via FTS5 and Vector Search

Observations become searchable through two mechanisms. The `ObservationPage` API in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (utilizing `ObservationRecord` structures) queries the **FTS5** index for text matches. When embeddings are enabled, vector similarity search retrieves semantically related observations.

```rust
// Retrieve all user-prompt observations for a session
let page = ObservationPage {
    session_id: Some(session_id),
    kinds: Some(vec![ObservationKind::UserPrompt]),
    order: ObservationOrder::Asc,
    limit: 100,
    ..Default::default()
};

let records = store.read_observations(page).await?;
for obs in records.observations {
    println!("{} – {}", obs.title, obs.body);
}

```

## Stage 6: Lifecycle Operations Safety Guards

Destructive lifecycle commands such as `purge-project` or `move-session` respect observation immutability. As documented in [`docs/lifecycle-ops.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/lifecycle-ops.md) (lines 331-336), the server refuses destructive operations while a live run holds a lease, preventing orphaned or lost observations.

This guarding mechanism ensures that no observation can be deleted or moved while actively participating in a running session.

## Core Guarantees and Constraints

**Sanitization and Bounded Size:** Every observation undergoes privacy stripping and size limitation at the HTTP boundary, ensuring reproducibility and privacy compliance.

**Immutable Core Fields:** Fields such as `kind`, `title`, and timestamp are stored on the lifecycle row and never mutate. Later stages only append derived data like embeddings or consolidation flags.

**Single-Writer Architecture:** The writer actor serializes all mutations, eliminating race conditions and ensuring that stages proceed in strict order.

## Summary

- Observations enter the system via HTTP POST to `/hook` with automatic `Sanitized<NewObservation>` wrapping
- The writer actor in [`writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/writer.rs) handles atomic insertion via `InsertObservation` at lines 124-130 and optional `InsertObservationIngest` at lines 127-134
- Session consolidation in [`session_consolidation.rs`](https://github.com/akitaonrails/ai-memory/blob/main/session_consolidation.rs) marks observations as consolidated after a `SessionEnd` hook
- `PruneConsolidatedObservations` (line 249) removes only consolidated data to preserve audit trails
- Retrieval occurs through [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) using FTS5 text search or vector similarity via `ObservationPage`
- Lifecycle guards documented in [`docs/lifecycle-ops.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/lifecycle-ops.md) prevent destructive operations on active observations

## Frequently Asked Questions

### What is the difference between InsertObservation and InsertObservationIngest?

**`InsertObservation`** performs the immediate atomic write to the SQLite `observations` table, returning a confirmation to the caller. **`InsertObservationIngest`** is an optional follow-up command that triggers asynchronous background processing, such as computing embeddings or staging auto-improvement tasks, without blocking the main execution flow.

### How does ai-memory protect sensitive data during the observation lifecycle?

Privacy enforcement occurs at the HTTP capture boundary through the `Sanitized<NewObservation>` type. The sanitizer strips private fields and enforces size limits (16 KiB for bodies, 2 KiB for notifications) before any disk write occurs, ensuring sensitive data never reaches persistent storage.

### Can observations be modified after they are inserted?

No. Core fields including `kind`, `title`, and timestamps are immutable once written. The system only appends derived metadata such as consolidation flags or embeddings. This append-only model maintains a strict audit trail of agent activities according to the repository's architecture.

### When are observations eligible for pruning?

Only observations marked as **consolidated**—those processed by the session consolidation logic after a `SessionEnd` hook—are eligible for pruning. The `PruneConsolidatedObservations` command specifically targets these records to prevent data loss of active session data.