The Lifecycle of an Observation in ai-memory: 8 Stages from Capture to Retrieval

An observation in ai-memory moves through eight distinct stages—from hook emission and HTTP capture, through SQLite insertion and optional ingestion, to consolidation, pruning, and retrieval—enforced by privacy sanitization and single-writer atomicity.

The ai-memory system captures discrete events from AI agent lifecycles and persists them as structured observations. Understanding the lifecycle of an observation in ai-memory is essential for developers optimizing storage, implementing custom hooks, or debugging session persistence. This guide traces the complete journey using actual source paths from the akitaonrails/ai-memory repository.

Stage 1: Hook Emission and HTTP Capture

The lifecycle begins when an agent CLI emits a lifecycle-hook event. These events include SessionStart, UserPromptSubmit, PostToolUse, or Stop hooks.

Lifecycle Hook Events

The hook scripts automatically post a JSON envelope to the server's /hook endpoint. This payload contains the raw observation data including kind, title, body, and session metadata.

Privacy Sanitization at the Boundary

Upon receipt, the server wraps the payload in a Sanitized<NewObservation> type. According to the ai-memory source code, this enforces privacy stripping and size constraints—16 KiB for body content and 2 KiB for notifications—before any disk write occurs. This guarantees that sensitive data never reaches the storage layer.

Stage 2: Atomic Insertion via the Writer Actor

All writes funnel through a single writer actor to prevent race conditions. In crates/ai-memory-store/src/writer.rs at lines 124-130, the InsertObservation { obs, reply } command persists the raw observation to the SQLite observations table.

The InsertObservation Command

The following Rust code demonstrates the sanitized capture and insertion:

// Capture a user-prompt observation from a hook
let payload = NewObservation {
    kind: ObservationKind::UserPrompt,
    title: "Ask about observation lifecycle".into(),
    body: "What is the lifecycle of an observation in ai-memory?".into(),
    // additional fields omitted
};

let sanitized = Sanitizer::sanitize(payload);          // privacy strip
store.insert_observation(sanitized).await?;            // InsertObservation

This single-writer guarantee ensures atomicity. The obs field contains the bounded observation data, while the reply channel returns the operation result to the caller.

Optional Ingestion Processing

Some observations require additional processing. The writer issues an InsertObservationIngest command (defined around lines 127-134 in writer.rs) to trigger background jobs. These asynchronous tasks compute vector embeddings or stage auto-improvement runs without blocking the main insertion path.

Stage 3: Session Consolidation

When a session ends via a SessionEnd hook, the system runs consolidation logic defined in crates/ai-memory-store/src/session_consolidation.rs. This process creates a deterministic page summarizing the session and may generate a handoff document.

During consolidation, the observation is marked as consolidated, establishing it as immutable historical data eligible for later pruning while preserving the complete audit trail of raw session events.

Stage 4: Lifecycle-Aware Pruning

Consolidated observations undergo pruning to maintain store compactness. The PruneConsolidatedObservations command (found around line 249 in crates/ai-memory-store/src/writer.rs) removes obsolete rows while retaining the latest version of each page.

Only observations that have been fully consolidated are eligible for deletion. This preserves raw session events until they have been summarized. The operation is typically invoked by a cron job:

// Prune consolidated observations
store.prune_consolidated_observations().await?;

Stage 5: Retrieval via FTS5 and Vector Search

Observations become searchable through two mechanisms. The ObservationPage API in crates/ai-memory-store/src/reader.rs (utilizing ObservationRecord structures) queries the FTS5 index for text matches. When embeddings are enabled, vector similarity search retrieves semantically related observations.

// Retrieve all user-prompt observations for a session
let page = ObservationPage {
    session_id: Some(session_id),
    kinds: Some(vec![ObservationKind::UserPrompt]),
    order: ObservationOrder::Asc,
    limit: 100,
    ..Default::default()
};

let records = store.read_observations(page).await?;
for obs in records.observations {
    println!("{} – {}", obs.title, obs.body);
}

Stage 6: Lifecycle Operations Safety Guards

Destructive lifecycle commands such as purge-project or move-session respect observation immutability. As documented in docs/lifecycle-ops.md (lines 331-336), the server refuses destructive operations while a live run holds a lease, preventing orphaned or lost observations.

This guarding mechanism ensures that no observation can be deleted or moved while actively participating in a running session.

Core Guarantees and Constraints

Sanitization and Bounded Size: Every observation undergoes privacy stripping and size limitation at the HTTP boundary, ensuring reproducibility and privacy compliance.

Immutable Core Fields: Fields such as kind, title, and timestamp are stored on the lifecycle row and never mutate. Later stages only append derived data like embeddings or consolidation flags.

Single-Writer Architecture: The writer actor serializes all mutations, eliminating race conditions and ensuring that stages proceed in strict order.

Summary

• Observations enter the system via HTTP POST to /hook with automatic Sanitized<NewObservation> wrapping
• The writer actor in writer.rs handles atomic insertion via InsertObservation at lines 124-130 and optional InsertObservationIngest at lines 127-134
• Session consolidation in session_consolidation.rs marks observations as consolidated after a SessionEnd hook
• PruneConsolidatedObservations (line 249) removes only consolidated data to preserve audit trails
• Retrieval occurs through reader.rs using FTS5 text search or vector similarity via ObservationPage
• Lifecycle guards documented in docs/lifecycle-ops.md prevent destructive operations on active observations

Frequently Asked Questions

What is the difference between InsertObservation and InsertObservationIngest?

InsertObservation performs the immediate atomic write to the SQLite observations table, returning a confirmation to the caller. InsertObservationIngest is an optional follow-up command that triggers asynchronous background processing, such as computing embeddings or staging auto-improvement tasks, without blocking the main execution flow.

How does ai-memory protect sensitive data during the observation lifecycle?

Privacy enforcement occurs at the HTTP capture boundary through the Sanitized<NewObservation> type. The sanitizer strips private fields and enforces size limits (16 KiB for bodies, 2 KiB for notifications) before any disk write occurs, ensuring sensitive data never reaches persistent storage.

Can observations be modified after they are inserted?

No. Core fields including kind, title, and timestamps are immutable once written. The system only appends derived metadata such as consolidation flags or embeddings. This append-only model maintains a strict audit trail of agent activities according to the repository's architecture.

When are observations eligible for pruning?

Only observations marked as consolidated—those processed by the session consolidation logic after a SessionEnd hook—are eligible for pruning. The PruneConsolidatedObservations command specifically targets these records to prevent data loss of active session data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}