# Data Flow for Lifecycle Hooks in ai-memory: HTTP Ingestion to Persistence Pipeline

> Explore the ai-memory data flow pipeline for lifecycle hooks Understand HTTP ingestion Axum routing policy filters rate limiting async persistence and session handoff

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: architecture
- Published: 2026-08-28

---

**Lifecycle hooks in ai-memory flow through a four-stage pipeline: HTTP entry via Axum routers, pre-processing with policy filters and rate limiting, asynchronous persistence via a writer actor, and optional handoff retrieval for new sessions.**

The **ai-memory lifecycle hook subsystem** ingests observation events from client agents, enforces capture policies and rate limits, persists data to storage, and supports session handoffs for state continuity. This article walks through the complete data flow using the actual Rust implementation in the `akitaonrails/ai-memory` repository.

## HTTP Entry Points: Router and Handlers

All hook traffic enters through `hook_router()` in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs)【L76-L84】. The router exposes three primary endpoints:

| Endpoint | Method | Handler |
|----------|--------|---------|
| `/hook` | POST | `handle_hook()` – single event ingestion |
| `/hook/batch` | POST | `handle_hook_batch()` – batched event ingestion |
| `/handoff` | GET | `handle_handoff()` – retrieve pending session handoff |

The Axum router deserializes query parameters into `HookQuery` and JSON bodies into structured payloads defined in [`crates/ai-memory-hooks/src/payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/payload.rs). Every request carries metadata identifying the agent, event type, session, and workspace context.

## Pre-Processing Pipeline: Filters, Policies, and Rate Limits

Before any event reaches storage, `handle_hook()` and `handle_hook_batch()` execute a strict pre-processing sequence. These steps protect downstream systems from malformed data, policy violations, and abuse.

### Assistant Message Stripping

Raw assistant messages are stripped to prevent circular ingestion. The code explicitly removes any `_ai_memory_assistant` field before deserialization【L102-L106】.

### Capture Policy Inspection

`inspect_capture_envelope()` in [`crates/ai-memory-hooks/src/capture_policy.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/capture_policy.rs) evaluates the `_ai_memory_capture` marker to determine disposition:

- **Keep** – proceed with full ingestion
- **MetadataOnly** – strip content, persist headers only
- **Drop** – silently discard the event

A `None` result from `inspect_capture_envelope()` triggers an immediate **202 Accepted** response with "capture policy dropped"【L81-L89】.

### Sub-Agent Drop Filter

Projects opting into `drop_subagent` behavior invoke `should_drop_subagent()`【L22-L30】. This function checks against `SubagentSessionSet` and silently drops events from sub-agent contexts to prevent noise amplification.

### Global Concurrent Limit

The `ingest_semaphore` enforces `DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT` concurrent operations via `try_acquire_owned()`【L33-L36】. Excess requests receive **503 Service Unavailable** without entering the processing queue.

### Per-Source Rate Limiting

`ingest_rate_key()` generates a deterministic key from user identity and session ID (or a hashed fallback for anonymous sources). `IngestRateLimiter::try_take()` applies token-bucket limiting per source【L38-L44】. Rate-limited events are skipped and counted toward metrics without failing the request.

## Persistence and Asynchronous Processing

Once pre-processing completes, the event enters the async ingestion path:

```rust
// Metrics recording
state.ingest_metrics.record_accepted();

// Background task spawned for actual write
tokio::spawn(async move {
    match process_envelope(state, envelope).await {
        Ok(_) => {
            record_persisted(&state, &envelope);
        }
        Err(e) => {
            // Error logged, metrics updated
        }
    }
});

```

From `handle_hook()`【L49-L70】:

1. **`process_envelope()`** hands the `HookEnvelope` to the writer actor in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs)
2. On success, **`record_persisted()`** captures the timestamp for latency metrics
3. The HTTP response (**202 Accepted**) is already dispatched—persistence happens asynchronously

For `SessionEnd` events, `synthesize_session_page()` in [`crates/ai-memory-hooks/src/workstream.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/workstream.rs) optionally generates a summary page via LLM or rule-based synthesis.

## Batch Processing Semantics

`POST /hook/batch` applies identical pre-processing per item with three key differences【L71-L78】【L84-L90】:

- **Size bound**: `MAX_HOOK_BATCH_ITEMS` limits batches to 256 events
- **Inline execution**: No background spawning—processing occurs synchronously
- **Fail-fast behavior**: First error aborts the entire batch; rate-limited or policy-dropped items are skipped and counted as committed【L98-L107】

The response returns `accepted` (total count) or `accepted_indices` (specific positions) depending on partial success modes.

## Handoff Retrieval Flow

New sessions query past state through the handoff mechanism:

```bash
curl "http://127.0.0.1:49374/handoff?agent=git&session_id=abcd1234"

```

`handle_handoff()`【L12-L22】 in [`router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/router.rs):

1. Parses `HandoffQuery` from URL parameters
2. Checks identity and admission skip headers (mirroring hook ingress)
3. Calls `fetch_and_accept_handoff()` to atomically retrieve and mark accepted any pending markdown for the `(workspace, project)` tuple
4. Returns the markdown body or empty string with **200 OK**

This enables warm-start scenarios where a new agent instance receives summarized context from a previous session.

## Practical Code Examples

### Single Event Ingestion

```bash
curl -X POST "http://127.0.0.1:49374/hook?event=ToolUse&agent=git" \
     -H "Content-Type: application/json" \
     -d '{
           "session_id":"abcd1234",
           "cwd":"/home/user/project",
           "tool_family":"file",
           "tool_name":"file",
           "_ai_memory_capture":"{\"disposition\":\"Keep\",\"tool_family\":\"File\"}"
         }'

# Response: 202 Accepted ("queued")

```

### Batched Event Ingestion

```bash
curl -X POST "http://127.0.0.1:49374/hook/batch" \
     -H "Content-Type: application/json" \
     -d '[{
           "url":"http://127.0.0.1:49374/hook?event=ToolUse&agent=git",
           "body":{"session_id":"s1","cwd":"/repo","tool_family":"file"}
         },
         {
           "url":"http://127.0.0.1:49374/hook?event=ToolUse&agent=git",
           "body":{"session_id":"s2","cwd":"/repo","tool_family":"file"}
         }]'

# Response: JSON with accepted count or indices

```

### Handoff Retrieval

```bash
curl "http://127.0.0.1:49374/handoff?agent=git&session_id=abcd1234"

# Response: markdown handoff document (or empty)

```

## Key Implementation Files

| File | Responsibility |
|------|--------------|
| [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) | Axum routes, HTTP handlers, request dispatch |
| [`crates/ai-memory-hooks/src/payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/payload.rs) | `HookEnvelope`, `HookQuery`, request parsing |
| [`crates/ai-memory-hooks/src/capture_policy.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/capture_policy.rs) | Policy inspection and enforcement logic |
| [`crates/ai-memory-hooks/src/workstream.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/workstream.rs) | Session summary synthesis |
| [`crates/ai-memory-hooks/src/log.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/log.rs) | Structured logging and metrics |
| [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) | Persistent storage actor (invoked indirectly) |

## Summary

- **Lifecycle hook data flow** in ai-memory spans four stages: HTTP ingress, pre-processing with policy/rate filters, async persistence, and handoff retrieval
- **Pre-processing** occurs synchronously—assistant stripping, capture policy inspection, sub-agent filtering, global semaphore acquisition, and per-source rate limiting
- **Persistence** is asynchronous after returning **202 Accepted**—the writer actor handles actual storage with metrics tracking
- **Batch mode** uses inline fail-fast processing with a 256-item limit
- **Handoffs** provide session continuity through atomic fetch-and-accept operations on pending markdown documents

## Frequently Asked Questions

### What happens when a lifecycle hook event violates a capture policy?

The `inspect_capture_envelope()` function returns `None` for drop dispositions, causing `handle_hook()` to immediately respond with **202 Accepted** and "capture policy dropped" without entering the persistence pipeline. Metadata-only policies strip content before storage. Both outcomes are logged and counted in ingestion metrics.

### How does ai-memory prevent overload from high-volume hook traffic?

Two mechanisms guard the system: a **global semaphore** (`ingest_semaphore`) caps concurrent in-flight events at `DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT`【L33-L36】, and **per-source rate limiting** via `IngestRateLimiter` enforces token-bucket constraints per user-session combination【L38-L44】. Excess global requests receive **503**; rate-limited sources are silently skipped without failing the request.

### What is the difference between single and batch hook processing?

Single events spawn a **background async task** for persistence after returning 202. Batches process **inline up to 256 items** with fail-fast semantics—any error aborts the entire batch, while policy/rate-limited items are skipped and counted as successful. Batch responses include acceptance metadata; single events do not.

### When should client agents use the handoff endpoint?

Call `/handoff` when starting a **new session that may continue work from a previous context**. The endpoint retrieves and atomically accepts any pending markdown summary generated from a prior `SessionEnd` event. This enables warm-start behavior without requiring clients to persist state across process restarts.