Data Flow for Lifecycle Hooks in ai-memory: HTTP Ingestion to Persistence Pipeline

Lifecycle hooks in ai-memory flow through a four-stage pipeline: HTTP entry via Axum routers, pre-processing with policy filters and rate limiting, asynchronous persistence via a writer actor, and optional handoff retrieval for new sessions.

The ai-memory lifecycle hook subsystem ingests observation events from client agents, enforces capture policies and rate limits, persists data to storage, and supports session handoffs for state continuity. This article walks through the complete data flow using the actual Rust implementation in the akitaonrails/ai-memory repository.

HTTP Entry Points: Router and Handlers

All hook traffic enters through hook_router() in crates/ai-memory-hooks/src/router.rs【L76-L84】. The router exposes three primary endpoints:

Endpoint	Method	Handler
/hook	POST	handle_hook() – single event ingestion
/hook/batch	POST	handle_hook_batch() – batched event ingestion
/handoff	GET	handle_handoff() – retrieve pending session handoff

The Axum router deserializes query parameters into HookQuery and JSON bodies into structured payloads defined in crates/ai-memory-hooks/src/payload.rs. Every request carries metadata identifying the agent, event type, session, and workspace context.

Pre-Processing Pipeline: Filters, Policies, and Rate Limits

Before any event reaches storage, handle_hook() and handle_hook_batch() execute a strict pre-processing sequence. These steps protect downstream systems from malformed data, policy violations, and abuse.

Assistant Message Stripping

Raw assistant messages are stripped to prevent circular ingestion. The code explicitly removes any _ai_memory_assistant field before deserialization【L102-L106】.

Capture Policy Inspection

inspect_capture_envelope() in crates/ai-memory-hooks/src/capture_policy.rs evaluates the _ai_memory_capture marker to determine disposition:

• Keep – proceed with full ingestion
• MetadataOnly – strip content, persist headers only
• Drop – silently discard the event

A None result from inspect_capture_envelope() triggers an immediate 202 Accepted response with "capture policy dropped"【L81-L89】.

Sub-Agent Drop Filter

Projects opting into drop_subagent behavior invoke should_drop_subagent()【L22-L30】. This function checks against SubagentSessionSet and silently drops events from sub-agent contexts to prevent noise amplification.

Global Concurrent Limit

The ingest_semaphore enforces DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT concurrent operations via try_acquire_owned()【L33-L36】. Excess requests receive 503 Service Unavailable without entering the processing queue.

Per-Source Rate Limiting

ingest_rate_key() generates a deterministic key from user identity and session ID (or a hashed fallback for anonymous sources). IngestRateLimiter::try_take() applies token-bucket limiting per source【L38-L44】. Rate-limited events are skipped and counted toward metrics without failing the request.

Persistence and Asynchronous Processing

Once pre-processing completes, the event enters the async ingestion path:

// Metrics recording
state.ingest_metrics.record_accepted();

// Background task spawned for actual write
tokio::spawn(async move {
    match process_envelope(state, envelope).await {
        Ok(_) => {
            record_persisted(&state, &envelope);
        }
        Err(e) => {
            // Error logged, metrics updated
        }
    }
});

From handle_hook()【L49-L70】:

1. process_envelope() hands the HookEnvelope to the writer actor in crates/ai-memory-store/src/writer.rs
2. On success, record_persisted() captures the timestamp for latency metrics
3. The HTTP response (202 Accepted) is already dispatched—persistence happens asynchronously

For SessionEnd events, synthesize_session_page() in crates/ai-memory-hooks/src/workstream.rs optionally generates a summary page via LLM or rule-based synthesis.

Batch Processing Semantics

POST /hook/batch applies identical pre-processing per item with three key differences【L71-L78】【L84-L90】:

• Size bound: MAX_HOOK_BATCH_ITEMS limits batches to 256 events
• Inline execution: No background spawning—processing occurs synchronously
• Fail-fast behavior: First error aborts the entire batch; rate-limited or policy-dropped items are skipped and counted as committed【L98-L107】

The response returns accepted (total count) or accepted_indices (specific positions) depending on partial success modes.

Handoff Retrieval Flow

New sessions query past state through the handoff mechanism:

curl "http://127.0.0.1:49374/handoff?agent=git&session_id=abcd1234"

handle_handoff()【L12-L22】 in router.rs:

1. Parses HandoffQuery from URL parameters
2. Checks identity and admission skip headers (mirroring hook ingress)
3. Calls fetch_and_accept_handoff() to atomically retrieve and mark accepted any pending markdown for the (workspace, project) tuple
4. Returns the markdown body or empty string with 200 OK

This enables warm-start scenarios where a new agent instance receives summarized context from a previous session.

Practical Code Examples

Single Event Ingestion

curl -X POST "http://127.0.0.1:49374/hook?event=ToolUse&agent=git" \
     -H "Content-Type: application/json" \
     -d '{
           "session_id":"abcd1234",
           "cwd":"/home/user/project",
           "tool_family":"file",
           "tool_name":"file",
           "_ai_memory_capture":"{\"disposition\":\"Keep\",\"tool_family\":\"File\"}"
         }'

# Response: 202 Accepted ("queued")

Batched Event Ingestion

curl -X POST "http://127.0.0.1:49374/hook/batch" \
     -H "Content-Type: application/json" \
     -d '[{
           "url":"http://127.0.0.1:49374/hook?event=ToolUse&agent=git",
           "body":{"session_id":"s1","cwd":"/repo","tool_family":"file"}
         },
         {
           "url":"http://127.0.0.1:49374/hook?event=ToolUse&agent=git",
           "body":{"session_id":"s2","cwd":"/repo","tool_family":"file"}
         }]'

# Response: JSON with accepted count or indices

Handoff Retrieval

curl "http://127.0.0.1:49374/handoff?agent=git&session_id=abcd1234"

# Response: markdown handoff document (or empty)

Key Implementation Files

File	Responsibility
crates/ai-memory-hooks/src/router.rs	Axum routes, HTTP handlers, request dispatch
crates/ai-memory-hooks/src/payload.rs	HookEnvelope, HookQuery, request parsing
crates/ai-memory-hooks/src/capture_policy.rs	Policy inspection and enforcement logic
crates/ai-memory-hooks/src/workstream.rs	Session summary synthesis
crates/ai-memory-hooks/src/log.rs	Structured logging and metrics
crates/ai-memory-store/src/writer.rs	Persistent storage actor (invoked indirectly)

Summary

• Lifecycle hook data flow in ai-memory spans four stages: HTTP ingress, pre-processing with policy/rate filters, async persistence, and handoff retrieval
• Pre-processing occurs synchronously—assistant stripping, capture policy inspection, sub-agent filtering, global semaphore acquisition, and per-source rate limiting
• Persistence is asynchronous after returning 202 Accepted—the writer actor handles actual storage with metrics tracking
• Batch mode uses inline fail-fast processing with a 256-item limit
• Handoffs provide session continuity through atomic fetch-and-accept operations on pending markdown documents

Frequently Asked Questions

What happens when a lifecycle hook event violates a capture policy?

The inspect_capture_envelope() function returns None for drop dispositions, causing handle_hook() to immediately respond with 202 Accepted and "capture policy dropped" without entering the persistence pipeline. Metadata-only policies strip content before storage. Both outcomes are logged and counted in ingestion metrics.

How does ai-memory prevent overload from high-volume hook traffic?

Two mechanisms guard the system: a global semaphore (ingest_semaphore) caps concurrent in-flight events at DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT【L33-L36】, and per-source rate limiting via IngestRateLimiter enforces token-bucket constraints per user-session combination【L38-L44】. Excess global requests receive 503; rate-limited sources are silently skipped without failing the request.

What is the difference between single and batch hook processing?

Single events spawn a background async task for persistence after returning 202. Batches process inline up to 256 items with fail-fast semantics—any error aborts the entire batch, while policy/rate-limited items are skipped and counted as successful. Batch responses include acceptance metadata; single events do not.

When should client agents use the handoff endpoint?

Call /handoff when starting a new session that may continue work from a previous context. The endpoint retrieves and atomically accepts any pending markdown summary generated from a prior SessionEnd event. This enables warm-start behavior without requiring clients to persist state across process restarts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}