What Is the Hook Capture Pipeline and How Are Observations Sanitized in ai-memory

The hook capture pipeline is a fire-and-forget, backpressured data flow that converts untrusted script output into safe, searchable observations by passing every payload through the ai-memory-hooks sanitizer before atomic storage.

The akitaonrails/ai-memory repository implements a strict trust boundary between external automation and its SQLite-backed memory store. Understanding the hook capture pipeline and how observations are sanitized is essential for anyone integrating custom scripts or debugging ingestion failures. The entire flow—from the CLI command to the single-writer actor—is designed to remain bounded, durable, and secure by default.

How the Hook Capture Pipeline Works

A six-stage pipeline moves data from a user script to indexed storage without blocking the caller. Each stage is deliberately bounded to prevent a misbehaving hook from stalling the service.

Step 1: Triggering a Hook from the CLI

A user-defined hook script writes a JSON payload to stdout and pipes it into the built-in hook capture command. The CLI, implemented in crates/ai-memory-cli/src/commands/hook_capture.rs, immediately POSTs this payload to the local MCP HTTP endpoint /hook. This keeps the automation layer simple and stateless.

Step 2: Reception and Immediate Acknowledgment

The MCP router in crates/ai-memory-mcp/src/router.rs accepts the request and returns 202 Accepted right away. If the server is saturated, it replies with 429 Too Many Requests instead of queueing indefinitely. The router never blocks the client while waiting for storage; the handshake ends as soon as the payload is handed to the next stage.

Step 3: Sanitizing Every Payload

All untrusted text crosses the trust boundary inside crates/ai-memory-hooks/src/lib.rs. The sanitize() function parses the raw JSON, strips any private data, enforces the bounded schema, and wraps the result in a Sanitized<NewObservation> type. Because the Sanitized wrapper has no public constructor other than sanitize(), no other module can bypass the sanitizer.

Step 4: Atomic Storage via the Single-Writer Actor

After sanitization, the observation is sent over an mpsc channel to the writer actor, defined in crates/ai-memory-store/src/writer.rs. The WriterHandle runs on a dedicated Tokio thread and executes a single SQLite transaction that inserts the observation and updates the full-text-search (FTS5) indexes atomically. This serializes writes and eliminates contention.

Step 5: Durable Commit and Full-Text Indexing

The storage layer finalizes each write using atomic file-write semantics—temporary file, rename, and fsync—ensuring durability even if the process crashes. The FTS5 index is updated inside the same transaction, so the observation becomes searchable immediately after commit. This logic is coordinated through crates/ai-memory-wiki/src/wiki.rs.

Step 6: Backpressure and Hard Timeouts

Hooks are fire-and-forget. Script hooks hard-timeout at ≤ 200 ms, and the server never performs an unbounded tokio::spawn fan-out on hook paths. If the writer queue is full, the /hook endpoint returns 429, forcing the caller to throttle rather than overwhelm the system.

How Observations Are Sanitized

The sanitization layer is the only authorized gate between untrusted input and trusted storage.

The sanitize() Function and Sanitized<T> Wrapper

In crates/ai-memory-hooks/src/lib.rs, the sanitize() function receives a NewObservationRaw struct and returns Result<Sanitized<NewObservation>, Error>. The Sanitized<T> wrapper is opaque: downstream modules can read the inner value but cannot construct an instance themselves. This design guarantees that every observation entering the store has passed through the same validation path.

Schema Validation and Private-Data Removal

The sanitizer validates required fields such as workspace_id, project_id, path, and content. It normalizes timestamps using the jiff crate and strips any fields marked as private in the schema. Because the function is the explicit trust boundary, there are no alternative code paths that allow raw payloads to reach the writer actor.

/// Example (simplified) from `crates/ai-memory-hooks/src/lib.rs`
pub fn sanitize(raw: NewObservationRaw) -> Result<Sanitized<NewObservation>, Error> {
    // Strip private fields
    let cleaned = raw.strip_private();
    // Validate schema
    cleaned.validate()?;
    // Normalise timestamps
    let ts = jiff::Timestamp::from_utc_nanoseconds(cleaned.timestamp)?;
    Ok(Sanitized::new(NewObservation {
        workspace_id: cleaned.workspace_id,
        project_id: cleaned.project_id,
        path: cleaned.path,
        content: cleaned.content,
        timestamp: ts,
        // …other public fields…
    }))
}

Practical Examples

Capturing a Hook from the CLI

# Run a user-defined script that emits JSON and pipe it to the CLI

my-hook-script | ai-memory hook capture --project myproj --workspace myws

The command reads the script’s stdout, POSTs it to /hook, and immediately returns 202 Accepted. The payload is then processed by the pipeline described above.

Sending a Test Payload with curl

curl -X POST http://127.0.0.1:49374/hook \
     -H "Content-Type: application/json" \
     -d '{"workspace_id":"ws1","project_id":"p1","path":"notes/todo.md","content":"Buy milk"}'

The server returns 202 right away, then internally calls sanitize() and stores the observation through the writer actor.

Observing Backpressure Under Load

# Simulate a saturated server

for i in {1..1000}; do
  curl -s -o /dev/null -X POST http://127.0.0.1:49374/hook \
       -d '{"workspace_id":"ws","project_id":"p","path":"x","content":"y"}' &
done
wait

If the writer queue overflows, the server replies with 429 Too Many Requests, signalling the caller to throttle.

Summary

• The hook capture pipeline starts in crates/ai-memory-cli/src/commands/hook_capture.rs and ends with an atomic SQLite commit in the writer actor.
• The MCP router in crates/ai-memory-mcp/src/router.rs immediately returns 202 or 429, keeping the interface non-blocking.
• Every payload is converted from NewObservationRaw to Sanitized<NewObservation> by the sanitize() function in crates/ai-memory-hooks/src/lib.rs.
• The single-writer actor in crates/ai-memory-store/src/writer.rs serializes all inserts and updates FTS5 indexes in one transaction.
• Hard timeouts of ≤ 200 ms and bounded concurrency prevent hook scripts from destabilizing the service.

Frequently Asked Questions

Why does the /hook endpoint return 202 instead of 200?

The 202 Accepted status signals that the request has been accepted for processing but not yet fully persisted. Because the hook capture pipeline is fire-and-forget, the server acknowledges receipt immediately and performs sanitization and storage asynchronously through the writer actor.

What happens if a hook payload contains invalid or private fields?

The sanitize() function in crates/ai-memory-hooks/src/lib.rs strips private fields and enforces the schema before the data reaches the store. If validation fails, the error is logged and the observation is rejected without corrupting the database.

Can I bypass the sanitizer to write observations faster?

No. The Sanitized<NewObservation> type has no public constructor outside of sanitize(), so no other crate—including the MCP router or the CLI—can inject an unverified payload into the writer actor.

How does the system prevent hooks from overwhelming the database?

The server caps hook execution at ≤ 200 ms and returns 429 Too Many Requests when the writer queue is saturated. This backpressure mechanism, combined with the single-writer actor design, ensures that ingestion never outpaces SQLite’s capacity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}