# How Hook Payload Sanitization Works in ai-memory: A Technical Deep Dive

> Explore hook payload sanitization in ai-memory. Discover the three-layer defense pipeline scrubbing secrets at router, work-stream, and storage levels for robust security.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-09-11

---

**Hook payload sanitization in ai-memory is implemented through a three-layer defense pipeline that scrubs secrets at the router, work-stream, and storage levels using the `Sanitizer` type defined in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs).**

The **ai-memory** project by akitaonrails treats every incoming hook payload as untrusted data. Before any observation reaches the SQLite database, the system passes it through a rigorous sanitization process that removes API keys, JWTs, and other sensitive patterns while enforcing size constraints.

## The Three-Layer Sanitization Pipeline

The architecture enforces **hook payload sanitization** at three distinct checkpoints to ensure no raw secrets ever reach persistent storage.

### Router-Level Sanitization

When the hook-ingress router receives an incoming observation, it immediately creates a `Sanitized` wrapper around the raw data. Inside [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs), the router calls `Sanitizer::scrub` on the title, body, and metadata fields at lines 26–25.

```rust
// Inside router.rs (simplified)
let raw_obs = incoming_observation;                         // untrusted JSON
let sanitized = Sanitized::new(raw_obs, &state.sanitizer); // ← scrubs title/body
store.write_observation(sanitized);

```

This first line of defense ensures that secrets present in the initial request never propagate downstream.

### Work-Stream Processing

The work-stream code in [`crates/ai-memory-hooks/src/workstream.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/workstream.rs) handles complex payloads containing multiple events such as checkpoints and tool calls. At line 750, the system iterates over each event and applies `sanitize_events`, which invokes `sanitizer.scrub` on every string field including content, checkpoint data, and user-provided metadata.

```rust
fn sanitize_events(sanitizer: &Sanitizer, events: &mut [Event]) -> Result<()> {
    for ev in events {
        ev.content = sanitizer.scrub(&ev.content);          // body
        if let Some(meta) = &mut ev.metadata {
            *meta = sanitizer.scrub(meta);
        }
    }
    Ok(())
}

```

This step is critical for nested structures where sensitive data might appear deep within tool outputs or checkpoint descriptions.

### Storage Layer Protection

The final checkpoint resides in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) at line 946. The `Writer` component accepts only the already-sanitized `Sanitized` observation type, guaranteeing that no raw, secret-containing text ever executes an SQLite insert.

```rust
// Writer receives only the Sanitized observation.
writer.send(StoreCommand::InsertObservation {
    observation: sanitized_observation,
});

```

## How the Sanitizer Scrubs Secrets

The core **sanitization** logic lives in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs) within the `SanitizerInner` struct. This type implements the `scrub(&self, input: &str) -> String` method, which returns a cleaned string ready for storage.

### Built-in Detection Patterns

The `Sanitizer` ships with a compiled list of regular expressions matching common secrets:

- API keys and tokens
- JSON Web Tokens (JWTs)
- OAuth tokens
- GCP service account keys

When the `scrub` method encounters a match, it replaces the sensitive substring with `[REDACTED]`.

### Configurable Rules and Allow-Lists

Operators can extend the **hook payload sanitization** rules through the `[sanitize]` section of the configuration file. The system supports:

- **Extra patterns**: Custom regex definitions for domain-specific secrets
- **Allow-lists**: Patterns explicitly permitted to prevent false-positive redaction of legitimate data

### Size Bounding

Before regex processing, the sanitizer enforces a default **2,000-byte limit** on strings. Any input exceeding this threshold is truncated, preventing denial-of-service attacks that attempt to store arbitrarily large payloads.

```rust
use ai_memory_core::Sanitizer;

// The built-in sanitizer knows the common secret patterns.
let sanitizer = Sanitizer::builtin();

// Example payload that contains a JWT token.
let raw = "User token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...";
let safe = sanitizer.scrub(raw);

assert!(safe.contains("[REDACTED]"));   // secret is removed

```

## Why This Architecture Matters

- **Security**: Secrets appearing in prompts, tool outputs, or user metadata never leak to the database or wiki interfaces
- **Determinism**: Identical inputs always produce identical sanitized outputs because the same `Sanitizer` instance is used throughout the pipeline
- **Bounded Storage**: The size-capping mechanism prevents resource exhaustion attacks

## Summary

- **ai-memory** implements hook payload sanitization through a defense-in-depth strategy spanning three architectural layers
- The `Sanitizer` type in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs) provides the core `scrub` functionality using compiled regex patterns
- Sanitization occurs at the router ([`router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/router.rs)), work-stream processor ([`workstream.rs`](https://github.com/akitaonrails/ai-memory/blob/main/workstream.rs)), and storage writer ([`writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/writer.rs)) levels
- Built-in patterns detect API keys, JWTs, and OAuth tokens, while operator configuration supports custom rules and allow-lists
- A 2,000-byte default limit prevents oversized payload attacks

## Frequently Asked Questions

### What types of secrets does the ai-memory sanitizer detect by default?

The default `Sanitizer::builtin()` configuration recognizes API keys, JSON Web Tokens (JWTs), OAuth tokens, and GCP service account keys through compiled regular expressions. When matched, these patterns are replaced with `[REDACTED]` before storage.

### Can I customize the sanitization rules for my deployment?

Yes. The `[sanitize]` section in the configuration file allows operators to extend the built-in patterns with custom regular expressions and define allow-lists to prevent false-positive redaction of specific strings that resemble secrets but are safe to store.

### What happens if a hook payload exceeds the size limit?

The sanitizer applies **bounding** before regex processing. Any string longer than the configured limit (defaulting to 2,000 bytes) is truncated to prevent denial-of-service attacks and ensure storage efficiency.

### Is metadata sanitized the same way as content body?

Absolutely. The work-stream processor in [`crates/ai-memory-hooks/src/workstream.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/workstream.rs) explicitly passes both `ev.content` and `ev.metadata` through the same `sanitizer.scrub` method, ensuring comprehensive protection across all text fields in the observation.