# How the ai-memory Server Sanitizes Untrusted Hook Payloads Before Storage

> Learn how the ai-memory server sanitizes untrusted hook payloads. It scrubs control characters, redacts secrets, limits length, and uses allow-lists before storing data.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-25

---

**The ai-memory server treats every incoming hook payload as untrusted data and forces it through a deterministic `Sanitizer` that scrubs control characters, redacts secrets using regex patterns, enforces a 2000-character limit, and applies user-defined allow-lists before persisting to SQLite.**

When lifecycle hooks post observations to the **ai-memory** server, the JSON payloads arrive from external sources and are inherently untrusted. The codebase implements a strict sanitization pipeline to ensure that no malicious content, secrets, or oversized data ever reaches the persistence layer.

## The Ingestion Pipeline

The sanitization process begins the moment the HTTP request hits the router and continues until the data is wrapped in a type-safe container.

### HTTP Handler and Raw Observation Parsing

The entry point for hook observations resides in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs). When a POST request arrives, the handler parses the JSON body into a `RawObservation` struct (lines 2465‑2476). At this stage, the data contains unvalidated text from external systems.

```rust
// router.rs – handling a POST /hook request
let raw_obs = json_body.into_raw_observation();          // untrusted payload
let sanitized = Sanitized::new(raw_obs, &state.sanitizer); // <-- scrubbing happens here
store_writer.store_observation(sanitized);               // safe to persist

```

The router instantiates a **Sanitizer** (lines 3345‑3354) that contains safe‑by‑default regexes for secret detection and an optional user-provided allow‑list.

### The Sanitizer Component

The `Sanitizer` struct defined in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs) serves as the sole authority for cleaning text. The router calls `Sanitized::new(raw_obs, &state.sanitizer)` (line 132), which internally invokes `Sanitizer::scrub` on the title, body, and metadata fields.

## The Scrubbing Algorithm

The `scrub` method implements a multi-layer defense strategy that processes strings deterministically.

### Control Character Removal and Whitespace Trimming

The first pass removes dangerous control characters while preserving newline integrity for formatting. The implementation filters the character iterator explicitly, ensuring no invisible or terminal escape sequences survive.

```rust
// sanitize.rs – the core scrubbing routine
impl Sanitizer {
    pub fn scrub(&self, input: &str) -> String {
        let trimmed = input.trim();                     // normalize whitespace
        let no_ctl = trimmed
            .chars()
            .filter(|c| !c.is_control() || *c == '\n')
            .collect::<String>();
        let redacted = self.regexes.iter().fold(no_ctl, |s, re| {
            re.replace_all(&s, "[REDACTED]").into_owned()
        });
        redacted.chars().take(2000).collect()           // enforce size limit
    }
}

```

### Secret Redaction and Pattern Matching

After normalization, the **Sanitizer** applies built‑in regexes designed to catch API keys, tokens, and other sensitive credentials. The regex set iterates over the cleaned string, replacing matches with the literal `[REDACTED]` token. Users may supply additional patterns while maintaining an allow‑list of safe words that should never be masked.

### Size Enforcement and Allow-Lists

The final stage enforces a hard **2000‑character limit** required by the storage schema. The method collects only the first 2000 characters, preventing buffer overflow or denial‑of‑service attacks through oversized payloads.

## Trust Boundaries in the Storage Layer

Once sanitization completes, the system establishes a strict trust boundary that downstream components rely upon.

### Writer Actor Assumptions

The sanitized data is wrapped in a `Sanitized<Observation>` type and handed to the writer actor in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) (lines 661‑665). This actor assumes the payload has already been scrubbed and writes directly to SQLite without additional checks, making the sanitization step the definitive security gate.

### Reader API Guarantees

The reader APIs in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (lines 695‑708) expose only already‑sanitized content to callers. Because the database contains only processed data, no secret ever leaks back to users, even if the original hook payload contained credentials or malicious strings.

## Summary

- **Every payload is untrusted**: The server assumes all incoming hook data is potentially malicious.
- **Deterministic scrubbing**: The `Sanitizer::scrub` method in [`sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/sanitize.rs) removes control characters, trims whitespace, and redacts secrets using regex patterns.
- **Hard limits**: A 2000‑character cap prevents storage abuse and ensures schema compliance.
- **Type-safe trust boundary**: The `Sanitized<T>` wrapper ensures that only cleaned data reaches [`writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/writer.rs), while [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) guarantees no raw content ever exits the system.

## Frequently Asked Questions

### What specific patterns does the Sanitizer redact by default?

The **Sanitizer** ships with built‑in regexes targeting common secret formats such as API keys and authentication tokens. While the exact regex definitions are customizable, the default set focuses on high‑entropy strings and known credential patterns that match typical secrets accidentally logged by hooks.

### Can users configure custom regex patterns or allow-lists?

Yes. The `Sanitizer` accepts an optional allow‑list of safe words that should never be redacted, along with user‑provided extra regex patterns. These configurations are loaded when the router instantiates the `Sanitizer` at startup (lines 3345‑3354 in [`router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/router.rs)), allowing operators to tune the balance between security and usability.

### Why is there a 2000-character limit on observations?

The limit aligns with the **SQLite storage layer** schema requirements and serves as a denial‑of‑service protection mechanism. By truncating input at 2000 characters during the `scrub` operation, the server prevents memory exhaustion and ensures consistent query performance regardless of hook payload size.

### How does the trust boundary prevent secret leakage?

The architecture treats the **Sanitizer** as the single security gate. Once data passes through `Sanitized::new` and reaches [`writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/writer.rs), it is considered safe. Because [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) only accesses sanitized records from the database, even a compromised read path cannot expose original secrets or malicious payloads that existed before scrubbing.