# How ai-memory Sanitizes Incoming Data Payloads: A Deep Dive into the Rust Implementation

> Learn how ai-memory sanitizes data payloads using regex secret masking, size limits, and a Sanitized newtype in Rust for secure storage in SQLite.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-09-12

---

**ai-memory sanitizes incoming data payloads through a three-step pipeline that applies regex-based secret masking, enforces configurable size limits, and wraps cleaned data in a `Sanitized<T>` newtype before persisting to SQLite.**

The akitaonrails/ai-memory project receives observation payloads from client-side lifecycle hooks through its MCP (Remote‑Message‑Control‑Protocol) endpoint. Because these payloads may contain arbitrary user-generated text—including secrets, large binary blobs, and personally identifiable information—the system implements a strict sanitization pipeline before any data reaches the SQLite store or search indexes.

## The Sanitization Architecture

The sanitization logic resides entirely within the `ai-memory-hooks` crate, structured across four critical source files that handle distinct responsibilities:

- **[`src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/sanitizer.rs)**: Implements the `Sanitized<T>` wrapper and the `sanitize` function that transforms raw payloads into safe, size-bounded equivalents.
- **[`src/payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/payload.rs)**: Defines concrete payload structures (`NewObservation`, `ToolUse`) that are fed into the sanitizer.
- **[`src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/router.rs)**: The HTTP handler for `/hook` that parses incoming JSON and immediately invokes `sanitize(payload)`.
- **[`src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/config.rs)**: Holds configurables such as `MAX_FIELD_BYTES` and compiled secret-matching regex patterns.

## Core Implementation Details

### src/sanitizer.rs: The Sanitized<T> Wrapper

The `sanitize` function in [`crates/ai-memory-hooks/src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/sanitizer.rs) serves as the primary entry point for data cleaning. It accepts any type implementing the `Sanitizeable` trait and returns a `Sanitized<T>` newtype, ensuring that only vetted data propagates through the system.

The function executes three sequential operations:

1. **Secret Masking**: Applies compiled regex patterns to strip API keys, tokens, and credential-like substrings.
2. **Size Enforcement**: Truncates fields exceeding the configured `MAX_FIELD_BYTES` limit.
3. **Type Wrapping**: Embeds the cleaned value in the `Sanitized` newtype to prevent accidental use of unprocessed data.

### src/payload.rs: Payload Structure Definitions

[`crates/ai-memory-hooks/src/payload.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/payload.rs) defines the concrete structures received from client hooks, including `NewObservation` and `ToolUse`. Each public field in these structs implements the `Sanitizeable` trait, enabling recursive cleaning of nested data structures before they reach the storage layer.

### src/router.rs: Ingress Point Enforcement

The HTTP handler defined in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) manages the `/hook` endpoint. Upon receiving a JSON payload, the router immediately calls `sanitize(payload)` and only forwards the result to `ai_memory_store::Writer` after successful validation. This guarantees that **all** persisted observations undergo sanitization before touching disk.

### src/config.rs: Security Configuration

[`crates/ai-memory-hooks/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/config.rs) maintains the security boundaries through values like `MAX_FIELD_BYTES` and secret-matching regexes (e.g., `(?i)token\s*=\s*['"][A-Za-z0-9+/=]{20,}['"]`). These settings can be overridden via [`.ai-memory.toml`](https://github.com/akitaonrails/ai-memory/blob/main/.ai-memory.toml) in the project root, but crucially, they can never be modified by the incoming payload itself, preventing privilege escalation attacks.

## The Sanitization Process Step-by-Step

When a client sends data to the `/hook` endpoint, the following pipeline executes:

1. **Regex Application**: The system iterates through compiled patterns to detect and mask tokens, API keys, and PII such as email addresses or URLs containing credentials.
2. **Field Truncation**: Any string field exceeding the byte limit defined in `MAX_FIELD_BYTES` is truncated to prevent cache exhaustion and memory pressure.
3. **Persistence Safety**: The cleaned data, now wrapped in `Sanitized<T>`, passes to the SQLite writer, ensuring that secrets never enter full-text search indexes or the web UI.

## Security and Privacy Guarantees

This ingress-focused approach provides three critical protections:

- **Secret Prevention**: By removing sensitive content at the entry point, the system never writes secrets to disk or exposes them through search results.
- **Resource Protection**: Uniform size limits prevent individual gigantic observations from exhausting SQLite's page cache or causing out-of-memory errors in writer threads.
- **Privacy Compliance**: PII patterns are redacted before storage, satisfying the privacy guarantees documented in [`SECURITY.md`](https://github.com/akitaonrails/ai-memory/blob/main/SECURITY.md).

## Using the Sanitizer in Custom Integrations

Developers can leverage the public sanitizer API when building custom hook handlers:

```rust
use ai_memory_hooks::{sanitize, NewObservation};

fn my_hook_handler(raw: NewObservation) -> Result<(), anyhow::Error> {
    // Convert the raw observation into a safe version
    let safe = sanitize(raw)?;
    // The safe value can now be stored without risk of leaking secrets
    ai_memory_store::Writer::store_observation(safe)?;
    Ok(())
}

```

This pattern ensures that custom extensions maintain the same security standards as the core system.

## Summary

- ai-memory sanitizes all incoming data through the `ai-memory-hooks` crate before persistence.
- The `sanitize` function in [`src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/sanitizer.rs) applies regex-based secret masking and size truncation.
- The `/hook` endpoint in [`src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/router.rs) enforces sanitization at the ingress point, guaranteeing no unprocessed data reaches SQLite.
- Configuration values in [`src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/config.rs) define security boundaries that cannot be overridden by payloads.
- The `Sanitized<T>` newtype pattern provides compile-time guarantees that only cleaned data enters the storage layer.

## Frequently Asked Questions

### What types of secrets does ai-memory detect and remove?

The sanitizer uses configurable regex patterns defined in [`src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/config.rs) to detect API keys, authentication tokens, and PII such as email addresses or URLs containing embedded credentials. These patterns match common secret formats like `(?i)token\s*=\s*['"][A-Za-z0-9+/=]{20,}['"]` and can be customized via [`.ai-memory.toml`](https://github.com/akitaonrails/ai-memory/blob/main/.ai-memory.toml).

### Can incoming payloads bypass the size limits?

No. The `MAX_FIELD_BYTES` configuration is hard-coded in [`src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/config.rs) and loaded from [`.ai-memory.toml`](https://github.com/akitaonrails/ai-memory/blob/main/.ai-memory.toml) at startup. Because the payload itself cannot modify these values, malicious or oversized inputs are always truncated during the sanitization phase in [`src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/sanitizer.rs) before reaching the storage layer.

### How does the Sanitized<T> wrapper improve security?

The `Sanitized<T>` newtype in [`src/sanitizer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/sanitizer.rs) provides a compile-time contract that distinguishes between raw, potentially dangerous data and vetted, safe data. Functions like `ai_memory_store::Writer::store_observation` accept only `Sanitized<T>` arguments, making it impossible to accidentally persist unsanitized observations.

### Where does sanitization occur in the request lifecycle?

Sanitization happens immediately after JSON parsing in [`src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/src/router.rs), which handles the MCP `/hook` endpoint. This ensures that secrets never touch the disk, search indexes, or web UI, as the data is cleaned before any persistence operations occur.