How ai-memory Sanitizes Incoming Data Payloads: A Deep Dive into the Rust Implementation

ai-memory sanitizes incoming data payloads through a three-step pipeline that applies regex-based secret masking, enforces configurable size limits, and wraps cleaned data in a Sanitized<T> newtype before persisting to SQLite.

The akitaonrails/ai-memory project receives observation payloads from client-side lifecycle hooks through its MCP (Remote‑Message‑Control‑Protocol) endpoint. Because these payloads may contain arbitrary user-generated text—including secrets, large binary blobs, and personally identifiable information—the system implements a strict sanitization pipeline before any data reaches the SQLite store or search indexes.

The Sanitization Architecture

The sanitization logic resides entirely within the ai-memory-hooks crate, structured across four critical source files that handle distinct responsibilities:

• src/sanitizer.rs: Implements the Sanitized<T> wrapper and the sanitize function that transforms raw payloads into safe, size-bounded equivalents.
• src/payload.rs: Defines concrete payload structures (NewObservation, ToolUse) that are fed into the sanitizer.
• src/router.rs: The HTTP handler for /hook that parses incoming JSON and immediately invokes sanitize(payload).
• src/config.rs: Holds configurables such as MAX_FIELD_BYTES and compiled secret-matching regex patterns.

Core Implementation Details

src/sanitizer.rs: The Sanitized Wrapper

The sanitize function in crates/ai-memory-hooks/src/sanitizer.rs serves as the primary entry point for data cleaning. It accepts any type implementing the Sanitizeable trait and returns a Sanitized<T> newtype, ensuring that only vetted data propagates through the system.

The function executes three sequential operations:

1. Secret Masking: Applies compiled regex patterns to strip API keys, tokens, and credential-like substrings.
2. Size Enforcement: Truncates fields exceeding the configured MAX_FIELD_BYTES limit.
3. Type Wrapping: Embeds the cleaned value in the Sanitized newtype to prevent accidental use of unprocessed data.

src/payload.rs: Payload Structure Definitions

crates/ai-memory-hooks/src/payload.rs defines the concrete structures received from client hooks, including NewObservation and ToolUse. Each public field in these structs implements the Sanitizeable trait, enabling recursive cleaning of nested data structures before they reach the storage layer.

src/router.rs: Ingress Point Enforcement

The HTTP handler defined in crates/ai-memory-hooks/src/router.rs manages the /hook endpoint. Upon receiving a JSON payload, the router immediately calls sanitize(payload) and only forwards the result to ai_memory_store::Writer after successful validation. This guarantees that all persisted observations undergo sanitization before touching disk.

src/config.rs: Security Configuration

crates/ai-memory-hooks/src/config.rs maintains the security boundaries through values like MAX_FIELD_BYTES and secret-matching regexes (e.g., (?i)token\s*=\s*['"][A-Za-z0-9+/=]{20,}['"]). These settings can be overridden via .ai-memory.toml in the project root, but crucially, they can never be modified by the incoming payload itself, preventing privilege escalation attacks.

The Sanitization Process Step-by-Step

When a client sends data to the /hook endpoint, the following pipeline executes:

1. Regex Application: The system iterates through compiled patterns to detect and mask tokens, API keys, and PII such as email addresses or URLs containing credentials.
2. Field Truncation: Any string field exceeding the byte limit defined in MAX_FIELD_BYTES is truncated to prevent cache exhaustion and memory pressure.
3. Persistence Safety: The cleaned data, now wrapped in Sanitized<T>, passes to the SQLite writer, ensuring that secrets never enter full-text search indexes or the web UI.

Security and Privacy Guarantees

This ingress-focused approach provides three critical protections:

• Secret Prevention: By removing sensitive content at the entry point, the system never writes secrets to disk or exposes them through search results.
• Resource Protection: Uniform size limits prevent individual gigantic observations from exhausting SQLite's page cache or causing out-of-memory errors in writer threads.
• Privacy Compliance: PII patterns are redacted before storage, satisfying the privacy guarantees documented in SECURITY.md.

Using the Sanitizer in Custom Integrations

Developers can leverage the public sanitizer API when building custom hook handlers:

use ai_memory_hooks::{sanitize, NewObservation};

fn my_hook_handler(raw: NewObservation) -> Result<(), anyhow::Error> {
    // Convert the raw observation into a safe version
    let safe = sanitize(raw)?;
    // The safe value can now be stored without risk of leaking secrets
    ai_memory_store::Writer::store_observation(safe)?;
    Ok(())
}

This pattern ensures that custom extensions maintain the same security standards as the core system.

Summary

• ai-memory sanitizes all incoming data through the ai-memory-hooks crate before persistence.
• The sanitize function in src/sanitizer.rs applies regex-based secret masking and size truncation.
• The /hook endpoint in src/router.rs enforces sanitization at the ingress point, guaranteeing no unprocessed data reaches SQLite.
• Configuration values in src/config.rs define security boundaries that cannot be overridden by payloads.
• The Sanitized<T> newtype pattern provides compile-time guarantees that only cleaned data enters the storage layer.

Frequently Asked Questions

What types of secrets does ai-memory detect and remove?

The sanitizer uses configurable regex patterns defined in src/config.rs to detect API keys, authentication tokens, and PII such as email addresses or URLs containing embedded credentials. These patterns match common secret formats like (?i)token\s*=\s*['"][A-Za-z0-9+/=]{20,}['"] and can be customized via .ai-memory.toml.

Can incoming payloads bypass the size limits?

No. The MAX_FIELD_BYTES configuration is hard-coded in src/config.rs and loaded from .ai-memory.toml at startup. Because the payload itself cannot modify these values, malicious or oversized inputs are always truncated during the sanitization phase in src/sanitizer.rs before reaching the storage layer.

How does the Sanitized wrapper improve security?

The Sanitized<T> newtype in src/sanitizer.rs provides a compile-time contract that distinguishes between raw, potentially dangerous data and vetted, safe data. Functions like ai_memory_store::Writer::store_observation accept only Sanitized<T> arguments, making it impossible to accidentally persist unsanitized observations.

Where does sanitization occur in the request lifecycle?

Sanitization happens immediately after JSON parsing in src/router.rs, which handles the MCP /hook endpoint. This ensures that secrets never touch the disk, search indexes, or web UI, as the data is cleaned before any persistence operations occur.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}