How the ai-memory Server Sanitizes Untrusted Hook Payloads Before Storage

The ai-memory server treats every incoming hook payload as untrusted data and forces it through a deterministic Sanitizer that scrubs control characters, redacts secrets using regex patterns, enforces a 2000-character limit, and applies user-defined allow-lists before persisting to SQLite.

When lifecycle hooks post observations to the ai-memory server, the JSON payloads arrive from external sources and are inherently untrusted. The codebase implements a strict sanitization pipeline to ensure that no malicious content, secrets, or oversized data ever reaches the persistence layer.

The Ingestion Pipeline

The sanitization process begins the moment the HTTP request hits the router and continues until the data is wrapped in a type-safe container.

HTTP Handler and Raw Observation Parsing

The entry point for hook observations resides in crates/ai-memory-hooks/src/router.rs. When a POST request arrives, the handler parses the JSON body into a RawObservation struct (lines 2465‑2476). At this stage, the data contains unvalidated text from external systems.

// router.rs – handling a POST /hook request
let raw_obs = json_body.into_raw_observation();          // untrusted payload
let sanitized = Sanitized::new(raw_obs, &state.sanitizer); // <-- scrubbing happens here
store_writer.store_observation(sanitized);               // safe to persist

The router instantiates a Sanitizer (lines 3345‑3354) that contains safe‑by‑default regexes for secret detection and an optional user-provided allow‑list.

The Sanitizer Component

The Sanitizer struct defined in crates/ai-memory-core/src/sanitize.rs serves as the sole authority for cleaning text. The router calls Sanitized::new(raw_obs, &state.sanitizer) (line 132), which internally invokes Sanitizer::scrub on the title, body, and metadata fields.

The Scrubbing Algorithm

The scrub method implements a multi-layer defense strategy that processes strings deterministically.

Control Character Removal and Whitespace Trimming

The first pass removes dangerous control characters while preserving newline integrity for formatting. The implementation filters the character iterator explicitly, ensuring no invisible or terminal escape sequences survive.

// sanitize.rs – the core scrubbing routine
impl Sanitizer {
    pub fn scrub(&self, input: &str) -> String {
        let trimmed = input.trim();                     // normalize whitespace
        let no_ctl = trimmed
            .chars()
            .filter(|c| !c.is_control() || *c == '\n')
            .collect::<String>();
        let redacted = self.regexes.iter().fold(no_ctl, |s, re| {
            re.replace_all(&s, "[REDACTED]").into_owned()
        });
        redacted.chars().take(2000).collect()           // enforce size limit
    }
}

Secret Redaction and Pattern Matching

After normalization, the Sanitizer applies built‑in regexes designed to catch API keys, tokens, and other sensitive credentials. The regex set iterates over the cleaned string, replacing matches with the literal [REDACTED] token. Users may supply additional patterns while maintaining an allow‑list of safe words that should never be masked.

Size Enforcement and Allow-Lists

The final stage enforces a hard 2000‑character limit required by the storage schema. The method collects only the first 2000 characters, preventing buffer overflow or denial‑of‑service attacks through oversized payloads.

Trust Boundaries in the Storage Layer

Once sanitization completes, the system establishes a strict trust boundary that downstream components rely upon.

Writer Actor Assumptions

The sanitized data is wrapped in a Sanitized<Observation> type and handed to the writer actor in crates/ai-memory-store/src/writer.rs (lines 661‑665). This actor assumes the payload has already been scrubbed and writes directly to SQLite without additional checks, making the sanitization step the definitive security gate.

Reader API Guarantees

The reader APIs in crates/ai-memory-store/src/reader.rs (lines 695‑708) expose only already‑sanitized content to callers. Because the database contains only processed data, no secret ever leaks back to users, even if the original hook payload contained credentials or malicious strings.

Summary

• Every payload is untrusted: The server assumes all incoming hook data is potentially malicious.
• Deterministic scrubbing: The Sanitizer::scrub method in sanitize.rs removes control characters, trims whitespace, and redacts secrets using regex patterns.
• Hard limits: A 2000‑character cap prevents storage abuse and ensures schema compliance.
• Type-safe trust boundary: The Sanitized<T> wrapper ensures that only cleaned data reaches writer.rs, while reader.rs guarantees no raw content ever exits the system.

Frequently Asked Questions

What specific patterns does the Sanitizer redact by default?

The Sanitizer ships with built‑in regexes targeting common secret formats such as API keys and authentication tokens. While the exact regex definitions are customizable, the default set focuses on high‑entropy strings and known credential patterns that match typical secrets accidentally logged by hooks.

Can users configure custom regex patterns or allow-lists?

Yes. The Sanitizer accepts an optional allow‑list of safe words that should never be redacted, along with user‑provided extra regex patterns. These configurations are loaded when the router instantiates the Sanitizer at startup (lines 3345‑3354 in router.rs), allowing operators to tune the balance between security and usability.

Why is there a 2000-character limit on observations?

The limit aligns with the SQLite storage layer schema requirements and serves as a denial‑of‑service protection mechanism. By truncating input at 2000 characters during the scrub operation, the server prevents memory exhaustion and ensures consistent query performance regardless of hook payload size.

How does the trust boundary prevent secret leakage?

The architecture treats the Sanitizer as the single security gate. Once data passes through Sanitized::new and reaches writer.rs, it is considered safe. Because reader.rs only accesses sanitized records from the database, even a compromised read path cannot expose original secrets or malicious payloads that existed before scrubbing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}