How Hook Payloads Are Sanitized Before Storage in ai-memory: A Security Deep Dive

Hook payloads in ai-memory undergo a mandatory three-stage sanitization pipeline using the Sanitizer type to scrub secrets via regex patterns, enforce size bounds, and guarantee that no untrusted data reaches SQLite storage.

The akitaonrails/ai-memory repository implements a zero-trust architecture for handling incoming webhook data. Every hook payload is treated as untrusted and must pass through rigorous sanitization before persistence. This article examines the specific Rust implementation that prevents secret leakage and ensures deterministic, bounded storage.

The Three-Stage Sanitization Pipeline

The system enforces sanitization at three critical integration points to provide defense-in-depth.

Stage 1: Router Ingress Scrubbing

In crates/ai-memory-hooks/src/router.rs (around line 2625), the hook-ingress router creates a Sanitized wrapper around the raw observation immediately upon receipt. This first line of defense calls Sanitizer::scrub on the title, body, and metadata fields before any further processing occurs.

// Simplified excerpt from router.rs
let raw_obs = incoming_observation;                         // untrusted JSON
let sanitized = Sanitized::new(raw_obs, &state.sanitizer); // scrubs title/body
store.write_observation(sanitized);

Stage 2: Work-Stream Event Processing

The work-stream processor in crates/ai-memory-hooks/src/workstream.rs (line 750) handles nested event structures. It iterates over each event in the payload—including checkpoints and tool calls—applying sanitize_events which invokes sanitizer.scrub on every string field.

fn sanitize_events(sanitizer: &Sanitizer, events: &mut [Event]) -> Result<()> {
    for ev in events {
        ev.content = sanitizer.scrub(&ev.content);          // body text
        if let Some(meta) = &mut ev.metadata {
            *meta = sanitizer.scrub(meta);                  // user metadata
        }
    }
    Ok(())
}

Stage 3: Storage Layer Verification

Finally, the Writer in crates/ai-memory-store/src/writer.rs (line 946) receives only the already-sanitized Sanitized observation. This guarantees that the SQLite backend never processes raw, potentially secret-containing text.

// Writer receives only sanitized data
writer.send(StoreCommand::InsertObservation {
    observation: sanitized_observation,
});

Core Sanitization Implementation

The Sanitizer type defined in crates/ai-memory-core/src/sanitize.rs provides the central logic for secret removal.

The scrub Method and SanitizerInner

The SanitizerInner struct implements the scrub(&self, input: &str) -> String method, which returns a cleaned string suitable for storage. This method applies:

• Built-in secret patterns: Compiled regular expressions matching common secrets including API keys, JWTs, OAuth tokens, and GCP keys.
• Operator-provided patterns: Custom regexes defined in the [sanitize] configuration section.
• Allow-lists: Optional patterns that prevent false-positive redaction of permitted strings.
• Size bounding: Strings exceeding the configured limit (default 2000 bytes) are truncated before regex processing to prevent denial-of-service.

Using the Sanitizer API

use ai_memory_core::Sanitizer;

// Initialize with built-in secret detection patterns
let sanitizer = Sanitizer::builtin();

// Example payload containing a JWT
let raw = "User token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...";
let safe = sanitizer.scrub(raw);

assert!(safe.contains("[REDACTED]"));   // Secret replaced

Security and Performance Guarantees

This architecture provides three critical guarantees:

1. Secret Containment: Sensitive data appearing in prompts, tool outputs, or user metadata never leaks to the database or wiki interface.
2. Deterministic Output: The same sanitizer instance produces identical outputs for identical inputs, ensuring consistent storage behavior.
3. Bounded Storage: The 2000-byte default truncation prevents attackers from exhausting storage with arbitrarily large payloads.

Summary

• Hook payloads in ai-memory are treated as untrusted by default and wrapped in a Sanitized type immediately upon ingress.
• The Sanitizer::scrub method in crates/ai-memory-core/src/sanitize.rs applies regex-based secret detection, allow-lists, and size bounding.
• Sanitization occurs at three layers: the router (initial scrubbing), the work-stream (event-level cleaning), and the writer (storage verification).
• Built-in patterns detect JWTs, API keys, OAuth tokens, and cloud provider credentials, with support for operator-defined custom rules.
• Size limits (default 2000 bytes) truncate oversized strings before processing to prevent DoS attacks.

Frequently Asked Questions

What types of secrets does ai-memory detect automatically?

The built-in sanitizer recognizes common high-entropy patterns including JSON Web Tokens (JWTs), API keys for major providers, OAuth tokens, Google Cloud Platform keys, and other credential formats. These patterns are compiled into the Sanitizer at initialization and applied via the scrub method.

Can operators customize the sanitization behavior?

Yes. Operators can extend the built-in patterns through the [sanitize] section of the configuration file, adding custom regular expressions for domain-specific secrets. Additionally, allow-lists can be configured to prevent false-positive redaction of specific permitted strings or patterns.

How does ai-memory handle oversized hook payloads?

Before regex processing, the sanitizer enforces a bounding limit of 2000 bytes by default. Any string exceeding this length is truncated. This prevents denial-of-service attacks that attempt to store arbitrarily large payloads or exhaust processing resources on massive regex operations.

Is metadata subjected to the same sanitization as content?

Yes. The work-stream processor explicitly scrubs both content fields and metadata fields. As shown in workstream.rs, the sanitize_events function applies sanitizer.scrub to the content body and iterates through any user-provided metadata, ensuring no secret-bearing fields escape processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}