How Hook Payload Sanitization Works in ai-memory: A Technical Deep Dive

Hook payload sanitization in ai-memory is implemented through a three-layer defense pipeline that scrubs secrets at the router, work-stream, and storage levels using the Sanitizer type defined in crates/ai-memory-core/src/sanitize.rs.

The ai-memory project by akitaonrails treats every incoming hook payload as untrusted data. Before any observation reaches the SQLite database, the system passes it through a rigorous sanitization process that removes API keys, JWTs, and other sensitive patterns while enforcing size constraints.

The Three-Layer Sanitization Pipeline

The architecture enforces hook payload sanitization at three distinct checkpoints to ensure no raw secrets ever reach persistent storage.

Router-Level Sanitization

When the hook-ingress router receives an incoming observation, it immediately creates a Sanitized wrapper around the raw data. Inside crates/ai-memory-hooks/src/router.rs, the router calls Sanitizer::scrub on the title, body, and metadata fields at lines 26–25.

// Inside router.rs (simplified)
let raw_obs = incoming_observation;                         // untrusted JSON
let sanitized = Sanitized::new(raw_obs, &state.sanitizer); // ← scrubs title/body
store.write_observation(sanitized);

This first line of defense ensures that secrets present in the initial request never propagate downstream.

Work-Stream Processing

The work-stream code in crates/ai-memory-hooks/src/workstream.rs handles complex payloads containing multiple events such as checkpoints and tool calls. At line 750, the system iterates over each event and applies sanitize_events, which invokes sanitizer.scrub on every string field including content, checkpoint data, and user-provided metadata.

fn sanitize_events(sanitizer: &Sanitizer, events: &mut [Event]) -> Result<()> {
    for ev in events {
        ev.content = sanitizer.scrub(&ev.content);          // body
        if let Some(meta) = &mut ev.metadata {
            *meta = sanitizer.scrub(meta);
        }
    }
    Ok(())
}

This step is critical for nested structures where sensitive data might appear deep within tool outputs or checkpoint descriptions.

Storage Layer Protection

The final checkpoint resides in crates/ai-memory-store/src/writer.rs at line 946. The Writer component accepts only the already-sanitized Sanitized observation type, guaranteeing that no raw, secret-containing text ever executes an SQLite insert.

// Writer receives only the Sanitized observation.
writer.send(StoreCommand::InsertObservation {
    observation: sanitized_observation,
});

How the Sanitizer Scrubs Secrets

The core sanitization logic lives in crates/ai-memory-core/src/sanitize.rs within the SanitizerInner struct. This type implements the scrub(&self, input: &str) -> String method, which returns a cleaned string ready for storage.

Built-in Detection Patterns

The Sanitizer ships with a compiled list of regular expressions matching common secrets:

• API keys and tokens
• JSON Web Tokens (JWTs)
• OAuth tokens
• GCP service account keys

When the scrub method encounters a match, it replaces the sensitive substring with [REDACTED].

Configurable Rules and Allow-Lists

Operators can extend the hook payload sanitization rules through the [sanitize] section of the configuration file. The system supports:

• Extra patterns: Custom regex definitions for domain-specific secrets
• Allow-lists: Patterns explicitly permitted to prevent false-positive redaction of legitimate data

Size Bounding

Before regex processing, the sanitizer enforces a default 2,000-byte limit on strings. Any input exceeding this threshold is truncated, preventing denial-of-service attacks that attempt to store arbitrarily large payloads.

use ai_memory_core::Sanitizer;

// The built-in sanitizer knows the common secret patterns.
let sanitizer = Sanitizer::builtin();

// Example payload that contains a JWT token.
let raw = "User token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...";
let safe = sanitizer.scrub(raw);

assert!(safe.contains("[REDACTED]"));   // secret is removed

Why This Architecture Matters

• Security: Secrets appearing in prompts, tool outputs, or user metadata never leak to the database or wiki interfaces
• Determinism: Identical inputs always produce identical sanitized outputs because the same Sanitizer instance is used throughout the pipeline
• Bounded Storage: The size-capping mechanism prevents resource exhaustion attacks

Summary

• ai-memory implements hook payload sanitization through a defense-in-depth strategy spanning three architectural layers
• The Sanitizer type in crates/ai-memory-core/src/sanitize.rs provides the core scrub functionality using compiled regex patterns
• Sanitization occurs at the router (router.rs), work-stream processor (workstream.rs), and storage writer (writer.rs) levels
• Built-in patterns detect API keys, JWTs, and OAuth tokens, while operator configuration supports custom rules and allow-lists
• A 2,000-byte default limit prevents oversized payload attacks

Frequently Asked Questions

What types of secrets does the ai-memory sanitizer detect by default?

The default Sanitizer::builtin() configuration recognizes API keys, JSON Web Tokens (JWTs), OAuth tokens, and GCP service account keys through compiled regular expressions. When matched, these patterns are replaced with [REDACTED] before storage.

Can I customize the sanitization rules for my deployment?

Yes. The [sanitize] section in the configuration file allows operators to extend the built-in patterns with custom regular expressions and define allow-lists to prevent false-positive redaction of specific strings that resemble secrets but are safe to store.

What happens if a hook payload exceeds the size limit?

The sanitizer applies bounding before regex processing. Any string longer than the configured limit (defaulting to 2,000 bytes) is truncated to prevent denial-of-service attacks and ensure storage efficiency.

Is metadata sanitized the same way as content body?

Absolutely. The work-stream processor in crates/ai-memory-hooks/src/workstream.rs explicitly passes both ev.content and ev.metadata through the same sanitizer.scrub method, ensuring comprehensive protection across all text fields in the observation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}