How AI Agent Session Observations Are Sanitized Before Storage in ai-memory

AI agent session observations are sanitized through a multi-layer pipeline that compiles regex patterns at startup, scrubs credentials using the Sanitizer::scrub method, enforces type safety via the Sanitized<T> wrapper, and applies a hard 16 KiB size limit before persisting to SQLite.

The ai-memory repository treats all incoming agent data as untrusted. Before any observation reaches the durable store, it must pass through a strict sanitization contract implemented in Rust. This architecture prevents credential leaks while maintaining observability by redacting sensitive patterns and respecting explicit allowlists.

The Sanitizer Architecture

The core of the privacy pipeline is the Sanitizer type defined in crates/ai-memory-core/src/sanitize.rs at lines 132-134. This stateful scrubber is designed for concurrent access using an Arc<SanitizerInner> backing, allowing cheap cloning across async boundaries without recompiling regex patterns.

Key design characteristics:

• Singleton pattern: Built once at startup and shared across the application lifecycle
• Thread-safe: Uses Arc internally for lock-free read access during scrubbing operations
• Fail-fast: Invalid user-supplied patterns abort initialization, guaranteeing the sanitizer is always functional

Built-in Redaction Patterns

The sanitizer ships with comprehensive credential detection via BUILTIN_PATTERN_STRS (lines 48-127 of sanitize.rs). These patterns cover:

• Bearer tokens and API keys (OpenAI, Anthropic, AWS)
• GitHub personal access tokens
• Private key blocks (RSA, EC, OpenSSH)
• URLs containing embedded credentials
• Generic high-entropy secrets

These regexes are compiled during Sanitizer::new and stored in the Arc-wrapped inner structure for efficient matching.

Configuration and Construction

Operator customization happens through SanitizeConfig, allowing extra_patterns and allowlist entries to augment or override built-in behavior. The constructor implementation (lines 165-187) validates all patterns at startup:

// Build a sanitizer at start-up using config from config.toml
let cfg = SanitizeConfig {
    extra_patterns: vec![r"my_secret_\w+".to_string()],
    allowlist: vec!["PROJECT_TOKEN".to_string()],
};
let sanitizer = Sanitizer::new(&cfg).expect("invalid pattern");

Any compilation error in user-provided regexes triggers an immediate panic during initialization, preventing runtime failures during scrubbing.

The Scrubbing Logic

The Sanitizer::scrub method (lines 196-200) implements the actual redaction logic:

1. Walks the input string sequentially
2. Replaces every pattern match with the literal [REDACTED]
3. Checks the allowlist before redacting; if the match contains an allowed term, the original text is preserved

// Scrub a raw observation string
let raw = "User entered OpenAI key: sk-abcdef1234567890";
let clean = sanitizer.scrub(raw);
// clean == "User entered OpenAI key: [REDACTED]"

This ensures that sensitive agent outputs are stripped while legitimate diagnostic tokens can pass through when explicitly allowed.

Type-Safe Enforcement with Sanitized

To guarantee that only scrubbed data reaches the persistence layer, ai-memory uses the Sanitized<T> wrapper (lines 228-229). This is a zero-cost abstraction that encodes the "sanitize-then-store" invariant into the type system.

The only way to create a persisted observation is via Sanitized::new, which requires:

// Wrap the cleaned observation so it can be stored
let obs = NewObservation::new(clean);
let sanitized_obs = Sanitized::new(obs, &sanitizer);
store.write_observation(sanitized_obs);

The store writer (in crates/ai-memory-store/src/writer.rs) accepts only Sanitized types, making it impossible to accidentally persist raw, unsanitized agent output.

Integration in the Hook Pipeline

The enforcement boundary appears in crates/ai-memory-hooks/src/router.rs around lines 2490-2505. When a lifecycle hook payload arrives at the server:

1. The router extracts the raw observation
2. Builds a Sanitized<NewObservation> by calling Sanitized::new(raw_obs, &state.sanitizer)
3. Passes the wrapped object to the store writer

This single chokepoint ensures every stored observation has passed through the sanitizer, regardless of the agent source or hook type.

Size Limits and Payload Validation

After content sanitization but before SQLite insertion, observations are bounded by OBSERVATION_BODY_MAX_BYTES (16 KiB), defined at lines 43-46 of sanitize.rs. Any payload exceeding this limit is rejected before reaching the writer, preventing denial-of-service attacks through oversized memory dumps or log floods.

Summary

• Untrusted by default: All agent observations are treated as potentially containing secrets
• Pattern-driven: Built-in regexes detect credentials while supporting custom extra_patterns
• Allowlist-aware: Explicit permits prevent over-redaction of necessary diagnostic tokens
• Type-safe: The Sanitized<T> wrapper makes unsafe storage unrepresentable in the codebase
• Size-guarded: Hard 16 KiB limit prevents store abuse

Frequently Asked Questions

How does ai-memory prevent API keys from leaking into stored observations?

According to the ai-memory source code, the Sanitizer::scrub method replaces any substring matching built-in credential patterns (like sk- prefixed OpenAI keys or AWS secrets) with the literal [REDACTED]. This happens automatically for all observations before they reach the SQLite store.

Can I customize which patterns get redacted in ai-memory?

Yes. The SanitizeConfig struct accepts extra_patterns (a vector of custom regex strings) and allowlist entries (strings that exempt matches from redaction). These are compiled alongside BUILTIN_PATTERN_STRS during Sanitizer::new initialization in crates/ai-memory-core/src/sanitize.rs.

What happens if an agent sends an observation larger than 16 KiB?

Observations exceeding OBSERVATION_BODY_MAX_BYTES (16 KiB) are rejected by the size guard before sanitization completes. This prevents oversized payloads from entering the SQLite database and protects against storage exhaustion attacks.

Is it possible to accidentally store unsanitized data in ai-memory?

No. The Sanitized<T> wrapper in sanitize.rs provides a type-safe boundary. The store writer only accepts Sanitized variants, and the hook router in crates/ai-memory-hooks/src/router.rs wraps all incoming observations before passing them to persistence. This makes unsanitized storage a compile-time error.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}