How ai-memory Sanitizes Incoming Data from Untrusted Sources

ai-memory treats every piece of user-generated text as untrusted and processes it through a stateful sanitizer that redacts credential-like patterns before the data reaches durable storage, ensuring secrets never persist to SQLite or downstream systems.

The ai-memory repository (akitaonrails/ai-memory) implements a defense-in-depth approach to data privacy by treating all incoming prompts, tool outputs, and hook payloads as potentially malicious. Before any observation reaches the SQLite database or markdown wiki files, it must pass through a rigorous sanitization pipeline that redacts sensitive patterns while enforcing size constraints. This article examines the Rust implementation details showing exactly how the system sanitizes incoming data from untrusted sources.

The Stateful Sanitizer Architecture

At the core of the protection layer sits the Sanitizer struct, instantiated once at program startup and shared across all components via Arc for cheap cloning. This singleton pattern ensures consistent redaction rules across every code path that handles observations.

Sanitizer Construction and Configuration

The runtime constructs the sanitizer via Sanitizer::new(&cfg) defined in crates/ai-memory-core/src/sanitize.rs (lines 65-88). This initialization combines built-in regex patterns for common credential formats—such as bearer tokens, PEM blocks, and URL-embedded passwords—with operator-provided extra_patterns and an optional allowlist. The resulting struct wraps all compiled regexes and the allowlist in an Arc, making it thread-safe and cheap to clone for every component that writes observations.

The Scrub Method and Redaction Logic

The scrub method (lines 200-214 of the same file) implements the actual redaction logic. It iterates through each compiled regex and applies it to the input string. When a match occurs, the method checks whether the match contains a substring present in the allowlist; if so, the text remains untouched. Otherwise, the method replaces the entire match with the literal [REDACTED] string. This stateful approach ensures that even complex multi-line patterns like OpenSSL private keys get fully redacted before storage.

Enforcing Sanitization at the Type Level

Beyond runtime filtering, ai-memory uses Rust's type system to enforce that only sanitized data reaches storage layers.

The Sanitized Wrapper

The Sanitized<T> type acts as a privacy boundary marker. The only public constructor, Sanitized::new, is implemented at lines 242-251 of crates/ai-memory-core/src/sanitize.rs. When creating a Sanitized<NewObservation>, this method automatically invokes sanitizer.scrub on both the title and body fields of the observation. Because Sanitized::new is the sole entry point for creating storable observations, the compiler prevents accidental storage of raw, unsanitized strings.

Size Limits and Truncation

After redaction, Sanitized::new enforces the universal 16 KiB ceiling defined by OBSERVATION_BODY_MAX_BYTES. The implementation uses a head-tail truncation helper that preserves the beginning and end of long text while removing the middle section, ensuring that oversized observations never hit the database while maintaining some context for debugging purposes.

Integration Points and Enforcement

Sanitization is not merely a utility but an architectural requirement enforced at system boundaries.

Hook Router Ingestion

All lifecycle-hook payloads enter the system through crates/ai-memory-hooks/src/router.rs. At line 2447, the hook router receives a raw NewObservation and immediately wraps it via Sanitized::new(raw_obs, &state.sanitizer). Only the resulting Sanitized wrapper proceeds to the store; the raw observation is consumed and cannot be accessed downstream. This single chokepoint guarantees that no hook payload bypasses the scrubbing logic.

Architectural Privacy Boundaries

According to the project's docs/ARCHITECTURE.md (lines 221-230), the HTTP and MCP APIs are explicitly designed so that they cannot bypass the sanitizer. This architectural decision establishes the Sanitizer as a privacy boundary for every incoming observation, whether sourced from user prompts, tool outputs, or external webhook integrations.

Configuration and Customization

Operators can tune the sanitization behavior without modifying source code.

Custom Patterns and Allowlists

The SanitizeConfig struct allows injection of extra_patterns (additional regex strings) and an allowlist (substrings that should never be redacted). For example, you might add a pattern for internal canary tokens while allowing a public project identifier to remain visible:

use ai_memory_core::{Sanitizer, SanitizeConfig};

let cfg = SanitizeConfig {
    extra_patterns: vec![r"CANARY-\d+".to_string()],
    allowlist: vec!["PROJECT_TOKEN_PUBLIC".to_string()],
};
let custom_sanitizer = Sanitizer::new(&cfg).expect("invalid regex");

assert!(custom_sanitizer.scrub("found CANARY-42 here").contains("[REDACTED]"));
assert!(custom_sanitizer.scrub("PROJECT_TOKEN_PUBLIC=abc").contains("PROJECT_TOKEN_PUBLIC"));

Summary

• Stateful singleton: The Sanitizer is built once at startup with built-in and custom regex patterns, then shared via Arc across all threads.
• Allowlist-aware redaction: The scrub method replaces matches with [REDACTED] unless they contain explicitly allowed substrings.
• Type-level enforcement: Sanitized::new is the only path to storage, automatically scrubbing titles and bodies while enforcing a 16 KiB size limit.
• Architectural guarantees: The hook router at crates/ai-memory-hooks/src/router.rs line 2447 and API design ensure no observation bypasses sanitization.
• Configurable protection: Custom patterns and allowlists let operators adapt the sanitizer to their specific security requirements without code changes.

Frequently Asked Questions

What credential patterns does ai-memory detect by default?

According to the source code in crates/ai-memory-core/src/sanitize.rs, the built-in patterns cover bearer tokens, PEM private-key blocks, URL-embedded credentials, and environment variable assignments matching common secret naming conventions like *_KEY or *_TOKEN. The regex set is compiled once during Sanitizer::new and applied uniformly to all input strings.

Can I add custom regex patterns to the sanitizer?

Yes. The SanitizeConfig struct accepts an extra_patterns vector of strings that get compiled into regexes alongside the built-in set. Pass this configuration to Sanitizer::new(&cfg) at application startup to augment the default detection rules with organization-specific patterns such as internal API key formats or canary tokens.

How does the allowlist prevent false positives?

The allowlist parameter in SanitizeConfig contains substrings that the scrub method checks before applying redaction. If a regex match contains any allowlisted substring, the sanitizer skips replacement for that specific match. This prevents legitimate public identifiers—such as PROJECT_TOKEN_PUBLIC—from being masked while still catching genuinely sensitive data that follows similar patterns.

Is it possible to bypass the sanitizer for trusted internal data?

No. The architecture documentation in docs/ARCHITECTURE.md explicitly states that the HTTP and MCP APIs cannot bypass the sanitizer. The Sanitized<T> type wrapper ensures that the compiler, not just runtime checks, prevents raw observations from reaching storage. Attempting to store unsanitized data would require modifying the core library to violate the type constraints defined in crates/ai-memory-core/src/sanitize.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}