# How ai-memory Sanitizes Incoming Data from Untrusted Sources > Discover how ai-memory sanitizes untrusted data by redacting credentials before storage. Learn how this stateful sanitizer protects your secrets from SQLite and downstream systems. Enhance your data security today. - Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory) - Tags: security - Published: 2026-08-21 --- **ai-memory treats every piece of user-generated text as untrusted and processes it through a stateful sanitizer that redacts credential-like patterns before the data reaches durable storage, ensuring secrets never persist to SQLite or downstream systems.** The `ai-memory` repository (akitaonrails/ai-memory) implements a defense-in-depth approach to data privacy by treating all incoming prompts, tool outputs, and hook payloads as potentially malicious. Before any observation reaches the SQLite database or markdown wiki files, it must pass through a rigorous sanitization pipeline that redacts sensitive patterns while enforcing size constraints. This article examines the Rust implementation details showing exactly how the system sanitizes incoming data from untrusted sources. ## The Stateful Sanitizer Architecture At the core of the protection layer sits the `Sanitizer` struct, instantiated once at program startup and shared across all components via `Arc` for cheap cloning. This singleton pattern ensures consistent redaction rules across every code path that handles observations. ### Sanitizer Construction and Configuration The runtime constructs the sanitizer via `Sanitizer::new(&cfg)` defined in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs) (lines 65-88). This initialization combines built-in regex patterns for common credential formats—such as bearer tokens, PEM blocks, and URL-embedded passwords—with operator-provided `extra_patterns` and an optional `allowlist`. The resulting struct wraps all compiled regexes and the allowlist in an `Arc`, making it thread-safe and cheap to clone for every component that writes observations. ### The Scrub Method and Redaction Logic The `scrub` method (lines 200-214 of the same file) implements the actual redaction logic. It iterates through each compiled regex and applies it to the input string. When a match occurs, the method checks whether the match contains a substring present in the allowlist; if so, the text remains untouched. Otherwise, the method replaces the entire match with the literal `[REDACTED]` string. This stateful approach ensures that even complex multi-line patterns like OpenSSL private keys get fully redacted before storage. ## Enforcing Sanitization at the Type Level Beyond runtime filtering, `ai-memory` uses Rust's type system to enforce that only sanitized data reaches storage layers. ### The Sanitized Wrapper The `Sanitized` type acts as a privacy boundary marker. The only public constructor, `Sanitized::new`, is implemented at lines 242-251 of [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs). When creating a `Sanitized`, this method automatically invokes `sanitizer.scrub` on both the title and body fields of the observation. Because `Sanitized::new` is the sole entry point for creating storable observations, the compiler prevents accidental storage of raw, unsanitized strings. ### Size Limits and Truncation After redaction, `Sanitized::new` enforces the universal **16 KiB ceiling** defined by `OBSERVATION_BODY_MAX_BYTES`. The implementation uses a head-tail truncation helper that preserves the beginning and end of long text while removing the middle section, ensuring that oversized observations never hit the database while maintaining some context for debugging purposes. ## Integration Points and Enforcement Sanitization is not merely a utility but an architectural requirement enforced at system boundaries. ### Hook Router Ingestion All lifecycle-hook payloads enter the system through [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs). At line 2447, the hook router receives a raw `NewObservation` and immediately wraps it via `Sanitized::new(raw_obs, &state.sanitizer)`. Only the resulting `Sanitized` wrapper proceeds to the store; the raw observation is consumed and cannot be accessed downstream. This single chokepoint guarantees that no hook payload bypasses the scrubbing logic. ### Architectural Privacy Boundaries According to the project's [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) (lines 221-230), the HTTP and MCP APIs are explicitly designed so that they cannot bypass the sanitizer. This architectural decision establishes the `Sanitizer` as a **privacy boundary** for every incoming observation, whether sourced from user prompts, tool outputs, or external webhook integrations. ## Configuration and Customization Operators can tune the sanitization behavior without modifying source code. ### Custom Patterns and Allowlists The `SanitizeConfig` struct allows injection of `extra_patterns` (additional regex strings) and an `allowlist` (substrings that should never be redacted). For example, you might add a pattern for internal canary tokens while allowing a public project identifier to remain visible: ```rust use ai_memory_core::{Sanitizer, SanitizeConfig}; let cfg = SanitizeConfig { extra_patterns: vec![r"CANARY-\d+".to_string()], allowlist: vec!["PROJECT_TOKEN_PUBLIC".to_string()], }; let custom_sanitizer = Sanitizer::new(&cfg).expect("invalid regex"); assert!(custom_sanitizer.scrub("found CANARY-42 here").contains("[REDACTED]")); assert!(custom_sanitizer.scrub("PROJECT_TOKEN_PUBLIC=abc").contains("PROJECT_TOKEN_PUBLIC")); ``` ## Summary - **Stateful singleton**: The `Sanitizer` is built once at startup with built-in and custom regex patterns, then shared via `Arc` across all threads. - **Allowlist-aware redaction**: The `scrub` method replaces matches with `[REDACTED]` unless they contain explicitly allowed substrings. - **Type-level enforcement**: `Sanitized::new` is the only path to storage, automatically scrubbing titles and bodies while enforcing a 16 KiB size limit. - **Architectural guarantees**: The hook router at [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) line 2447 and API design ensure no observation bypasses sanitization. - **Configurable protection**: Custom patterns and allowlists let operators adapt the sanitizer to their specific security requirements without code changes. ## Frequently Asked Questions ### What credential patterns does ai-memory detect by default? According to the source code in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs), the built-in patterns cover bearer tokens, PEM private-key blocks, URL-embedded credentials, and environment variable assignments matching common secret naming conventions like `*_KEY` or `*_TOKEN`. The regex set is compiled once during `Sanitizer::new` and applied uniformly to all input strings. ### Can I add custom regex patterns to the sanitizer? Yes. The `SanitizeConfig` struct accepts an `extra_patterns` vector of strings that get compiled into regexes alongside the built-in set. Pass this configuration to `Sanitizer::new(&cfg)` at application startup to augment the default detection rules with organization-specific patterns such as internal API key formats or canary tokens. ### How does the allowlist prevent false positives? The `allowlist` parameter in `SanitizeConfig` contains substrings that the `scrub` method checks before applying redaction. If a regex match contains any allowlisted substring, the sanitizer skips replacement for that specific match. This prevents legitimate public identifiers—such as `PROJECT_TOKEN_PUBLIC`—from being masked while still catching genuinely sensitive data that follows similar patterns. ### Is it possible to bypass the sanitizer for trusted internal data? No. The architecture documentation in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) explicitly states that the HTTP and MCP APIs cannot bypass the sanitizer. The `Sanitized` type wrapper ensures that the compiler, not just runtime checks, prevents raw observations from reaching storage. Attempting to store unsanitized data would require modifying the core library to violate the type constraints defined in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs).