How ai-memory's Sanitize Boundary Prevents Secrets from Entering the Store

ai-memory enforces a compile-time guarantee that no raw text can be persisted to storage by requiring all writes to pass through a Sanitized<T> wrapper that scrubs credentials using configurable regex patterns.

The akitaonrails/ai-memory codebase implements a defense-in-depth approach to secret management. Rather than relying on post-hoc scanning or manual review, the system embeds privacy controls directly into the type system and data lifecycle. This article explains how the sanitize boundary works, where it is enforced, and how you can configure it for your deployment.

Core Components of the Sanitize Boundary

The protection layer consists of two tightly-coupled mechanisms: a stateful Sanitizer that performs pattern-based redaction, and a generic Sanitized<T> wrapper that makes the boundary unbypassable.

The Sanitizer State Machine

Defined in [crates/ai-memory-core/src/sanitize.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs), the Sanitizer struct compiles regex patterns at initialization and applies them uniformly across the application.

Built-in detection patterns cover common secret formats:

• Bearer tokens and API key prefixes (sk-, Bearer , etc.)
• PEM-encoded cryptographic material
• URL-embedded credentials (user:pass@host)
• Private key file paths and environment variable assignments

These patterns are compiled once during Sanitizer::new() or Sanitizer::builtin() and shared via Arc, ensuring consistent behavior without runtime recompilation.

Configuration options extend or relax the default rules:

// From sanitize.rs lines 55-62
pub struct SanitizeConfig {
    /// Additional regex patterns to match beyond built-ins
    pub extra_patterns: Vec<String>,
    /// Substrings that must never be redacted even if they match patterns
    pub allowlist: Vec<String>,
}

The core scrubbing logic in scrub() (lines 199-210) iterates all compiled patterns, replacing matches with the literal string [REDACTED] unless the match contains an allowlist entry:

// Conceptual implementation
pub fn scrub(&self, input: &str) -> String {
    let mut output = input.to_string();
    for regex in &self.compiled {
        output = regex.replace_all(&output, "[REDACTED]").to_string();
    }
    output
}

The Sanitized<T> Compile-Time Boundary

The Sanitized<T> wrapper provides the enforcement mechanism. Located in the same [sanitize.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs) file (lines 42-50), this type has only one valid constructor:

// Lines 42-44: Safe constructor is the only way to build this type
impl<T> Sanitized<T> {
    pub fn new(value: T, sanitizer: &Sanitizer) -> Self
    where
        T: Sanitize, // Requires implementors to define scrubbing behavior
    { /* ... */ }
}

For observations, the implementation impl Sanitized<NewObservation> performs three operations:

1. Scrubs the title field using sanitizer.scrub()
2. Scrubs the body field using sanitizer.scrub()
3. Enforces OBSERVATION_BODY_MAX_BYTES = 16 KiB to bound storage consumption

The result: the compiler rejects any attempt to persist unsanitized data. The store writer in [crates/ai-memory-store/src/writer.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs) (lines 660-680) accepts only Sanitized<NewObservation> values, making the boundary architectural rather than advisory.

Where the Boundary Is Enforced

The sanitize boundary applies at every text ingress point across the codebase.

Lifecycle Hooks Entry Point

In [crates/ai-memory-hooks/src/router.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) (lines 2490-2505), incoming JSON payloads from external systems are immediately wrapped:

// Router converts HTTP payload to internal representation
let raw_observation: NewObservation = parse_json(body)?;
let sanitized = Sanitized::new(raw_observation, &self.sanitizer);
// Only `sanitized` can be passed to the store
store.insert_observation(sanitized)?;

This ensures that malicious or accidentally-leaked credentials in hook payloads never reach durable storage.

Wiki Content Processing

The wiki subsystem in [crates/ai-memory-wiki/src/wiki.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs) applies sanitization at multiple points:

• Lines 166-180: Page titles and bodies are scrubbed before markdown file creation
• Lines 1107-1172: Front-matter metadata undergoes the same processing

This protects against credential leakage through documentation that might contain copied terminal output or configuration examples.

MCP API Hand-Off

The Model Context Protocol server in [crates/ai-memory-mcp/src/server.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 761-805) sanitizes:

• Client-provided observation names
• Feedback reason strings
• Custom metadata fields

This prevents LLM-generated content or external tool outputs from contaminating the observation store.

Practical Configuration and Usage

Building a Custom Sanitizer

use ai_memory_core::{Sanitizer, SanitizeConfig, Sanitized, NewObservation};

// Step 1: Define custom patterns and allowlist exceptions
let cfg = SanitizeConfig {
    extra_patterns: vec![
        r"CANARY-\d{6}".to_string(),      // Internal canary tokens
        r"proj_[a-z0-9]{32}".to_string(), // Project-specific API format
    ],
    allowlist: vec![
        "PROJECT_TOKEN_PUBLIC".to_string(), // Safe to log this literal
    ],
};

// Step 2: Compile sanitizer (fails fast on invalid regex)
let sanitizer = Sanitizer::new(&cfg).expect("valid regex patterns");

// Step 3: Create observation with potential secrets
let raw = NewObservation {
    session_id: SessionId::new(),
    workspace_id: WorkspaceId::new(),
    project_id: ProjectId::new(),
    kind: ObservationKind::UserPrompt,
    extension: None,
    source_event: None,
    title: "OPENAI_API_KEY=sk-secret123abc".into(),
    body: "Error connecting to db: postgresql://admin:hunter2@prod.internal/ai_db".into(),
    importance: 5,
};

// Step 4: Apply boundary - this is required, not optional
let safe = Sanitized::new(raw, &sanitizer);

// Step 5: Extract and persist (only possible after sanitization)
let inner: NewObservation = safe.into_inner();
writer.insert_observation(inner).await?;

Result Inspection

After processing, sensitive values are replaced:

Field	Original	After Scrub()
title	OPENAI_API_KEY=sk-secret123abc	OPENAI_API_KEY=[REDACTED]
body	postgresql://admin:hunter2@...	postgresql://[REDACTED]@...

The [REDACTED] literal is chosen specifically to be:

• Human-readable in logs and debugging
• Unlikely to collide with legitimate data
• Consistent across all redaction sites for searchability

Architecture Benefits

Type system enforcement eliminates entire categories of bugs. Developers cannot accidentally call writer.insert_observation(raw_value) because the method signature requires Sanitized<NewObservation>.

Global consistency emerges from the Arc<Sanitizer> sharing pattern. All components—hooks, wiki, MCP—use identical regex compilation and matching logic.

Configurable without recompilation via the [sanitize] section in config.toml. Operators can tune detection without modifying source code or rebuilding binaries.

Summary

• Sanitizer compiles regex patterns at startup and provides the scrub() method for credential redaction
• Sanitized<T> creates a compile-time boundary that prevents unsanitized data from reaching storage APIs
• The boundary applies at every ingress point: lifecycle hooks, wiki writes, and MCP endpoints
• Configuration supports custom patterns via extra_patterns and safe-words via allowlist without code changes
• The store writer in ai-memory-store/src/writer.rs only accepts sanitized observations, making enforcement automatic

Frequently Asked Questions

What secret patterns does ai-memory detect by default?

The built-in patterns cover Bearer tokens, OpenAI/API key prefixes (sk-, pk-), PEM blocks, URL credentials, private key paths, and common environment variable assignments. The full list is defined in [sanitize.rs lines 48-78](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs#L48-L78).

Can I add custom patterns without modifying the source code?

Yes. Add an [sanitize] section to your config.toml with extra_patterns containing valid regex strings. The sanitizer will compile these alongside built-ins at startup. Invalid patterns cause immediate failure with a descriptive error.

What happens if legitimate data accidentally matches a secret pattern?

Use the allowlist configuration option to specify substrings that must never be redacted. For example, allowlist = ["PUBLIC_TOKEN_EXAMPLE"] preserves that literal even if it matches a regex. Apply this sparingly and document each exception.

Is there any way to bypass the sanitize boundary?

No—not without modifying the source code to change writer.rs method signatures. The Sanitized<T> type has no public constructors other than ::new(), and that constructor requires a Sanitizer reference. The Rust compiler enforces this guarantee.

Does sanitization affect performance?

The Sanitizer is built once and shared via Arc, so regex compilation cost is paid at startup. The scrub() method runs linearly over input text against compiled patterns. For typical observation sizes (under 16 KiB), overhead is negligible compared to SQLite I/O.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}