# How AI Agent Session Observations Are Sanitized Before Storage in ai-memory

> Discover how ai-memory sanitizes AI agent session observations using a multi-layer pipeline. Learn about regex compilation, credential scrubbing, type safety, and size limits before data storage.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-31

---

**AI agent session observations are sanitized through a multi-layer pipeline that compiles regex patterns at startup, scrubs credentials using the `Sanitizer::scrub` method, enforces type safety via the `Sanitized<T>` wrapper, and applies a hard 16 KiB size limit before persisting to SQLite.**

The **ai-memory** repository treats all incoming agent data as untrusted. Before any observation reaches the durable store, it must pass through a strict sanitization contract implemented in Rust. This architecture prevents credential leaks while maintaining observability by redacting sensitive patterns and respecting explicit allowlists.

## The Sanitizer Architecture

The core of the privacy pipeline is the **`Sanitizer`** type defined in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs) at lines 132-134. This stateful scrubber is designed for concurrent access using an `Arc<SanitizerInner>` backing, allowing cheap cloning across async boundaries without recompiling regex patterns.

**Key design characteristics:**

- **Singleton pattern:** Built once at startup and shared across the application lifecycle
- **Thread-safe:** Uses `Arc` internally for lock-free read access during scrubbing operations
- **Fail-fast:** Invalid user-supplied patterns abort initialization, guaranteeing the sanitizer is always functional

## Built-in Redaction Patterns

The sanitizer ships with comprehensive credential detection via **`BUILTIN_PATTERN_STRS`** (lines 48-127 of [`sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/sanitize.rs)). These patterns cover:

- Bearer tokens and API keys (OpenAI, Anthropic, AWS)
- GitHub personal access tokens
- Private key blocks (RSA, EC, OpenSSH)
- URLs containing embedded credentials
- Generic high-entropy secrets

These regexes are compiled during `Sanitizer::new` and stored in the `Arc`-wrapped inner structure for efficient matching.

## Configuration and Construction

Operator customization happens through `SanitizeConfig`, allowing `extra_patterns` and `allowlist` entries to augment or override built-in behavior. The constructor implementation (lines 165-187) validates all patterns at startup:

```rust
// Build a sanitizer at start-up using config from config.toml
let cfg = SanitizeConfig {
    extra_patterns: vec![r"my_secret_\w+".to_string()],
    allowlist: vec!["PROJECT_TOKEN".to_string()],
};
let sanitizer = Sanitizer::new(&cfg).expect("invalid pattern");

```

Any compilation error in user-provided regexes triggers an immediate panic during initialization, preventing runtime failures during scrubbing.

## The Scrubbing Logic

The `Sanitizer::scrub` method (lines 196-200) implements the actual redaction logic:

1. Walks the input string sequentially
2. Replaces every pattern match with the literal **`[REDACTED]`**
3. Checks the `allowlist` before redacting; if the match contains an allowed term, the original text is preserved

```rust
// Scrub a raw observation string
let raw = "User entered OpenAI key: sk-abcdef1234567890";
let clean = sanitizer.scrub(raw);
// clean == "User entered OpenAI key: [REDACTED]"

```

This ensures that sensitive agent outputs are stripped while legitimate diagnostic tokens can pass through when explicitly allowed.

## Type-Safe Enforcement with Sanitized<T>

To guarantee that only scrubbed data reaches the persistence layer, **ai-memory** uses the **`Sanitized<T>`** wrapper (lines 228-229). This is a zero-cost abstraction that encodes the "sanitize-then-store" invariant into the type system.

The only way to create a persisted observation is via `Sanitized::new`, which requires:

```rust
// Wrap the cleaned observation so it can be stored
let obs = NewObservation::new(clean);
let sanitized_obs = Sanitized::new(obs, &sanitizer);
store.write_observation(sanitized_obs);

```

The store writer (in [`crates/ai-memory-store/src/writer.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/writer.rs)) accepts only `Sanitized` types, making it impossible to accidentally persist raw, unsanitized agent output.

## Integration in the Hook Pipeline

The enforcement boundary appears in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) around lines 2490-2505. When a lifecycle hook payload arrives at the server:

1. The router extracts the raw observation
2. Builds a `Sanitized<NewObservation>` by calling `Sanitized::new(raw_obs, &state.sanitizer)`
3. Passes the wrapped object to the store writer

This single chokepoint ensures **every** stored observation has passed through the sanitizer, regardless of the agent source or hook type.

## Size Limits and Payload Validation

After content sanitization but before SQLite insertion, observations are bounded by **`OBSERVATION_BODY_MAX_BYTES`** (16 KiB), defined at lines 43-46 of [`sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/sanitize.rs). Any payload exceeding this limit is rejected before reaching the writer, preventing denial-of-service attacks through oversized memory dumps or log floods.

## Summary

- **Untrusted by default:** All agent observations are treated as potentially containing secrets
- **Pattern-driven:** Built-in regexes detect credentials while supporting custom `extra_patterns`
- **Allowlist-aware:** Explicit permits prevent over-redaction of necessary diagnostic tokens
- **Type-safe:** The `Sanitized<T>` wrapper makes unsafe storage unrepresentable in the codebase
- **Size-guarded:** Hard 16 KiB limit prevents store abuse

## Frequently Asked Questions

### How does ai-memory prevent API keys from leaking into stored observations?

According to the ai-memory source code, the `Sanitizer::scrub` method replaces any substring matching built-in credential patterns (like `sk-` prefixed OpenAI keys or AWS secrets) with the literal `[REDACTED]`. This happens automatically for all observations before they reach the SQLite store.

### Can I customize which patterns get redacted in ai-memory?

Yes. The `SanitizeConfig` struct accepts `extra_patterns` (a vector of custom regex strings) and `allowlist` entries (strings that exempt matches from redaction). These are compiled alongside `BUILTIN_PATTERN_STRS` during `Sanitizer::new` initialization in [`crates/ai-memory-core/src/sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/sanitize.rs).

### What happens if an agent sends an observation larger than 16 KiB?

Observations exceeding `OBSERVATION_BODY_MAX_BYTES` (16 KiB) are rejected by the size guard before sanitization completes. This prevents oversized payloads from entering the SQLite database and protects against storage exhaustion attacks.

### Is it possible to accidentally store unsanitized data in ai-memory?

No. The `Sanitized<T>` wrapper in [`sanitize.rs`](https://github.com/akitaonrails/ai-memory/blob/main/sanitize.rs) provides a type-safe boundary. The store writer only accepts `Sanitized` variants, and the hook router in [`crates/ai-memory-hooks/src/router.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-hooks/src/router.rs) wraps all incoming observations before passing them to persistence. This makes unsanitized storage a compile-time error.