# How to Implement Log Sanitization for Privacy: A Rule-Based Approach for AI Agents

> Implement log sanitization for privacy by removing PII and secrets with a rule-based regex engine. Learn this offline, ML-free approach for AI agents.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-26

---

**Log sanitization for privacy removes personally identifiable information (PII) and secrets from application logs using a rule-based regex engine that operates offline without requiring machine learning models.**

The `bojieli/ai-agent-book` repository demonstrates a production-ready implementation of log sanitization for privacy that prioritizes accuracy and speed. This Python-based system uses carefully ordered regular expressions and optional validator functions to redact sensitive data before logs are persisted or transmitted.

## Architecture of the Rule-Based Sanitization Engine

The sanitization engine in [`chapter2/log-sanitization/regex_sanitizer.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/log-sanitization/regex_sanitizer.py) follows a deterministic, priority-driven architecture. Unlike model-based approaches, this implementation relies entirely on pattern matching and algorithmic validation to ensure zero external dependencies and predictable performance.

### Rule Definitions and Priority Ordering

The engine centers on the **`_RULES`** list defined in [`regex_sanitizer.py`](https://github.com/bojieli/ai-agent-book/blob/main/regex_sanitizer.py) (starting at line 58). Each rule is a dictionary containing:

- **Category**: A human-readable label (e.g., `private_key`, `credit_card`) used for reporting and classification.
- **Placeholder**: The replacement token (e.g., `[REDACTED_PRIVATE_KEY]`) that substitutes the sensitive value.
- **Compiled Regex**: A pre-compiled pattern that identifies the secret type.
- **Group Specification**: Which capture group to redact (0 indicates the entire match).
- **Validator (Optional)**: A callable function that performs secondary verification to reduce false positives.

Rules are ordered by priority within `_RULES`, ensuring that high-risk patterns (like API keys) are evaluated before lower-risk generic patterns.

### Validation Functions for Accuracy

To minimize false positives, the engine implements specialized validators:

- **`_luhn_ok`** (lines 31-44): Validates credit card numbers using the Luhn algorithm checksum.
- **`_cn_id_ok`** (lines 47-55): Validates Chinese mainland ID numbers using the official government checksum formula.

These validators act as gatekeepers; matches only proceed to redaction if the validator returns `True`.

## Core Implementation in regex_sanitizer.py

The **`sanitize()`** function (lines 86-99) serves as the primary entry point for log sanitization for privacy. It implements a multi-stage pipeline:

1. Iterates through `_RULES` in priority order.
2. Applies each regex pattern globally to the input text.
3. Executes optional validators on matches.
4. Replaces matched substrings with designated placeholders.
5. Records metadata including `category`, `value`, `placeholder`, `start`, and `end` positions.

```python
from regex_sanitizer import sanitize

raw_log = """
API key: sk-abc123def456ghi789jkl
Credit card: 4111 1111 1111 1111
Authorization: Bearer abcdef123456
"""

sanitized, findings = sanitize(raw_log)
print(sanitized)

# Output:

# API key: [REDACTED_API_KEY]

# Credit card: [REDACTED_CREDIT_CARD]

# Authorization: Bearer [REDACTED_BEARER_TOKEN]

print(f"Found {len(findings)} secrets")

```

The function returns a tuple containing the sanitized text and a list of dictionaries detailing each redaction.

## High-Level API Integration

For AI agent workflows, the **`Agent`** class in [`agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agent.py) provides convenience methods that wrap the core sanitizer. The **`sanitize_conversation()`** method (lines 246-274) processes entire dialogue histories:

```python
from agent import Agent

conv = [
    {"role": "user", "content": "My password is hunter2"},
    {"role": "assistant", "content": "Got it!"},
]

agent = Agent()
result = agent.sanitize_conversation(conv, test_id="demo")
print(result["sanitized_text"])

# → "My password is [REDACTED_SECRET]"

print(result["metrics"])

# → {'sanitized_length': 42, 'replacements': 1, ...}

```

This integration automatically applies **`sanitize_text()`** to each message while collecting metrics such as redaction counts and length deltas.

## Command-Line Batch Processing

The [`main.py`](https://github.com/bojieli/ai-agent-book/blob/main/main.py) script (lines 102-131) exposes the sanitization engine as a CLI tool for batch operations:

```bash
python -m chapter2.log-sanitization.main \
    --input logs/app.log \
    --output logs/app.sanitized.log \
    --mode offline

```

The CLI reads input files, invokes the `sanitize()` function, writes sanitized copies, and generates summary reports showing redaction statistics by category.

## Summary

- **Log sanitization for privacy** in the ai-agent-book repository uses a deterministic, rule-based engine located in [`regex_sanitizer.py`](https://github.com/bojieli/ai-agent-book/blob/main/regex_sanitizer.py).
- The **`_RULES`** list defines detection patterns with priority ordering, while **`_luhn_ok`** and **`_cn_id_ok`** validators reduce false positives.
- The **`sanitize()`** function performs the core redaction logic, returning both cleaned text and detailed metadata.
- The **`Agent`** class provides high-level conversation sanitization with built-in metrics tracking.
- Batch processing is available via the CLI interface in [`main.py`](https://github.com/bojieli/ai-agent-book/blob/main/main.py) for enterprise log management pipelines.

## Frequently Asked Questions

### How does the rule-based approach compare to machine learning for log sanitization?

The rule-based approach in `bojieli/ai-agent-book` offers deterministic performance and zero external dependencies, making it ideal for offline environments. Unlike ML models, it requires no training data or GPU resources, though it may need periodic updates to regex patterns when new secret formats emerge.

### What types of sensitive data can the engine detect?

According to the source code in [`regex_sanitizer.py`](https://github.com/bojieli/ai-agent-book/blob/main/regex_sanitizer.py), the engine detects API keys, private keys, credit card numbers, bearer tokens, Chinese ID numbers, and various credential patterns in URLs. Each category uses specific regex patterns with optional checksum validators to ensure accuracy.

### How can I add custom sanitization rules?

You can extend the `_RULES` list in [`regex_sanitizer.py`](https://github.com/bojieli/ai-agent-book/blob/main/regex_sanitizer.py) by appending dictionaries containing your regex pattern, placeholder text, category label, and optional validator function. Ensure higher-risk patterns appear earlier in the list to prevent partial masking by more general rules.

### Is the sanitization process reversible?

No, the process is intentionally destructive for security. As implemented in the `sanitize()` function, original values are replaced with placeholders like `[REDACTED_API_KEY]` without maintaining a mapping table, ensuring that sanitized logs contain no recoverable sensitive data.