How to Implement Log Sanitization for Privacy: A Rule-Based Approach for AI Agents

Log sanitization for privacy removes personally identifiable information (PII) and secrets from application logs using a rule-based regex engine that operates offline without requiring machine learning models.

The bojieli/ai-agent-book repository demonstrates a production-ready implementation of log sanitization for privacy that prioritizes accuracy and speed. This Python-based system uses carefully ordered regular expressions and optional validator functions to redact sensitive data before logs are persisted or transmitted.

Architecture of the Rule-Based Sanitization Engine

The sanitization engine in chapter2/log-sanitization/regex_sanitizer.py follows a deterministic, priority-driven architecture. Unlike model-based approaches, this implementation relies entirely on pattern matching and algorithmic validation to ensure zero external dependencies and predictable performance.

Rule Definitions and Priority Ordering

The engine centers on the _RULES list defined in regex_sanitizer.py (starting at line 58). Each rule is a dictionary containing:

• Category: A human-readable label (e.g., private_key, credit_card) used for reporting and classification.
• Placeholder: The replacement token (e.g., [REDACTED_PRIVATE_KEY]) that substitutes the sensitive value.
• Compiled Regex: A pre-compiled pattern that identifies the secret type.
• Group Specification: Which capture group to redact (0 indicates the entire match).
• Validator (Optional): A callable function that performs secondary verification to reduce false positives.

Rules are ordered by priority within _RULES, ensuring that high-risk patterns (like API keys) are evaluated before lower-risk generic patterns.

Validation Functions for Accuracy

To minimize false positives, the engine implements specialized validators:

• _luhn_ok (lines 31-44): Validates credit card numbers using the Luhn algorithm checksum.
• _cn_id_ok (lines 47-55): Validates Chinese mainland ID numbers using the official government checksum formula.

These validators act as gatekeepers; matches only proceed to redaction if the validator returns True.

Core Implementation in regex_sanitizer.py

The sanitize() function (lines 86-99) serves as the primary entry point for log sanitization for privacy. It implements a multi-stage pipeline:

1. Iterates through _RULES in priority order.
2. Applies each regex pattern globally to the input text.
3. Executes optional validators on matches.
4. Replaces matched substrings with designated placeholders.
5. Records metadata including category, value, placeholder, start, and end positions.

from regex_sanitizer import sanitize

raw_log = """
API key: sk-abc123def456ghi789jkl
Credit card: 4111 1111 1111 1111
Authorization: Bearer abcdef123456
"""

sanitized, findings = sanitize(raw_log)
print(sanitized)

# Output:

# API key: [REDACTED_API_KEY]

# Credit card: [REDACTED_CREDIT_CARD]

# Authorization: Bearer [REDACTED_BEARER_TOKEN]

print(f"Found {len(findings)} secrets")

The function returns a tuple containing the sanitized text and a list of dictionaries detailing each redaction.

High-Level API Integration

For AI agent workflows, the Agent class in agent.py provides convenience methods that wrap the core sanitizer. The sanitize_conversation() method (lines 246-274) processes entire dialogue histories:

from agent import Agent

conv = [
    {"role": "user", "content": "My password is hunter2"},
    {"role": "assistant", "content": "Got it!"},
]

agent = Agent()
result = agent.sanitize_conversation(conv, test_id="demo")
print(result["sanitized_text"])

# → "My password is [REDACTED_SECRET]"

print(result["metrics"])

# → {'sanitized_length': 42, 'replacements': 1, ...}

This integration automatically applies sanitize_text() to each message while collecting metrics such as redaction counts and length deltas.

Command-Line Batch Processing

The main.py script (lines 102-131) exposes the sanitization engine as a CLI tool for batch operations:

python -m chapter2.log-sanitization.main \
    --input logs/app.log \
    --output logs/app.sanitized.log \
    --mode offline

The CLI reads input files, invokes the sanitize() function, writes sanitized copies, and generates summary reports showing redaction statistics by category.

Summary

• Log sanitization for privacy in the ai-agent-book repository uses a deterministic, rule-based engine located in regex_sanitizer.py.
• The _RULES list defines detection patterns with priority ordering, while _luhn_ok and _cn_id_ok validators reduce false positives.
• The sanitize() function performs the core redaction logic, returning both cleaned text and detailed metadata.
• The Agent class provides high-level conversation sanitization with built-in metrics tracking.
• Batch processing is available via the CLI interface in main.py for enterprise log management pipelines.

Frequently Asked Questions

How does the rule-based approach compare to machine learning for log sanitization?

The rule-based approach in bojieli/ai-agent-book offers deterministic performance and zero external dependencies, making it ideal for offline environments. Unlike ML models, it requires no training data or GPU resources, though it may need periodic updates to regex patterns when new secret formats emerge.

What types of sensitive data can the engine detect?

According to the source code in regex_sanitizer.py, the engine detects API keys, private keys, credit card numbers, bearer tokens, Chinese ID numbers, and various credential patterns in URLs. Each category uses specific regex patterns with optional checksum validators to ensure accuracy.

How can I add custom sanitization rules?

You can extend the _RULES list in regex_sanitizer.py by appending dictionaries containing your regex pattern, placeholder text, category label, and optional validator function. Ensure higher-risk patterns appear earlier in the list to prevent partial masking by more general rules.

Is the sanitization process reversible?

No, the process is intentionally destructive for security. As implemented in the sanitize() function, original values are replaced with placeholders like [REDACTED_API_KEY] without maintaining a mapping table, ensuring that sanitized logs contain no recoverable sensitive data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}