What Does the sanitize.py Script Do in book-to-skill? A Security-Focused Text Sanitization Guide

The sanitize.py script removes invisible Unicode characters from extracted text to prevent document-borne prompt injection attacks.

The sanitize.py script in the virgiliojr94/book-to-skill repository implements the deterministic text-sanitization layer that secures the pipeline against Trojan-Source style attacks and hidden payload injection. Located at book_to_skill/sanitize.py, this module defines the authoritative set of invisible codepoints that must be stripped from parsed documents before the content reaches downstream language models or generated skills.

Core Responsibilities of sanitize.py

The script operates as a defensive security barrier with three primary responsibilities: defining dangerous Unicode ranges, exposing detection predicates, and providing the public sanitization API.

Defining Invisible Unicode Attack Vectors

At the top of book_to_skill/sanitize.py (lines 4-100), the script declares comprehensive sets of invisible or misleading Unicode codepoints that can be weaponized for prompt injection:

• _ZERO_WIDTH_CODEPOINTS – zero-width spaces and non-joiners that hide text in plain sight
• _BIDI_CONTROL_CODEPOINTS – bidirectional text controls that can reorder displayed text
• _INVISIBLE_LETTER_CODEPOINTS – non-spacing marks and variation selectors
• _ANNOTATION_FORMAT_CODEPOINTS – deprecated annotation formats
• _TAG_BLOCK_START/END – language tag block delimiters
• _VARIATION_SELECTOR_RANGES – Unicode variation selectors
• _MUSICAL_FORMAT_RANGE and _DEPRECATED_FORMAT_RANGE – specialized formatting controls

These definitions establish a single source of truth for what constitutes a dangerous invisible character across the entire book-to-skill codebase.

The is_invisible_codepoint Predicate

The script exposes a reusable predicate function to maintain consistency between extraction and validation phases:

def is_invisible_codepoint(codepoint: int) -> bool

Implemented in lines 121-136, this function checks whether a given Unicode scalar value falls into any of the prohibited sets or ranges. Both the deterministic extractor (scripts/extract.py) and the generated-skill scanner (tools/scan_generated_skill.py) import this predicate to guarantee identical character classification.

The sanitize_extracted_text Public API

Lines 139-151 implement the primary entry point for text cleaning:

def sanitize_extracted_text(text: str) -> tuple[str, int]

This function iterates over each character in the input string, discarding any codepoint flagged by is_invisible_codepoint. It returns a tuple containing the cleaned string and the count of removed characters, enabling audit logs and security monitoring.

How sanitize.py Prevents Prompt Injection

Invisible Unicode characters enable document-borne prompt injection attacks where malicious instructions hide within seemingly benign text. For example, zero-width spaces (\u200b) can break tokenization boundaries, while bidirectional overrides (\u202e) can visually reorder text to fool human reviewers while preserving the original instruction sequence for language models.

By stripping these characters immediately after parsing (in book_to_skill/parsers/*) and before skill generation, sanitize.py ensures that the final extracted prose contains no hidden semantics that could alter model-level interpretation.

Integration with the book-to-skill Pipeline

The sanitization layer sits between document parsing and skill generation:

1. Extraction Phase: scripts/extract.py calls sanitize_extracted_text after PDF, EPUB, or HTML parsing completes
2. Validation Phase: tools/scan_generated_skill.py re-uses is_invisible_codepoint to verify that generated skills contain no injected invisible characters
3. Testing: tests/test_sanitize_extracted_text.py confirms that all defined invisible categories are correctly removed

This architecture prevents drift between extraction and scanning by enforcing the same invisible-character definitions across both operations.

Practical Code Examples

Cleaning Extracted Book Text

from book_to_skill.sanitize import sanitize_extracted_text

# Text containing ZERO WIDTH SPACE (\u200b), ZERO WIDTH NON-JOINER (\u200c),

# and RIGHT-TO-LEFT OVERRIDE (\u202e)

raw = "Hello\u200b\u200c World\u202e!"

clean, removed = sanitize_extracted_text(raw)

print(clean)      # → "Hello World!"

print(removed)    # → 3

Direct Predicate Usage

from book_to_skill.sanitize import is_invisible_codepoint

print(is_invisible_codepoint(ord('\u200b')))  # → True

print(is_invisible_codepoint(ord('A')))       # → False

Summary

• sanitize.py serves as the security-focused text sanitization layer for the book-to-skill repository, located at book_to_skill/sanitize.py
• The script defines eight categories of dangerous invisible Unicode codepoints (lines 4-100) including zero-width characters, bidirectional controls, and deprecated annotation formats
• is_invisible_codepoint provides a reusable predicate used by both the extractor and the skill scanner to maintain consistent security policies
• sanitize_extracted_text returns a cleaned string and removal count, invoked after document parsing to harden content against Trojan-Source attacks
• The module is consumed by scripts/extract.py and tools/scan_generated_skill.py, ensuring invisible-character logic remains synchronized across the pipeline

Frequently Asked Questions

What types of invisible characters does sanitize.py remove?

The script targets eight specific categories: zero-width codepoints, bidirectional controls, invisible letter codepoints, annotation format characters, tag block delimiters, variation selectors, musical format controls, and deprecated format ranges. These definitions cover the full spectrum of Unicode characters capable of altering text interpretation while remaining invisible to human readers.

How does sanitize.py integrate with the book-to-skill extractor?

After scripts/extract.py parses a document (PDF, EPUB, or HTML), it passes the raw extracted text to sanitize_extracted_text before any downstream processing occurs. This ensures that invisible payloads are eliminated immediately after parsing but before the content reaches the skill generation logic.

What does the sanitize_extracted_text function return?

According to the source code in lines 139-151, sanitize_extracted_text returns a tuple containing two elements: the sanitized string with all invisible characters removed, and an integer representing the count of characters that were stripped during the operation.

Why is Unicode sanitization important for AI-generated skills?

Invisible Unicode characters enable prompt injection attacks where malicious instructions hide within extracted text. Without sanitization, these characters could alter how language models interpret the content while remaining invisible to human reviewers, potentially causing the generated skills to execute unintended behaviors or leak sensitive information.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}