# What Does the sanitize.py Script Do in book-to-skill? A Security-Focused Text Sanitization Guide

> Discover what the sanitize.py script does in book-to-skill. Learn how it removes invisible Unicode characters to defend against prompt injection attacks and secure your documents.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: security-focused-guide
- Published: 2026-08-27

---

**The [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) script removes invisible Unicode characters from extracted text to prevent document-borne prompt injection attacks.**

The [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) script in the `virgiliojr94/book-to-skill` repository implements the deterministic **text-sanitization layer** that secures the pipeline against *Trojan-Source* style attacks and hidden payload injection. Located at [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), this module defines the authoritative set of invisible codepoints that must be stripped from parsed documents before the content reaches downstream language models or generated skills.

## Core Responsibilities of sanitize.py

The script operates as a defensive security barrier with three primary responsibilities: defining dangerous Unicode ranges, exposing detection predicates, and providing the public sanitization API.

### Defining Invisible Unicode Attack Vectors

At the top of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) (lines 4-100), the script declares comprehensive sets of **invisible or misleading Unicode codepoints** that can be weaponized for prompt injection:

- **`_ZERO_WIDTH_CODEPOINTS`** – zero-width spaces and non-joiners that hide text in plain sight
- **`_BIDI_CONTROL_CODEPOINTS`** – bidirectional text controls that can reorder displayed text
- **`_INVISIBLE_LETTER_CODEPOINTS`** – non-spacing marks and variation selectors
- **`_ANNOTATION_FORMAT_CODEPOINTS`** – deprecated annotation formats
- **`_TAG_BLOCK_START/END`** – language tag block delimiters
- **`_VARIATION_SELECTOR_RANGES`** – Unicode variation selectors
- **`_MUSICAL_FORMAT_RANGE`** and **`_DEPRECATED_FORMAT_RANGE`** – specialized formatting controls

These definitions establish a **single source of truth** for what constitutes a dangerous invisible character across the entire `book-to-skill` codebase.

### The is_invisible_codepoint Predicate

The script exposes a reusable predicate function to maintain consistency between extraction and validation phases:

```python
def is_invisible_codepoint(codepoint: int) -> bool

```

Implemented in lines 121-136, this function checks whether a given Unicode scalar value falls into any of the prohibited sets or ranges. Both the deterministic extractor ([`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py)) and the generated-skill scanner ([`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)) import this predicate to guarantee identical character classification.

### The sanitize_extracted_text Public API

Lines 139-151 implement the primary entry point for text cleaning:

```python
def sanitize_extracted_text(text: str) -> tuple[str, int]

```

This function iterates over each character in the input string, discarding any codepoint flagged by `is_invisible_codepoint`. It returns a **tuple containing the cleaned string and the count of removed characters**, enabling audit logs and security monitoring.

## How sanitize.py Prevents Prompt Injection

Invisible Unicode characters enable **document-borne prompt injection** attacks where malicious instructions hide within seemingly benign text. For example, zero-width spaces (`\u200b`) can break tokenization boundaries, while bidirectional overrides (`\u202e`) can visually reorder text to fool human reviewers while preserving the original instruction sequence for language models.

By stripping these characters immediately after parsing (in `book_to_skill/parsers/*`) and before skill generation, [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) ensures that the final extracted prose contains no hidden semantics that could alter model-level interpretation.

## Integration with the book-to-skill Pipeline

The sanitization layer sits between document parsing and skill generation:

1. **Extraction Phase**: [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) calls `sanitize_extracted_text` after PDF, EPUB, or HTML parsing completes
2. **Validation Phase**: [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) re-uses `is_invisible_codepoint` to verify that generated skills contain no injected invisible characters
3. **Testing**: [`tests/test_sanitize_extracted_text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_extracted_text.py) confirms that all defined invisible categories are correctly removed

This architecture prevents drift between extraction and scanning by enforcing the same invisible-character definitions across both operations.

## Practical Code Examples

### Cleaning Extracted Book Text

```python
from book_to_skill.sanitize import sanitize_extracted_text

# Text containing ZERO WIDTH SPACE (\u200b), ZERO WIDTH NON-JOINER (\u200c),

# and RIGHT-TO-LEFT OVERRIDE (\u202e)

raw = "Hello\u200b\u200c World\u202e!"

clean, removed = sanitize_extracted_text(raw)

print(clean)      # → "Hello World!"

print(removed)    # → 3

```

### Direct Predicate Usage

```python
from book_to_skill.sanitize import is_invisible_codepoint

print(is_invisible_codepoint(ord('\u200b')))  # → True

print(is_invisible_codepoint(ord('A')))       # → False

```

## Summary

- **[`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py)** serves as the security-focused text sanitization layer for the `book-to-skill` repository, located at [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)
- The script defines eight categories of dangerous invisible Unicode codepoints (lines 4-100) including zero-width characters, bidirectional controls, and deprecated annotation formats
- **`is_invisible_codepoint`** provides a reusable predicate used by both the extractor and the skill scanner to maintain consistent security policies
- **`sanitize_extracted_text`** returns a cleaned string and removal count, invoked after document parsing to harden content against *Trojan-Source* attacks
- The module is consumed by [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) and [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py), ensuring invisible-character logic remains synchronized across the pipeline

## Frequently Asked Questions

### What types of invisible characters does sanitize.py remove?

The script targets eight specific categories: zero-width codepoints, bidirectional controls, invisible letter codepoints, annotation format characters, tag block delimiters, variation selectors, musical format controls, and deprecated format ranges. These definitions cover the full spectrum of Unicode characters capable of altering text interpretation while remaining invisible to human readers.

### How does sanitize.py integrate with the book-to-skill extractor?

After [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) parses a document (PDF, EPUB, or HTML), it passes the raw extracted text to `sanitize_extracted_text` before any downstream processing occurs. This ensures that invisible payloads are eliminated immediately after parsing but before the content reaches the skill generation logic.

### What does the sanitize_extracted_text function return?

According to the source code in lines 139-151, `sanitize_extracted_text` returns a tuple containing two elements: the sanitized string with all invisible characters removed, and an integer representing the count of characters that were stripped during the operation.

### Why is Unicode sanitization important for AI-generated skills?

Invisible Unicode characters enable prompt injection attacks where malicious instructions hide within extracted text. Without sanitization, these characters could alter how language models interpret the content while remaining invisible to human reviewers, potentially causing the generated skills to execute unintended behaviors or leak sensitive information.