# What Is the Purpose of the sanitize.py Script in book-to-skill?

> Learn how the sanitize.py script in book-to-skill removes invisible Unicode characters for clean, deterministic text processing and ensures reliable input for your projects.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-30

---

**The sanitize.py script in book-to-speak removes invisible Unicode codepoints—such as zero-width spaces and bidirectional controls—from raw text extracts to guarantee clean, deterministic input for downstream processing.**

The `book-to-skill` repository ingests documents in formats like PDF, RTF, HTML, and EPUB to build skill mappings. Located at [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), this module acts as a defensive filter that eliminates hidden characters before they can distort string length, break tokenization, or introduce security vulnerabilities.

## Core Sanitization Functions

The module exposes two public utilities designed to handle Unicode sanitization at different levels of granularity.

### `is_invisible_codepoint()`

This function detects characters that render invisibly or alter text direction without visual representation. According to the source code in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), it identifies **zero-width spaces (U+200B)**, **BIDI controls such as U+202A**, and various annotation controls. These codepoints can alter the logical order of strings or hide malicious payloads, making their detection critical for safe text processing.

### `sanitize_extracted_text()`

The primary entry point for the module, `sanitize_extracted_text(text: str) -> Tuple[str, int]`, iterates through input strings and strips every invisible codepoint identified by `is_invisible_codepoint()`. It returns a tuple containing the cleaned string and an integer count of removed characters. This ensures that metrics and skill extractions calculated later in the pipeline reflect only visible, user-intended content.

## Where sanitize.py Is Used in the Codebase

The sanitization logic is invoked at critical boundaries within the `virgiliojr94/book-to-skill` repository to prevent corruption of processed data.

- **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** – After parsing a document (for example, extracting text from a PDF), the code calls `sanitize_extracted_text` at line 941 to cleanse the content before storage or analysis.
- **[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** – This utility script imports `is_invisible_codepoint` to audit generated skill artifacts, flagging any stray invisible characters at line 25 before they reach production outputs.

## Usage Examples

The following patterns demonstrate how to integrate the sanitization utilities into custom workflows:

```python
from book_to_skill.sanitize import sanitize_extracted_text

raw = "Hello\u200BWorld\u202E!"   # contains zero-width space and a BIDI control

clean, removed = sanitize_extracted_text(raw)
print(clean)    # → "HelloWorld!"

print(removed)  # → 2

```

To check individual characters manually:

```python
from book_to_skill.sanitize import is_invisible_codepoint

char = "\u202B"  # Right-to-Left Embedding

print(is_invisible_codepoint(char))   # → True

```

## Test Coverage

The repository maintains rigorous test suites to validate sanitization behavior across Unicode categories:

- **[`tests/test_sanitize_extracted_text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_extracted_text.py)** – Confirms that zero-width characters and tag-block codepoints are removed while preserving multilingual text.
- **[`tests/test_sanitize_bidi_controls.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_bidi_controls.py)** – Verifies that bidirectional controls such as U+202E and U+200F are stripped and that the removal count is accurate.
- **[`tests/test_sanitize_annotation_controls.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_annotation_controls.py)** – Ensures that annotation controls in the U+FFF9 to U+FFFC range are eliminated from processed strings.

## Summary

- **[`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py)** provides the `is_invisible_codepoint()` and `sanitize_extracted_text()` utilities to cleanse raw document extracts.
- It targets **zero-width spaces**, **BIDI controls**, and **annotation characters** that can break downstream processing.
- The module is invoked from **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** (line 941) and **[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** (line 25) to protect the data pipeline.
- Comprehensive tests in **`tests/test_sanitize_*.py`** verify correct handling of Unicode edge cases.

## Frequently Asked Questions

### What types of invisible characters does sanitize.py remove?

The script specifically targets **zero-width spaces (U+200B)**, **bidirectional formatting controls (U+202A, U+202B, U+202E, U+200F)**, and **annotation controls (U+FF9–U+FFFC)**. These characters do not render visibly but can alter string logic, directionality, or length calculations.

### How does sanitize.py report what it has cleaned?

The `sanitize_extracted_text()` function returns a tuple containing the cleaned string and an integer representing the total count of invisible codepoints removed. This allows calling code to log sanitization metrics or audit the extent of text transformation.

### Where is the sanitization logic triggered during document processing?

According to the source code, **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** invokes `sanitize_extracted_text` at line 941 immediately after parsing raw documents, ensuring that all text entering the skill-generation pipeline is pre-cleaned.

### Is there a way to validate that generated skills contain no invisible characters?

Yes. The standalone utility script **[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** imports `is_invisible_codepoint` and scans output artifacts at line 25, providing a final validation layer to catch any invisible characters that may have bypassed earlier filtering stages.