What Is the Purpose of the sanitize.py Script in book-to-skill?

The sanitize.py script in book-to-speak removes invisible Unicode codepoints—such as zero-width spaces and bidirectional controls—from raw text extracts to guarantee clean, deterministic input for downstream processing.

The book-to-skill repository ingests documents in formats like PDF, RTF, HTML, and EPUB to build skill mappings. Located at book_to_skill/sanitize.py, this module acts as a defensive filter that eliminates hidden characters before they can distort string length, break tokenization, or introduce security vulnerabilities.

Core Sanitization Functions

The module exposes two public utilities designed to handle Unicode sanitization at different levels of granularity.

is_invisible_codepoint()

This function detects characters that render invisibly or alter text direction without visual representation. According to the source code in book_to_skill/sanitize.py, it identifies zero-width spaces (U+200B), BIDI controls such as U+202A, and various annotation controls. These codepoints can alter the logical order of strings or hide malicious payloads, making their detection critical for safe text processing.

sanitize_extracted_text()

The primary entry point for the module, sanitize_extracted_text(text: str) -> Tuple[str, int], iterates through input strings and strips every invisible codepoint identified by is_invisible_codepoint(). It returns a tuple containing the cleaned string and an integer count of removed characters. This ensures that metrics and skill extractions calculated later in the pipeline reflect only visible, user-intended content.

Where sanitize.py Is Used in the Codebase

The sanitization logic is invoked at critical boundaries within the virgiliojr94/book-to-skill repository to prevent corruption of processed data.

• book_to_skill/utils.py – After parsing a document (for example, extracting text from a PDF), the code calls sanitize_extracted_text at line 941 to cleanse the content before storage or analysis.
• tools/scan_generated_skill.py – This utility script imports is_invisible_codepoint to audit generated skill artifacts, flagging any stray invisible characters at line 25 before they reach production outputs.

Usage Examples

The following patterns demonstrate how to integrate the sanitization utilities into custom workflows:

from book_to_skill.sanitize import sanitize_extracted_text

raw = "Hello\u200BWorld\u202E!"   # contains zero-width space and a BIDI control

clean, removed = sanitize_extracted_text(raw)
print(clean)    # → "HelloWorld!"

print(removed)  # → 2

To check individual characters manually:

from book_to_skill.sanitize import is_invisible_codepoint

char = "\u202B"  # Right-to-Left Embedding

print(is_invisible_codepoint(char))   # → True

Test Coverage

The repository maintains rigorous test suites to validate sanitization behavior across Unicode categories:

• tests/test_sanitize_extracted_text.py – Confirms that zero-width characters and tag-block codepoints are removed while preserving multilingual text.
• tests/test_sanitize_bidi_controls.py – Verifies that bidirectional controls such as U+202E and U+200F are stripped and that the removal count is accurate.
• tests/test_sanitize_annotation_controls.py – Ensures that annotation controls in the U+FFF9 to U+FFFC range are eliminated from processed strings.

Summary

• sanitize.py provides the is_invisible_codepoint() and sanitize_extracted_text() utilities to cleanse raw document extracts.
• It targets zero-width spaces, BIDI controls, and annotation characters that can break downstream processing.
• The module is invoked from book_to_skill/utils.py (line 941) and tools/scan_generated_skill.py (line 25) to protect the data pipeline.
• Comprehensive tests in tests/test_sanitize_*.py verify correct handling of Unicode edge cases.

Frequently Asked Questions

What types of invisible characters does sanitize.py remove?

The script specifically targets zero-width spaces (U+200B), bidirectional formatting controls (U+202A, U+202B, U+202E, U+200F), and annotation controls (U+FF9–U+FFFC). These characters do not render visibly but can alter string logic, directionality, or length calculations.

How does sanitize.py report what it has cleaned?

The sanitize_extracted_text() function returns a tuple containing the cleaned string and an integer representing the total count of invisible codepoints removed. This allows calling code to log sanitization metrics or audit the extent of text transformation.

Where is the sanitization logic triggered during document processing?

According to the source code, book_to_skill/utils.py invokes sanitize_extracted_text at line 941 immediately after parsing raw documents, ensuring that all text entering the skill-generation pipeline is pre-cleaned.

Is there a way to validate that generated skills contain no invisible characters?

Yes. The standalone utility script tools/scan_generated_skill.py imports is_invisible_codepoint and scans output artifacts at line 25, providing a final validation layer to catch any invisible characters that may have bypassed earlier filtering stages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}