Complete List of Unicode Characters Stripped by the book-to-skill Sanitizer

The sanitize_extracted_text function removes eight distinct categories of invisible or format-control Unicode characters—including zero-width spaces, bidirectional overrides, variation selectors, and tag block codepoints—to prevent hidden prompt injection attacks in text extracted by the virgiliojr94/book-to-skill repository.

The virgiliojr94/book-to-skill project extracts prose from documents for downstream LLM processing, making aggressive input sanitization critical for security. The specific Unicode characters stripped by the sanitizer are hardcoded as constant sets and ranges in book_to_skill/sanitize.py, targeting codepoints that either render invisibly or reorder text without changing logical content. The public API sanitize_extracted_text(text: str) -> (str, int) iterates over each character, checks membership via is_invisible_codepoint(ord(character)), and returns the cleaned string along with a removal count.

Zero-Width Spacers and Invisible Separators

The sanitizer eliminates every character defined in the _ZERO_WIDTH_CODEPOINTS frozenset. These 12 codepoints occupy zero horizontal space, allowing attackers to embed hidden instructions between visible tokens:

• U+200B (Zero Width Space)
• U+200C (Zero Width Non-Joiner)
• U+200D (Zero Width Joiner)
• U+2060 (Word Joiner)
• U+FEFF (Byte Order Mark / Zero Width No-Break Space)
• U+00AD (Soft Hyphen)
• U+034F (Combining Grapheme Joiner)
• U+180E (Mongolian Vowel Separator)
• U+2061 (Function Application)
• U+2062 (Invisible Times)
• U+2063 (Invisible Separator)
• U+2064 (Invisible Plus)

Bidirectional Control Characters (Trojan-Source)

The _BIDI_CONTROL_CODEPOINTS set targets 12 directional formatting characters capable of reordering displayed text without altering the logical string order read by models. These "Trojan-Source" vectors include:

• U+200E (Left-to-Right Mark)
• U+200F (Right-to-Left Mark)
• U+061C (Arabic Letter Mark)
• U+202A (Left-to-Right Embedding)
• U+202B (Right-to-Left Embedding)
• U+202C (Pop Directional Formatting)
• U+202D (Left-to-Right Override)
• U+202E (Right-to-Left Override)
• U+2066 (Left-to-Right Isolate)
• U+2067 (Right-to-Left Isolate)
• U+2068 (First Strong Isolate)
• U+2069 (Pop Directional Isolate)

Invisible Letters

Four codepoints in _INVISIBLE_LETTER_CODEPOINTS represent "hangul filler" and other letter-like characters that possess no visual width, allowing hidden payloads to survive whitespace normalization:

• U+115F (Hangul Choseong Filler)
• U+1160 (Hangul Jungseong Filler)
• U+3164 (Hangul Filler)
• U+FFA0 (Halfwidth Hangul Filler)

Deprecated Format and Annotation Controls

The sanitizer strips legacy annotation format characters defined in _ANNOTATION_FORMAT_CODEPOINTS, which formerly marked interlinear annotations and invisible terminators:

• U+206A through U+206F (Inhibit Symmetric Swapping, Activate Symmetric Swapping, Inhibit Arabic Form Shaping, Activate Arabic Form Shaping, National Digit Shapes, Nominal Digit Shapes)
• U+FFF9 (Interlinear Annotation Anchor)
• U+FFFA (Interlinear Annotation Separator)
• U+FFFB (Interlinear Annotation Terminator)

Hidden Payload Vectors: Tag Blocks and Variation Selectors

Three specific ranges are treated as high-capacity smuggling channels capable of encoding arbitrary ASCII payloads invisibly.

Unicode Tag Block: All codepoints from 0xE0000 to 0xE007F (inclusive) are removed based on _TAG_BLOCK_START and _TAG_BLOCK_END constants. This 128-character private-use range allows embedding arbitrary data that renders invisibly.

Variation Selectors: Two ranges defined in _VARIATION_SELECTOR_RANGES are stripped:

• U+FE00 through U+FE0F (Variation Selectors 1–16)
• U+E0100 through U+E01EF (Variation Selectors 17–256)

These combining marks modify the preceding base glyph but can be abused to encode hidden bits when chained.

Musical Format Controls: The range 0x1D173 to 0x1D17A (inclusive), defined by _MUSICAL_FORMAT_RANGE, removes eight zero-width musical notation format characters used for beaming and slurring.

Implementation in sanitize_extracted_text

The stripping logic resides in book_to_skill/sanitize.py. The sanitize_extracted_text function iterates the input string and calls is_invisible_codepoint(), which checks membership against all constant sets and range boundaries described above. If a character’s ordinal value matches any criteria, it is excluded from the output accumulator.

Test coverage confirming this behavior exists in:

• tests/test_sanitize_extracted_text.py (general functionality and counting)
• tests/test_sanitize_bidi_controls.py (bidirectional character verification)
• tests/test_sanitize_annotation_controls.py (annotation format character verification)

Code Examples

from book_to_skill.sanitize import sanitize_extracted_text

# Example: a string containing a zero‑width space (U+200B) and a right‑to‑left mark (U+200F)

raw = "Hello\u200BWorld\u200F!"
clean, removed = sanitize_extracted_text(raw)

print(clean)   # → "HelloWorld!"

print(removed) # → 2  (the zero‑width space and the RTL mark were stripped)

# Removing a variation selector sequence (U+FE0F) that would otherwise be invisible

raw = "👍\uFE0F is a thumb‑up emoji with a variation selector"
clean, removed = sanitize_extracted_text(raw)

print(clean)   # → "👍 is a thumb‑up emoji with a variation selector"

print(removed) # → 1

# Stripping an entire Unicode tag block payload

raw = "Secret\uE0001\uE0002\uE0003Text"
clean, removed = sanitize_extracted_text(raw)

print(clean)   # → "SecretText"

print(removed) # → 3

Summary

• The sanitizer targets eight distinct categories of invisible or format-control Unicode characters defined in book_to_skill/sanitize.py.
• Zero-width spacers, bidirectional controls, invisible letters, and annotation format characters are explicitly enumerated in frozenset constants.
• Range-based removal covers the Unicode tag block (0xE0000–0xE007F), variation selectors (0xFE00–0xFE0F and 0xE0100–0xE01EF), and musical format controls (0x1D173–0x1D17A).
• The sanitize_extracted_text function returns a tuple of (cleaned_string, removal_count) after processing through is_invisible_codepoint.
• Comprehensive unit tests in tests/test_sanitize_extracted_text.py and focused test files verify removal accuracy for each attack vector.

Frequently Asked Questions

Which Unicode ranges does the sanitizer treat as hidden payload vectors?

The implementation interprets three specific ranges as high-risk smuggling channels: the Unicode tag block (0xE0000–0xE007F), variation selectors (0xFE00–0xFE0F and 0xE0100–0xE01EF), and musical format controls (0x1D173–0x1D17A). These are checked via the _TAG_BLOCK_START, _VARIATION_SELECTOR_RANGES, and _MUSICAL_FORMAT_RANGE constants in book_to_skill/sanitize.py.

How can I verify the sanitizer is working correctly?

The repository includes focused test suites in tests/test_sanitize_extracted_text.py, tests/test_sanitize_bidi_controls.py, and tests/test_sanitize_annotation_controls.py. These verify that sanitize_extracted_text correctly identifies and removes characters from each category while returning accurate removal counts.

Does the sanitizer remove all zero-width characters?

Yes, the implementation specifically targets every zero-width spacer defined in _ZERO_WIDTH_CODEPOINTS, including zero-width space (U+200B), zero-width non-joiner (U+200C), zero-width joiner (U+200D), word joiner (U+2060), and soft hyphen (U+00AD), plus the invisible mathematical operators U+2061 through U+2064.

What return value does sanitize_extracted_text provide?

According to the function signature in book_to_skill/sanitize.py, sanitize_extracted_text(text: str) -> (str, int) returns a tuple where the first element is the cleaned string with all invisible characters removed, and the second element is an integer count of how many codepoints were stripped during processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}