Which Invisible Unicode Characters Are Stripped by the Book-to-Skill Sanitizer?

The sanitize_extracted_text function removes eight distinct categories of invisible Unicode characters—including zero-width spaces, bidirectional controls, invisible Hangul letters, and Unicode tag characters—by delegating detection to the is_invisible_codepoint helper in book_to_skill/sanitize.py.

The book-to-skill repository processes documents to extract structured skill data while neutralizing hidden Unicode payloads that could enable prompt-injection attacks. The sanitizer defends against these threats by explicitly enumerating and removing specific invisible Unicode characters that render no visible glyph but alter logical text interpretation.

Categories of Invisible Unicode Characters Removed

The sanitizer targets characters grouped by functional similarity. Each category is defined as explicit sets or ranges within book_to_skill/sanitize.py.

Zero-Width and Invisible Spacers

These characters occupy no visual space but modify text layout or provide insertion points for hidden payloads. The sanitizer removes:

• 0x200B (ZERO WIDTH SPACE)
• 0x200C (ZERO WIDTH NON-JOINER)
• 0x200D (ZERO WIDTH JOINER)
• 0x2060 (WORD JOINER)
• 0xFEFF (ZERO WIDTH NO-BREAK SPACE)
• 0x00AD (SOFT HYPHEN)
• 0x034F (COMBINING GRAPHEME JOINER)
• 0x180E (MONGOLIAN VOWEL SEPARATOR)
• 0x2061 (FUNCTION APPLICATION)
• 0x2062 (INVISIBLE TIMES)
• 0x2063 (INVISIBLE SEPARATOR)
• 0x2064 (INVISIBLE PLUS)

These code points are defined in lines 9–22 of book_to_skill/sanitize.py.

Bidirectional Formatting Controls (Trojan Source)

Controls that change visual ordering without affecting logical character sequence, enabling source-code spoofing attacks. The sanitizer strips:

• 0x200E (LEFT-TO-RIGHT MARK)
• 0x200F (RIGHT-TO-LEFT MARK)
• 0x061C (ARABIC LETTER MARK)
• 0x202A–0x202E (embedding and override controls)
• 0x2066–0x2069 (isolate controls)

These ranges appear in lines 35–48 of book_to_skill/sanitize.py.

Invisible Hangul Letters

Korean filler characters that occupy zero width and bypass whitespace-normalization filters:

• 0x115F (HANGUL CHOSEONG FILLER)
• 0x1160 (HANGUL JUNGSEONG FILLER)
• 0x3164 (HANGUL FILLER)
• 0xFFA0 (HALFWIDTH HANGUL FILLER)

Defined in lines 53–58 of book_to_skill/sanitize.py.

Annotation Format Controls

Deprecated formatting characters originally intended for complex scripts or annotations that render nothing:

• 0x206A–0x206F (symmetric swapping and Arabic form shaping controls)
• 0xFFF9–0xFFFB (interlinear annotation anchor, separator, and terminator)

These appear in lines 64–74 of book_to_skill/sanitize.py and are also caught by a dedicated range check in lines 106–109.

Unicode Tag Block Characters

Historically used for language tagging, this range now serves as a mechanism to smuggle ASCII payloads via invisible code points:

• Range 0xE0000–0xE007F

Defined in lines 83–87 of book_to_skill/sanitize.py.

Variation Selectors

Combining marks that modify glyph presentation without visible width, capable of encoding hidden binary data:

• 0xFE00–0xFE0F
• 0xE0100–0xE01EF

Handled by range checks in lines 98–101 of book_to_skill/sanitize.py.

Musical Format Controls

Zero-width format characters from the musical symbols block:

• Range 0x1D173–0x1D17A

Defined in lines 116–119 of book_to_skill/sanitize.py.

How the Sanitizer Detects and Removes Hidden Characters

The detection logic resides in is_invisible_codepoint, which accepts an integer code point and returns True if the character belongs to any forbidden set or range. The public function sanitize_extracted_text iterates over input text, filters through this predicate, and returns the cleaned string plus a count of removed characters.

from book_to_skill.sanitize import sanitize_extracted_text

# Example containing several invisible characters:

raw = "Hello\u200B\u200C\u200DWorld\u202E!"   # ZERO WIDTH SPACE, NON-JOINER, JOINER, RIGHT-TO-LEFT OVERRIDE

clean, removed = sanitize_extracted_text(raw)
print(clean)   # → "HelloWorld!"

print(removed) # → 4

You can also use the helper directly to audit individual characters:

from book_to_skill.sanitize import is_invisible_codepoint

print(is_invisible_codepoint(ord('\u2066')))  # LEFT-TO-RIGHT ISOLATE → True

print(is_invisible_codepoint(ord('A')))      # Regular character → False

Implementation Details in the Codebase

The sanitization pipeline is centralized in book_to_skill/sanitize.py, with comprehensive test coverage ensuring no invisible characters survive processing:

• book_to_skill/sanitize.py – Defines all forbidden code point sets, the is_invisible_codepoint predicate, and the sanitize_extracted_text entry point.
• tests/test_sanitize_extracted_text.py – Unit tests verifying removal of each character category while preserving visible text.
• tools/validate_skill.py – Consumes the sanitizer during skill validation to ensure generated outputs contain no hidden Unicode payloads.

Summary

• The sanitizer removes eight categories of invisible Unicode characters defined explicitly in book_to_skill/sanitize.py.
• Zero-width spaces, bidirectional controls, Hangul fillers, annotation marks, Unicode tag blocks, variation selectors, and musical format controls are all stripped.
• The is_invisible_codepoint helper implements the detection logic, while sanitize_extracted_text provides the public API for cleaning strings.
• Each removal category targets specific attack vectors, from Trojan Source spoofing to hidden ASCII smuggling via language tags.

Frequently Asked Questions

What is the difference between sanitize_extracted_text and is_invisible_codepoint?

sanitize_extracted_text is the public API that processes entire strings, iterates through characters, and returns a cleaned version plus a removal count. is_invisible_codepoint is the internal helper that performs the actual membership test against forbidden code point sets and ranges, returning a boolean for single integer code points.

Why does the sanitizer remove bidirectional formatting controls?

Bidirectional controls such as U+202E (RIGHT-TO-LEFT OVERRIDE) enable Trojan Source attacks, where source code appears logically valid but executes maliciously due to visual reordering. Stripping these characters prevents extracted document text from concealing harmful syntax via directionality spoofing.

Can variation selectors be used to hide malicious code?

Yes. Variation selectors (U+FE00–U+FE0F and U+E0100–U+E01EF) modify glyph presentation without displaying visible marks. Attackers can encode binary data or hidden instructions by combining these with visible base characters, making their removal essential for secure text extraction.

How can I test if a string contains invisible characters before processing?

Import is_invisible_codepoint from book_to_skill/sanitize and test each character's ordinal value, or simply run sanitize_extracted_text and check the returned removal count. If the count exceeds zero, the original string contained characters flagged by the sanitizer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}