What Unicode Characters Are Stripped for Security in book‑to‑skill

The book‑to‑skill library removes four categories of dangerous Unicode characters—zero‑width spacers, bidirectional controls, invisible letters, and tag block characters—to prevent document‑borne prompt injection attacks.

When processing ebooks and documents for AI training or inference, invisible characters can hide malicious instructions that humans cannot see but language models execute. The book‑to‑skill repository implements a defensive sanitization layer in book_to_skill/sanitize.py that strips specific Unicode codepoints before text reaches downstream LLMs.

Categories of Stripped Unicode Characters

The sanitization logic targets four distinct attack vectors defined as constants in the source code. The is_invisible_codepoint predicate (lines 72‑82) checks input against these sets.

Zero‑Width and Invisible Spacers

These characters render no visible glyph, allowing attackers to embed hidden text between visible words. According to book_to_skill/sanitize.py (lines 9‑22), the library defines _ZERO_WIDTH_CODEPOINTS to catch:

• 0x200B (ZERO WIDTH SPACE)
• 0x200C (ZERO WIDTH NON‑JOINER)
• 0x200D (ZERO WIDTH JOINER)
• 0x2060 (WORD JOINER)
• 0xFEFF (ZERO WIDTH NO‑BREAK SPACE/BOM)
• 0x00AD (SOFT HYPHEN)
• 0x034F (COMBINING GRAPHEME JOINER)
• 0x180E (MONGOLIAN VOWEL SEPARATOR)
• 0x2061‑0x2064 (FUNCTION APPLICATION, INVISIBLE TIMES, INVISIBLE SEPARATOR, INVISIBLE PLUS)

Bidirectional (Bidi) Control Characters

These "Trojan‑Source" characters reorder visual text without changing logical order, enabling hidden instructions that appear benign to human reviewers. Defined in _BIDI_CONTROL_CODEPOINTS (lines 35‑48), the stripped controls include:

• 0x200E (LEFT‑TO‑RIGHT MARK) and 0x200F (RIGHT‑TO‑LEFT MARK)
• 0x061C (ARABIC LETTER MARK)
• 0x202A‑0x202E (LEFT‑TO‑RIGHT/RIGHT‑TO‑LEFT EMBEDDING, POP DIRECTIONAL FORMATTING, LEFT‑TO‑RIGHT/RIGHT‑TO‑LEFT OVERRIDE)
• 0x2066‑0x2069 (LEFT‑TO‑RIGHT ISOLATE, RIGHT‑TO‑LEFT ISOLATE, FIRST STRONG ISOLATE, POP DIRECTIONAL ISOLATE)

Invisible Letters

Letter‑type characters occupying zero width survive whitespace normalization and can pad hidden payloads. The _INVISIBLE_LETTER_CODEPOINTS set (lines 53‑58) targets:

• 0x115F (HANGUL CHOSEONG FILLER)
• 0x1160 (HANGUL JUNGSEONG FILLER)
• 0x3164 (HANGUL FILLER)
• 0xFFA0 (HALFWIDTH HANGUL FILLER)

Tag Block Characters

The entire private‑use block U+E0000‑U+E007F, originally intended for language tags, is now repurposed to smuggle arbitrary ASCII payloads as invisible tags. The is_invisible_codepoint predicate checks this range inline at lines 80‑82, stripping any codepoint where 0xE0000 ≤ cp ≤ 0xE007F.

How the Sanitization Works

The core implementation resides in book_to_skill/sanitize.py. The sanitize_extracted_text function (lines 85‑96) walks input strings character by character, invokes is_invisible_codepoint to detect malicious codepoints, discards them, and returns a tuple of cleaned text plus the count of removed characters.

from book_to_skill.sanitize import sanitize_extracted_text

# Example: a string containing a zero‑width space and a right‑to‑left mark

raw = "Hello\u200BWorld\u200F!"   # ZERO WIDTH SPACE + RIGHT‑TO‑LEFT MARK

clean, removed = sanitize_extracted_text(raw)

print(clean)      # -> "HelloWorld!"

print(removed)    # -> 2 (both invisible characters were stripped)

Security Implications

Stripping these Unicode characters for security mitigates document‑borne prompt injection attacks. Attackers can embed instructions in ebooks or articles that remain invisible to human reviewers but are processed by LLMs. By removing zero‑width spacers, Bidi controls, and tag blocks, book‑to‑skill prevents attacks such as CVE‑2021‑42574 (Trojan‑Source) and ensures only visible, human‑readable content reaches the model.

Practical Usage Examples

Remove tag‑block characters that smuggle hidden payloads:

# Example: stripping a tag‑block character (U+E0061)

raw = "Safe text" + chr(0xE0061) + " more safe text"
clean, removed = sanitize_extracted_text(raw)

print(clean)      # -> "Safe text more safe text"

print(removed)    # -> 1

Use the low‑level predicate directly for custom pipelines or debugging:

from book_to_skill.sanitize import is_invisible_codepoint

assert is_invisible_codepoint(0x200B)  # ZERO WIDTH SPACE → True

assert not is_invisible_codepoint(ord('A'))  # Regular letter → False

Summary

• book‑to‑skill strips four Unicode categories: zero‑width spacers, Bidi controls, invisible letters, and tag blocks.
• The logic is implemented in book_to_skill/sanitize.py within is_invisible_codepoint (lines 72‑82) and sanitize_extracted_text (lines 85‑96).
• These characters enable "Trojan‑Source" attacks and invisible prompt injection via hidden instructions.
• The sanitizer returns cleaned text and a removal count, making it safe to process untrusted documents for LLM consumption.

Frequently Asked Questions

Why does book‑to‑skill strip Unicode characters instead of escaping them?

Stripping ensures zero‑width and directional control characters cannot alter the logical interpretation of text processed by language models. Escaping would preserve the codepoints in the data stream, allowing potential exploitation during tokenization or rendering. The sanitize_extracted_text function removes them entirely to guarantee only visible glyphs reach downstream pipelines.

Which file contains the list of dangerous codepoints?

The constants defining stripped characters live in book_to_skill/sanitize.py. Lines 9‑22 define _ZERO_WIDTH_CODEPOINTS, lines 35‑48 define _BIDI_CONTROL_CODEPOINTS, and lines 53‑58 define _INVISIBLE_LETTER_CODEPOINTS. The tag block range is checked inline within the is_invisible_codepoint function at lines 80‑82.

Can the sanitizer handle mixed‑language documents with legitimate directional text?

Yes. The sanitizer targets only explicit directional control characters (LRM, RLM, overrides, and isolates) that manipulate text ordering, not the natural directional properties of scripts like Arabic or Hebrew. Legitimate bidirectional text encoded without these explicit control characters passes through unchanged.

How can I verify that my document contains invisible characters before processing?

Import the is_invisible_codepoint predicate from book_to_skill.sanitize and iterate over your string to audit specific codepoints. For production workflows, call sanitize_extracted_text directly; it returns a count of removed characters, alerting you to the presence of invisible payloads in the source document.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}