# What Unicode Characters Are Stripped for Security in book‑to‑skill

> Learn what dangerous Unicode characters book-to-skill strips for security. Discover how zero-width spacers, bidirectional controls, invisible letters, and tag block characters prevent prompt injection attacks.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: security
- Published: 2026-08-23

---

**The book‑to‑skill library removes four categories of dangerous Unicode characters—zero‑width spacers, bidirectional controls, invisible letters, and tag block characters—to prevent document‑borne prompt injection attacks.**

When processing ebooks and documents for AI training or inference, invisible characters can hide malicious instructions that humans cannot see but language models execute. The **book‑to‑skill** repository implements a defensive sanitization layer in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) that strips specific Unicode codepoints before text reaches downstream LLMs.

## Categories of Stripped Unicode Characters

The sanitization logic targets four distinct attack vectors defined as constants in the source code. The `is_invisible_codepoint` predicate (lines 72‑82) checks input against these sets.

### Zero‑Width and Invisible Spacers

These characters render no visible glyph, allowing attackers to embed hidden text between visible words. According to [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) (lines 9‑22), the library defines `_ZERO_WIDTH_CODEPOINTS` to catch:

- `0x200B` (ZERO WIDTH SPACE)
- `0x200C` (ZERO WIDTH NON‑JOINER)
- `0x200D` (ZERO WIDTH JOINER)
- `0x2060` (WORD JOINER)
- `0xFEFF` (ZERO WIDTH NO‑BREAK SPACE/BOM)
- `0x00AD` (SOFT HYPHEN)
- `0x034F` (COMBINING GRAPHEME JOINER)
- `0x180E` (MONGOLIAN VOWEL SEPARATOR)
- `0x2061`‑`0x2064` (FUNCTION APPLICATION, INVISIBLE TIMES, INVISIBLE SEPARATOR, INVISIBLE PLUS)

### Bidirectional (Bidi) Control Characters

These "Trojan‑Source" characters reorder visual text without changing logical order, enabling hidden instructions that appear benign to human reviewers. Defined in `_BIDI_CONTROL_CODEPOINTS` (lines 35‑48), the stripped controls include:

- `0x200E` (LEFT‑TO‑RIGHT MARK) and `0x200F` (RIGHT‑TO‑LEFT MARK)
- `0x061C` (ARABIC LETTER MARK)
- `0x202A`‑`0x202E` (LEFT‑TO‑RIGHT/RIGHT‑TO‑LEFT EMBEDDING, POP DIRECTIONAL FORMATTING, LEFT‑TO‑RIGHT/RIGHT‑TO‑LEFT OVERRIDE)
- `0x2066`‑`0x2069` (LEFT‑TO‑RIGHT ISOLATE, RIGHT‑TO‑LEFT ISOLATE, FIRST STRONG ISOLATE, POP DIRECTIONAL ISOLATE)

### Invisible Letters

Letter‑type characters occupying zero width survive whitespace normalization and can pad hidden payloads. The `_INVISIBLE_LETTER_CODEPOINTS` set (lines 53‑58) targets:

- `0x115F` (HANGUL CHOSEONG FILLER)
- `0x1160` (HANGUL JUNGSEONG FILLER)
- `0x3164` (HANGUL FILLER)
- `0xFFA0` (HALFWIDTH HANGUL FILLER)

### Tag Block Characters

The entire private‑use block U+E0000‑U+E007F, originally intended for language tags, is now repurposed to smuggle arbitrary ASCII payloads as invisible tags. The `is_invisible_codepoint` predicate checks this range inline at lines 80‑82, stripping any codepoint where `0xE0000 ≤ cp ≤ 0xE007F`.

## How the Sanitization Works

The core implementation resides in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py). The `sanitize_extracted_text` function (lines 85‑96) walks input strings character by character, invokes `is_invisible_codepoint` to detect malicious codepoints, discards them, and returns a tuple of cleaned text plus the count of removed characters.

```python
from book_to_skill.sanitize import sanitize_extracted_text

# Example: a string containing a zero‑width space and a right‑to‑left mark

raw = "Hello\u200BWorld\u200F!"   # ZERO WIDTH SPACE + RIGHT‑TO‑LEFT MARK

clean, removed = sanitize_extracted_text(raw)

print(clean)      # -> "HelloWorld!"

print(removed)    # -> 2 (both invisible characters were stripped)

```

## Security Implications

Stripping these **Unicode characters for security** mitigates document‑borne prompt injection attacks. Attackers can embed instructions in ebooks or articles that remain invisible to human reviewers but are processed by LLMs. By removing zero‑width spacers, Bidi controls, and tag blocks, book‑to‑skill prevents attacks such as CVE‑2021‑42574 (Trojan‑Source) and ensures only visible, human‑readable content reaches the model.

## Practical Usage Examples

Remove tag‑block characters that smuggle hidden payloads:

```python

# Example: stripping a tag‑block character (U+E0061)

raw = "Safe text" + chr(0xE0061) + " more safe text"
clean, removed = sanitize_extracted_text(raw)

print(clean)      # -> "Safe text more safe text"

print(removed)    # -> 1

```

Use the low‑level predicate directly for custom pipelines or debugging:

```python
from book_to_skill.sanitize import is_invisible_codepoint

assert is_invisible_codepoint(0x200B)  # ZERO WIDTH SPACE → True

assert not is_invisible_codepoint(ord('A'))  # Regular letter → False

```

## Summary

- **book‑to‑skill** strips four Unicode categories: zero‑width spacers, Bidi controls, invisible letters, and tag blocks.
- The logic is implemented in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) within `is_invisible_codepoint` (lines 72‑82) and `sanitize_extracted_text` (lines 85‑96).
- These characters enable "Trojan‑Source" attacks and invisible prompt injection via hidden instructions.
- The sanitizer returns cleaned text and a removal count, making it safe to process untrusted documents for LLM consumption.

## Frequently Asked Questions

### Why does book‑to‑skill strip Unicode characters instead of escaping them?

Stripping ensures zero‑width and directional control characters cannot alter the logical interpretation of text processed by language models. Escaping would preserve the codepoints in the data stream, allowing potential exploitation during tokenization or rendering. The `sanitize_extracted_text` function removes them entirely to guarantee only visible glyphs reach downstream pipelines.

### Which file contains the list of dangerous codepoints?

The constants defining stripped characters live in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py). Lines 9‑22 define `_ZERO_WIDTH_CODEPOINTS`, lines 35‑48 define `_BIDI_CONTROL_CODEPOINTS`, and lines 53‑58 define `_INVISIBLE_LETTER_CODEPOINTS`. The tag block range is checked inline within the `is_invisible_codepoint` function at lines 80‑82.

### Can the sanitizer handle mixed‑language documents with legitimate directional text?

Yes. The sanitizer targets only explicit directional *control* characters (LRM, RLM, overrides, and isolates) that manipulate text ordering, not the natural directional properties of scripts like Arabic or Hebrew. Legitimate bidirectional text encoded without these explicit control characters passes through unchanged.

### How can I verify that my document contains invisible characters before processing?

Import the `is_invisible_codepoint` predicate from `book_to_skill.sanitize` and iterate over your string to audit specific codepoints. For production workflows, call `sanitize_extracted_text` directly; it returns a count of removed characters, alerting you to the presence of invisible payloads in the source document.