# Which Invisible Unicode Characters Are Stripped by the Book-to-Skill Sanitizer?

> Discover which invisible Unicode characters the book-to-skill sanitizer removes, including zero-width spaces and bidirectional controls. Learn how it cleans text effectively.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-28

---

**The `sanitize_extracted_text` function removes eight distinct categories of invisible Unicode characters—including zero-width spaces, bidirectional controls, invisible Hangul letters, and Unicode tag characters—by delegating detection to the `is_invisible_codepoint` helper in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).**

The `book-to-skill` repository processes documents to extract structured skill data while neutralizing hidden Unicode payloads that could enable prompt-injection attacks. The sanitizer defends against these threats by explicitly enumerating and removing specific invisible Unicode characters that render no visible glyph but alter logical text interpretation.

## Categories of Invisible Unicode Characters Removed

The sanitizer targets characters grouped by functional similarity. Each category is defined as explicit sets or ranges within [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### Zero-Width and Invisible Spacers

These characters occupy no visual space but modify text layout or provide insertion points for hidden payloads. The sanitizer removes:

- `0x200B` (ZERO WIDTH SPACE)
- `0x200C` (ZERO WIDTH NON-JOINER)
- `0x200D` (ZERO WIDTH JOINER)
- `0x2060` (WORD JOINER)
- `0xFEFF` (ZERO WIDTH NO-BREAK SPACE)
- `0x00AD` (SOFT HYPHEN)
- `0x034F` (COMBINING GRAPHEME JOINER)
- `0x180E` (MONGOLIAN VOWEL SEPARATOR)
- `0x2061` (FUNCTION APPLICATION)
- `0x2062` (INVISIBLE TIMES)
- `0x2063` (INVISIBLE SEPARATOR)
- `0x2064` (INVISIBLE PLUS)

These code points are defined in lines 9–22 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### Bidirectional Formatting Controls (Trojan Source)

Controls that change visual ordering without affecting logical character sequence, enabling source-code spoofing attacks. The sanitizer strips:

- `0x200E` (LEFT-TO-RIGHT MARK)
- `0x200F` (RIGHT-TO-LEFT MARK)
- `0x061C` (ARABIC LETTER MARK)
- `0x202A`–`0x202E` (embedding and override controls)
- `0x2066`–`0x2069` (isolate controls)

These ranges appear in lines 35–48 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### Invisible Hangul Letters

Korean filler characters that occupy zero width and bypass whitespace-normalization filters:

- `0x115F` (HANGUL CHOSEONG FILLER)
- `0x1160` (HANGUL JUNGSEONG FILLER)
- `0x3164` (HANGUL FILLER)
- `0xFFA0` (HALFWIDTH HANGUL FILLER)

Defined in lines 53–58 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### Annotation Format Controls

Deprecated formatting characters originally intended for complex scripts or annotations that render nothing:

- `0x206A`–`0x206F` (symmetric swapping and Arabic form shaping controls)
- `0xFFF9`–`0xFFFB` (interlinear annotation anchor, separator, and terminator)

These appear in lines 64–74 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) and are also caught by a dedicated range check in lines 106–109.

### Unicode Tag Block Characters

Historically used for language tagging, this range now serves as a mechanism to smuggle ASCII payloads via invisible code points:

- Range `0xE0000`–`0xE007F`

Defined in lines 83–87 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### Variation Selectors

Combining marks that modify glyph presentation without visible width, capable of encoding hidden binary data:

- `0xFE00`–`0xFE0F`
- `0xE0100`–`0xE01EF`

Handled by range checks in lines 98–101 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### Musical Format Controls

Zero-width format characters from the musical symbols block:

- Range `0x1D173`–`0x1D17A`

Defined in lines 116–119 of [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

## How the Sanitizer Detects and Removes Hidden Characters

The detection logic resides in `is_invisible_codepoint`, which accepts an integer code point and returns `True` if the character belongs to any forbidden set or range. The public function `sanitize_extracted_text` iterates over input text, filters through this predicate, and returns the cleaned string plus a count of removed characters.

```python
from book_to_skill.sanitize import sanitize_extracted_text

# Example containing several invisible characters:

raw = "Hello\u200B\u200C\u200DWorld\u202E!"   # ZERO WIDTH SPACE, NON-JOINER, JOINER, RIGHT-TO-LEFT OVERRIDE

clean, removed = sanitize_extracted_text(raw)
print(clean)   # → "HelloWorld!"

print(removed) # → 4

```

You can also use the helper directly to audit individual characters:

```python
from book_to_skill.sanitize import is_invisible_codepoint

print(is_invisible_codepoint(ord('\u2066')))  # LEFT-TO-RIGHT ISOLATE → True

print(is_invisible_codepoint(ord('A')))      # Regular character → False

```

## Implementation Details in the Codebase

The sanitization pipeline is centralized in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), with comprehensive test coverage ensuring no invisible characters survive processing:

- **[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)** – Defines all forbidden code point sets, the `is_invisible_codepoint` predicate, and the `sanitize_extracted_text` entry point.
- **[`tests/test_sanitize_extracted_text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_extracted_text.py)** – Unit tests verifying removal of each character category while preserving visible text.
- **[`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py)** – Consumes the sanitizer during skill validation to ensure generated outputs contain no hidden Unicode payloads.

## Summary

- The sanitizer removes **eight categories** of invisible Unicode characters defined explicitly in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).
- **Zero-width spaces**, **bidirectional controls**, **Hangul fillers**, **annotation marks**, **Unicode tag blocks**, **variation selectors**, and **musical format controls** are all stripped.
- The **`is_invisible_codepoint`** helper implements the detection logic, while **`sanitize_extracted_text`** provides the public API for cleaning strings.
- Each removal category targets specific attack vectors, from Trojan Source spoofing to hidden ASCII smuggling via language tags.

## Frequently Asked Questions

### What is the difference between `sanitize_extracted_text` and `is_invisible_codepoint`?

`sanitize_extracted_text` is the public API that processes entire strings, iterates through characters, and returns a cleaned version plus a removal count. `is_invisible_codepoint` is the internal helper that performs the actual membership test against forbidden code point sets and ranges, returning a boolean for single integer code points.

### Why does the sanitizer remove bidirectional formatting controls?

Bidirectional controls such as `U+202E` (RIGHT-TO-LEFT OVERRIDE) enable **Trojan Source** attacks, where source code appears logically valid but executes maliciously due to visual reordering. Stripping these characters prevents extracted document text from concealing harmful syntax via directionality spoofing.

### Can variation selectors be used to hide malicious code?

Yes. Variation selectors (`U+FE00`–`U+FE0F` and `U+E0100`–`U+E01EF`) modify glyph presentation without displaying visible marks. Attackers can encode binary data or hidden instructions by combining these with visible base characters, making their removal essential for secure text extraction.

### How can I test if a string contains invisible characters before processing?

Import `is_invisible_codepoint` from `book_to_skill/sanitize` and test each character's ordinal value, or simply run `sanitize_extracted_text` and check the returned removal count. If the count exceeds zero, the original string contained characters flagged by the sanitizer.