# Complete List of Unicode Characters Stripped by the book-to-skill Sanitizer

> Discover the 8 Unicode character categories the book-to-skill sanitizer removes to block prompt injection. See the full list including zero-width spaces and bidirectional overrides.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: api-reference
- Published: 2026-08-30

---

**The `sanitize_extracted_text` function removes eight distinct categories of invisible or format-control Unicode characters—including zero-width spaces, bidirectional overrides, variation selectors, and tag block codepoints—to prevent hidden prompt injection attacks in text extracted by the virgiliojr94/book-to-skill repository.**

The virgiliojr94/book-to-skill project extracts prose from documents for downstream LLM processing, making aggressive input sanitization critical for security. The specific Unicode characters stripped by the sanitizer are hardcoded as constant sets and ranges in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), targeting codepoints that either render invisibly or reorder text without changing logical content. The public API `sanitize_extracted_text(text: str) -> (str, int)` iterates over each character, checks membership via `is_invisible_codepoint(ord(character))`, and returns the cleaned string along with a removal count.

## Zero-Width Spacers and Invisible Separators

The sanitizer eliminates every character defined in the `_ZERO_WIDTH_CODEPOINTS` frozenset. These 12 codepoints occupy zero horizontal space, allowing attackers to embed hidden instructions between visible tokens:

- **U+200B** (Zero Width Space)
- **U+200C** (Zero Width Non-Joiner)
- **U+200D** (Zero Width Joiner)
- **U+2060** (Word Joiner)
- **U+FEFF** (Byte Order Mark / Zero Width No-Break Space)
- **U+00AD** (Soft Hyphen)
- **U+034F** (Combining Grapheme Joiner)
- **U+180E** (Mongolian Vowel Separator)
- **U+2061** (Function Application)
- **U+2062** (Invisible Times)
- **U+2063** (Invisible Separator)
- **U+2064** (Invisible Plus)

## Bidirectional Control Characters (Trojan-Source)

The `_BIDI_CONTROL_CODEPOINTS` set targets 12 directional formatting characters capable of reordering displayed text without altering the logical string order read by models. These "Trojan-Source" vectors include:

- **U+200E** (Left-to-Right Mark)
- **U+200F** (Right-to-Left Mark)
- **U+061C** (Arabic Letter Mark)
- **U+202A** (Left-to-Right Embedding)
- **U+202B** (Right-to-Left Embedding)
- **U+202C** (Pop Directional Formatting)
- **U+202D** (Left-to-Right Override)
- **U+202E** (Right-to-Left Override)
- **U+2066** (Left-to-Right Isolate)
- **U+2067** (Right-to-Left Isolate)
- **U+2068** (First Strong Isolate)
- **U+2069** (Pop Directional Isolate)

## Invisible Letters

Four codepoints in `_INVISIBLE_LETTER_CODEPOINTS` represent "hangul filler" and other letter-like characters that possess no visual width, allowing hidden payloads to survive whitespace normalization:

- **U+115F** (Hangul Choseong Filler)
- **U+1160** (Hangul Jungseong Filler)
- **U+3164** (Hangul Filler)
- **U+FFA0** (Halfwidth Hangul Filler)

## Deprecated Format and Annotation Controls

The sanitizer strips legacy annotation format characters defined in `_ANNOTATION_FORMAT_CODEPOINTS`, which formerly marked interlinear annotations and invisible terminators:

- **U+206A** through **U+206F** (Inhibit Symmetric Swapping, Activate Symmetric Swapping, Inhibit Arabic Form Shaping, Activate Arabic Form Shaping, National Digit Shapes, Nominal Digit Shapes)
- **U+FFF9** (Interlinear Annotation Anchor)
- **U+FFFA** (Interlinear Annotation Separator)
- **U+FFFB** (Interlinear Annotation Terminator)

## Hidden Payload Vectors: Tag Blocks and Variation Selectors

Three specific ranges are treated as high-capacity smuggling channels capable of encoding arbitrary ASCII payloads invisibly.

**Unicode Tag Block:** All codepoints from `0xE0000` to `0xE007F` (inclusive) are removed based on `_TAG_BLOCK_START` and `_TAG_BLOCK_END` constants. This 128-character private-use range allows embedding arbitrary data that renders invisibly.

**Variation Selectors:** Two ranges defined in `_VARIATION_SELECTOR_RANGES` are stripped:
- **U+FE00** through **U+FE0F** (Variation Selectors 1–16)
- **U+E0100** through **U+E01EF** (Variation Selectors 17–256)

These combining marks modify the preceding base glyph but can be abused to encode hidden bits when chained.

**Musical Format Controls:** The range `0x1D173` to `0x1D17A` (inclusive), defined by `_MUSICAL_FORMAT_RANGE`, removes eight zero-width musical notation format characters used for beaming and slurring.

## Implementation in sanitize_extracted_text

The stripping logic resides in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py). The `sanitize_extracted_text` function iterates the input string and calls `is_invisible_codepoint()`, which checks membership against all constant sets and range boundaries described above. If a character’s ordinal value matches any criteria, it is excluded from the output accumulator.

Test coverage confirming this behavior exists in:
- [`tests/test_sanitize_extracted_text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_extracted_text.py) (general functionality and counting)
- [`tests/test_sanitize_bidi_controls.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_bidi_controls.py) (bidirectional character verification)
- [`tests/test_sanitize_annotation_controls.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_annotation_controls.py) (annotation format character verification)

## Code Examples

```python
from book_to_skill.sanitize import sanitize_extracted_text

# Example: a string containing a zero‑width space (U+200B) and a right‑to‑left mark (U+200F)

raw = "Hello\u200BWorld\u200F!"
clean, removed = sanitize_extracted_text(raw)

print(clean)   # → "HelloWorld!"

print(removed) # → 2  (the zero‑width space and the RTL mark were stripped)

```

```python

# Removing a variation selector sequence (U+FE0F) that would otherwise be invisible

raw = "👍\uFE0F is a thumb‑up emoji with a variation selector"
clean, removed = sanitize_extracted_text(raw)

print(clean)   # → "👍 is a thumb‑up emoji with a variation selector"

print(removed) # → 1

```

```python

# Stripping an entire Unicode tag block payload

raw = "Secret\uE0001\uE0002\uE0003Text"
clean, removed = sanitize_extracted_text(raw)

print(clean)   # → "SecretText"

print(removed) # → 3

```

## Summary

- The sanitizer targets **eight distinct categories** of invisible or format-control Unicode characters defined in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).
- **Zero-width spacers**, **bidirectional controls**, **invisible letters**, and **annotation format characters** are explicitly enumerated in frozenset constants.
- **Range-based removal** covers the Unicode tag block (`0xE0000–0xE007F`), variation selectors (`0xFE00–0xFE0F` and `0xE0100–0xE01EF`), and musical format controls (`0x1D173–0x1D17A`).
- The `sanitize_extracted_text` function returns a tuple of `(cleaned_string, removal_count)` after processing through `is_invisible_codepoint`.
- Comprehensive unit tests in [`tests/test_sanitize_extracted_text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_extracted_text.py) and focused test files verify removal accuracy for each attack vector.

## Frequently Asked Questions

### Which Unicode ranges does the sanitizer treat as hidden payload vectors?

The implementation interprets three specific ranges as high-risk smuggling channels: the Unicode tag block (`0xE0000–0xE007F`), variation selectors (`0xFE00–0xFE0F` and `0xE0100–0xE01EF`), and musical format controls (`0x1D173–0x1D17A`). These are checked via the `_TAG_BLOCK_START`, `_VARIATION_SELECTOR_RANGES`, and `_MUSICAL_FORMAT_RANGE` constants in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py).

### How can I verify the sanitizer is working correctly?

The repository includes focused test suites in [`tests/test_sanitize_extracted_text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_extracted_text.py), [`tests/test_sanitize_bidi_controls.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_bidi_controls.py), and [`tests/test_sanitize_annotation_controls.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_sanitize_annotation_controls.py). These verify that `sanitize_extracted_text` correctly identifies and removes characters from each category while returning accurate removal counts.

### Does the sanitizer remove all zero-width characters?

Yes, the implementation specifically targets every zero-width spacer defined in `_ZERO_WIDTH_CODEPOINTS`, including zero-width space (`U+200B`), zero-width non-joiner (`U+200C`), zero-width joiner (`U+200D`), word joiner (`U+2060`), and soft hyphen (`U+00AD`), plus the invisible mathematical operators `U+2061` through `U+2064`.

### What return value does `sanitize_extracted_text` provide?

According to the function signature in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), `sanitize_extracted_text(text: str) -> (str, int)` returns a tuple where the first element is the cleaned string with all invisible characters removed, and the second element is an integer count of how many codepoints were stripped during processing.