# How pdf-inspector Handles CJK and RTL Text: Ligature Expansion and NFKC Normalization Explained > Discover how pdf-inspector expertly handles CJK and RTL text with NFKC normalization and ligature expansion for accurate, searchable results. Learn about its advanced text processing. - Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector) - Tags: deep-dive - Published: 2026-08-14 --- **pdf-inspector processes CJK and RTL text through character classification, NFKC Unicode normalization, ligature expansion, and direction-aware layout sorting to produce semantically correct, searchable output.** The `firecrawl/pdf-inspector` Rust library implements sophisticated text processing for non-Latin scripts. Its pipeline correctly identifies Chinese, Japanese, Korean (CJK), and right-to-left (RTL) scripts, applies targeted Unicode normalization, and preserves logical reading order—critical for accurate PDF-to-Markdown conversion. ## Character Classification for CJK and RTL Scripts pdf-inspector's detection system uses Unicode range matching to categorize characters before any transformation occurs. ### CJK Detection with `is_cjk_char` In [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) (lines 87-102), the `is_cjk_char` function identifies CJK scripts by checking against multiple Unicode blocks: - Hangul Jamo and Hangul syllables - Hiragana and Katakana - CJK Unified Ideographs - CJK Compatibility Ideographs - Half-width forms (Japanese/Chinese punctuation) ```rust use pdf_inspector::text_utils::is_cjk_char; fn demo_cjk() { let zh = '中'; // CJK Unified Ideograph let ja = 'ひ'; // Hiragana let ko = '한'; // Hangul syllable println!("Chinese: {}", is_cjk_char(zh)); // true println!("Japanese: {}", is_cjk_char(ja)); // true println!("Korean: {}", is_cjk_char(ko)); // true } ``` ### RTL Detection with `is_rtl_char` The companion function `is_rtl_char` (lines 104-118) recognizes: - Hebrew - Arabic (including all presentation forms) - Syriac - Thaana - NKo, Samaritan, Mandaic ```rust use pdf_inspector::text_utils::is_rtl_char; fn demo_rtl() { let he = 'ש'; // Hebrew let ar = 'م'; // Arabic let sy = 'ܐ'; // Syriac println!("Hebrew: {}", is_rtl_char(he)); // true println!("Arabic: {}", is_rtl_char(ar)); // true println!("Syriac: {}", is_rtl_char(sy)); // true } ``` These classification helpers enable downstream decisions about word boundaries, spacing, and text direction. ## Ligature Expansion and NFKC Normalization The `expand_ligatures` function (lines 93-60 in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs)) handles Unicode standardization through a multi-stage pipeline. ### Processing Pipeline 1. **Strips control characters** – removes invisible formatting codes 2. **Detects Arabic Presentation Forms** – checks for U+FB00–U+FDFF range 3. **Conditionally applies NFKC** – only when presentation forms are present, avoiding unnecessary computation 4. **Expands Latin ligatures** – converts `ff` → `ff`, `fi` → `fi`, `fl` → `fl`, etc. 5. **Removes invisible characters** – deletes zero-width joiners and non-joiners 6. **Normalizes whitespace** – collapses multiple spaces ```rust use pdf_inspector::text_utils::expand_ligatures; fn demo_normalization() { // Latin ligature expansion let with_ligature = "firstly"; // U+FB01 ligature println!("{}", expand_ligatures(with_ligature)); // Output: "firstly" // Soft hyphen removal let with_hyphen = "invis\u{00AD}ible"; println!("{}", expand_ligatures(with_hyphen)); // Output: "invisible" } ``` ### Arabic Presentation Forms and Visual-to-Logical Reordering When Arabic Presentation Forms are detected, pdf-inspector performs two critical operations: 1. **NFKC normalization** – converts presentation forms (like U+FEE1, U+FEF3) to their base Arabic characters 2. **Visual-to-logical reordering** – calls `reverse_visual_arabic` (lines 62-70, 71-110) to restore correct reading order ```rust use pdf_inspector::text_utils::{expand_ligatures, reverse_visual_arabic}; fn demo_arabic() { // Mixed visual-order Arabic with Latin ligature let raw = "\u{FB01}rst\u{00AD}ly \u{FEE1}\u{FEF3}"; // first­ly (with ligature and soft-hyphen) + visual-order Arabic let cleaned = expand_ligatures(raw); // NFKC applied, ligature expanded, then reordered to logical RTL println!("cleaned: {}", cleaned); // Output: "firstly" followed by base Arabic in logical order } ``` The `reverse_visual_arabic` function handles both pure RTL runs and mixed LTR/RTL bidirectional text. ## Direction-Aware Layout and Line Building ### RTL-Aware Sorting with `sort_line_items` At lines 45-52 in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs), `sort_line_items` determines text direction using `is_rtl_text` (which aggregates characters via `is_rtl_char`). The function then orders items either left-to-right or right-to-left accordingly: ```rust // Called during Markdown conversion in src/markdown/convert.rs pub fn sort_line_items(items: &mut [TextItem]) { let rtl = is_rtl_text(items); if rtl { items.sort_by(|a, b| b.bounds.x().partial_cmp(&a.bounds.x()).unwrap()); } else { items.sort_by(|a, b| a.bounds.x().partial_cmp(&b.bounds.x()).unwrap()); } } ``` This ensures that extracted RTL text maintains correct logical order regardless of how the PDF stores positional data. ### CJK-Aware Gap Collection and Joining pdf-inspector's spacing logic special-cases CJK scripts at two points: **`collect_gap_ratios` (lines 69-81)** – skips gap ratio calculation for CJK pairs entirely: ```rust // Pseudocode from source analysis if is_cjk_char(left) || is_cjk_char(right) { continue; // Skip gap collection for CJK } ``` **`should_join_items` (lines 103-108, fallback at 398-410)** – applies CJK-specific joining rules: - CJK items are **always joined** when gap < 0.8 × character width - No space insertion between CJK characters (contextually appropriate, as CJK scripts don't use inter-word spaces) ```rust // Fallback branch for CJK items if is_cjk_char(left_last) || is_cjk_char(right_first) { return gap < char_width * 0.8; // Always join close CJK fragments } ``` This prevents incorrect space insertion in Chinese, Japanese, or Korean text while preserving normal spacing rules for Latin scripts. ## Complete Processing Flow pdf-inspector's CJK and RTL handling operates across five stages: | Stage | Location | Key Function | |-------|----------|--------------| | Extraction | [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) | `is_cjk_char`, `is_rtl_char` classify characters | | Preprocessing | [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs) | `expand_ligatures` normalizes Unicode | | Normalization | [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) | `reverse_visual_arabic` fixes RTL order | | Line building | [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) | `sort_line_items` arranges by direction | | Spacing | [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) | `should_join_items` applies CJK/RTL rules | The output is semantically correct Markdown with searchable text, proper logical order for RTL scripts, and appropriate word boundaries for all supported languages. ## Summary - **Character classification** via `is_cjk_char` and `is_rtl_char` enables script-specific processing decisions throughout the pipeline - **NFKC normalization** is applied conditionally when Arabic Presentation Forms are detected, converting presentation forms to base characters - **Ligature expansion** converts typographic ligatures (`fi`, `ff`, `fl`) to their component characters for searchability - **Visual-to-logical reordering** via `reverse_visual_arabic` restores correct reading order for RTL text in visually-ordered PDFs - **CJK-aware spacing** prevents incorrect word breaks in Chinese, Japanese, and Korean by skipping gap heuristics and always joining proximal CJK items - **Direction-aware sorting** in `sort_line_items` ensures RTL text flows right-to-left in the final output ## Frequently Asked Questions ### When does pdf-inspector apply NFKC normalization? NFKC normalization is applied only when Arabic Presentation Forms (U+FB00–U+FDFF) are detected in the input. This conditional approach, implemented in `expand_ligatures` at [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) lines 93-60, avoids unnecessary computation for pure Latin text while ensuring Arabic compatibility characters are properly normalized to their base forms. ### Why does pdf-inspector skip gap ratio collection for CJK text? CJK scripts do not use spaces between words, so gap-based heuristics for word boundary detection are inappropriate. The `collect_gap_ratios` function (lines 69-81) explicitly excludes CJK pairs from gap measurement, preventing the spacing algorithms from incorrectly splitting CJK text into words based on visual separation. ### How does pdf-inspector handle mixed LTR and RTL text on the same line? The `reverse_visual_arabic` function (lines 62-70, 71-110) handles bidirectional text by identifying pure RTL runs and mixed LTR/RTL segments. It reverses visual-order Arabic while preserving the relative position of embedded LTR text (like numbers or Latin words), producing logically-ordered output that matches standard Unicode bidirectional algorithm expectations. ### What ligatures does pdf-inspector expand beyond Latin typographic forms? In addition to common Latin ligatures (`ff`, `fi`, `fl`, `ffi`, `ffl`), the `expand_ligatures` function removes soft hyphens (`\u{00AD}`), zero-width joiners, and other invisible Unicode characters. The expansion is performed after NFKC normalization, ensuring that all text is both searchable and semantically correct before final Markdown generation.