# How pdf‑inspector Handles CJK and RTL Text: Complete Implementation Guide > Learn how pdf-inspector expertly handles CJK and RTL text with Unicode checks, direction-aware sorting, and visual-order glyph run fixing in this implementation guide. - Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector) - Tags: how-to-guide - Published: 2026-09-01 --- **pdf‑inspector treats Chinese‑Japanese‑Korean (CJK) and right‑to‑left (RTL) scripts as first‑class citizens throughout the extraction pipeline, detecting scripts via Unicode range checks, applying direction‑aware sorting, and fixing visual‑order glyph runs.** pdf‑inspector, Firecrawl's Rust‑based PDF text extraction engine, implements dedicated handling for complex scripts that lack word separators or read in reverse order. The implementation spans character classification, directional analysis, layout processing, and visual‑order correction—all concentrated in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) with orchestration from the layout and markdown modules. ## Character Classification: `is_cjk_char` and `is_rtl_char` The foundation of CJK and RTL support lies in two predicate functions that classify individual code points. ### CJK Detection `is_cjk_char` matches Unicode ranges for: - Hangul Jamo (U+1100–U+11FF) - CJK Symbols and Punctuation (U+3000–U+303F) - Hiragana (U+3040–U+309F) - Katakana (U+30A0–U+30FF) - CJK Ideographs (U+4E00–U+9FFF) - Hangul Syllables (U+AC00–U+D7AF) - CJK Compatibility Ideographs (U+F900–U+FAFF) - Halfwidth/Fullwidth forms (U+FF00–U+FFEF) ```rust // src/text_utils.rs (lines 87-95) pub fn is_cjk_char(c: char) -> bool { matches!(c, '\u{1100}'..='\u{11FF}' | // Hangul Jamo '\u{3000}'..='\u{303F}' | // CJK Symbols and Punctuation '\u{3040}'..='\u{309F}' | // Hiragana '\u{30A0}'..='\u{30FF}' | // Katakana '\u{4E00}'..='\u{9FFF}' | // CJK Unified Ideographs '\u{AC00}'..='\u{D7AF}' | // Hangul Syllables '\u{F900}'..='\u{FAFF}' | // CJK Compatibility Ideographs '\u{FF00}'..='\u{FFEF}') // Halfwidth and Fullwidth Forms } ``` ### RTL Detection `is_rtl_char` identifies bidirectional scripts including Hebrew, Arabic, Syriac, Thaana, NKo, Samaritan, Mandaic, and their presentation forms. ```rust // src/text_utils.rs (lines 97-119) pub fn is_rtl_char(c: char) -> bool { matches!(c, '\u{0590}'..='\u{05FF}' | // Hebrew '\u{0600}'..='\u{06FF}' | // Arabic '\u{0700}'..='\u{074F}' | // Syriac '\u{0780}'..='\u{07BF}' | // Thaana '\u{07C0}'..='\u{07FF}' | // NKo '\u{0800}'..='\u{083F}' | // Samaritan '\u{0840}'..='\u{085F}' | // Mandaic '\u{08A0}'..='\u{08FF}' | // Arabic Extended-A '\u{FB1D}'..='\u{FB4F}' | // Hebrew presentation forms '\u{FB50}'..='\u{FDFF}' | // Arabic presentation forms A '\u{FE70}'..='\u{FEFF}') // Arabic presentation forms B } ``` ## Directional Analysis: `is_rtl_text` and `sort_line_items` For mixed or whole‑line directionality, pdf‑inspector implements statistical detection rather than relying on single‑character checks. ### Line‑Level RTL Detection `is_rtl_text` scans a collection of strings, counts **strong RTL letters** (ignoring combining marks and CJK characters), and returns `true` when RTL letters outnumber LTR letters. This prevents neutral characters or embedded CJK from skewing the determination. ```rust // src/text_utils.rs (lines 127-146) pub fn is_rtl_text<'a>(texts: impl Iterator) -> bool { let mut rtl_count = 0; let mut ltr_count = 0; for text in texts { for c in text.chars() { if c.is_ascii_alphabetic() { ltr_count += 1; } else if is_rtl_char(c) { rtl_count += 1; } // CJK and combining marks are deliberately ignored } } rtl_count > ltr_count } ``` ### Direction‑Aware Sorting `sort_line_items` uses the result of `is_rtl_text` to determine the correct reading order for `TextItem` collections. RTL lines are sorted by descending **x** coordinate; embedded LTR runs within RTL lines receive corrective handling to preserve logical order. ```rust // src/text_utils.rs (lines 148-162) pub fn sort_line_items(items: &mut [TextItem]) { if items.is_empty() { return; } let rtl = is_rtl_text(items.iter().map(|i| &i.text)); if rtl { // Sort right-to-left (descending x, then ascending y) items.sort_by(|a, b| { b.bbox.x1.partial_cmp(&a.bbox.x1) .unwrap() .then_with(|| a.bbox.y1.partial_cmp(&b.bbox.y1).unwrap()) }); } else { // Standard left-to-right sort items.sort_by(|a, b| { a.bbox.x1.partial_cmp(&b.bbox.x1) .unwrap() .then_with(|| a.bbox.y1.partial_cmp(&b.bbox.y1).unwrap()) }); } } ``` ## Layout‑Aware Processing for CJK and RTL pdf‑inspector adapts spacing, table detection, and cell ordering based on script classification. ### CJK Word Joining: `should_join_items` CJK languages do not use spaces between words. The `should_join_items` function contains a **fast‑path that always joins adjacent items when either item contains a CJK character**, bypassing normal gap‑ratio heuristics entirely. ```rust // src/text_utils.rs (lines 87-95, fallback block in should_join_items) fn should_join_items(prev: &TextItem, curr: &TextItem, threshold: f64) -> bool { // ... gap calculation logic ... // Fast-path: always join CJK characters if prev.text.ends_with(|c| is_cjk_char(c)) || curr.text.starts_with(|c| is_cjk_char(c)) { return true; } // Normal Otsu-based threshold comparison for non-CJK text gap_ratio < threshold } ``` This prevents insertion of spurious spaces in Japanese, Chinese, or Korean documents. ### RTL Table Cell Ordering: `sort_rtl_cell_items` For table structures with RTL content, `sort_rtl_cell_items` groups items by baseline, sorts each horizontal band right‑to‑left, then restores embedded LTR runs to ensure correct reading order. ```rust // src/text_utils.rs (lines 162-185) pub fn sort_rtl_cell_items(items: &mut [TextItem]) { // Group by baseline (y-coordinate) let mut bands: HashMap, Vec<&mut TextItem>> = HashMap::new(); for item in items.iter_mut() { let y_key = OrderedFloat(item.bbox.y1); bands.entry(y_key).or_default().push(item); } // Sort each band right-to-left for (_, band) in &mut bands { band.sort_by(|a, b| b.bbox.x1.partial_cmp(&a.bbox.x1).unwrap()); // Restore LTR runs within RTL context restore_ltr_runs(band); } } ``` ## Visual‑Order Correction: `fix_visual_order_rtl` Many PDFs—especially scanned documents—store RTL glyphs in **visual order** (left‑to‑right as painted) rather than logical order. `fix_visual_order_rtl` detects this convention by counting geometric votes and reverses affected runs when visual order dominates. ```rust // src/text_utils.rs (lines 556-584) pub fn fix_visual_order_rtl( items: &mut [TextItem], visual_candidates: &[usize], logical_votes: usize, ) { // Count geometric direction votes from item positions let mut visual_votes = 0; for window in visual_candidates.windows(2) { let i = window[0]; let j = window[1]; if items[i].bbox.x1 < items[j].bbox.x1 { visual_votes += 1; // Rightward progression suggests visual order } else { logical_votes += 1; // Leftward progression suggests logical order } } // When visual order dominates, reverse each RTL item's text if visual_votes > logical_votes { for &idx in visual_candidates { if is_rtl_text(std::iter::once(&items[idx].text)) { items[idx].text = reverse_visual_arabic(&items[idx].text); } } } } ``` The companion function `reverse_visual_arabic` handles Arabic‑specific shaping concerns during reversal. ## Gap‑Ratio Calculations: CJK Exclusion from Heuristics To prevent CJK text from distorting spacing thresholds used for Latin‑based content, `collect_gap_ratios` explicitly skips any item pair where either boundary character is CJK. ```rust // src/text_utils.rs (lines 438-449) fn collect_gap_ratios(items: &[TextItem]) -> Vec { let mut ratios = Vec::new(); for window in items.windows(2) { let prev = &window[0]; let curr = &window[1]; // Skip if CJK present at boundaries if prev.text.ends_with(|c| is_cjk_char(c)) || curr.text.starts_with(|c| is_cjk_char(c)) { continue; } let gap = curr.bbox.x0 - prev.bbox.x1; let font_size = prev.font_size.max(curr.font_size); ratios.push(gap / font_size); } ratios } ``` This exclusion ensures that Otsu thresholding for word‑boundary detection remains accurate for non‑CJK scripts. ## Complete Usage Example ```rust use pdf_inspector::text_utils::{ is_cjk_char, is_rtl_char, is_rtl_text, sort_line_items, fix_visual_order_rtl }; /// Detect script of a single character fn classify_character(ch: char) -> &'static str { if is_cjk_char(ch) { "CJK" } else if is_rtl_char(ch) { "RTL" } else { "Other" } } /// Process a line of extracted text items fn process_line(items: &mut [TextItem]) { // Determine if entire line is RTL let rtl = is_rtl_text(items.iter().map(|i| &i.text)); println!("Line direction: {}", if rtl { "RTL" } else { "LTR" }); // Apply appropriate sorting sort_line_items(items); // After initial extraction, check for visual-order PDF artifacts let candidates: Vec = items.iter() .enumerate() .filter(|(_, item)| is_rtl_text(std::iter::once(&item.text))) .map(|(i, _)| i) .collect(); fix_visual_order_rtl(items, &candidates, 0); } ``` ## Key Files in the CJK and RTL Pipeline | File | Responsibility | |------|---------------| | [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) | Core classification (`is_cjk_char`, `is_rtl_char`), direction analysis (`is_rtl_text`), sorting (`sort_line_items`, `sort_rtl_cell_items`), visual correction (`fix_visual_order_rtl`), and CJK‑aware gap handling | | [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs) | Invokes `should_join_items` with CJK fast‑path for word boundary detection | | [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) | Orchestrates line extraction and applies RTL‑aware sorting per line | | [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) | Final Markdown output preserving RTL directionality and CJK spacing | ## Summary - **Detection relies on explicit Unicode ranges**: `is_cjk_char` and `is_rtl_char` in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) identify scripts without external dependencies. - **Direction is determined statistically**: `is_rtl_text` counts strong RTL letters while ignoring CJK and combining marks, preventing false positives. - **CJK receives special spacing treatment**: The fast‑path in `should_join_items` always joins adjacent CJK items, eliminating spurious word breaks. - **RTL supports both logical and visual conventions**: `sort_line_items` handles logical‑order PDFs; `fix_visual_order_rtl` corrects visual‑order artifacts common in scanned documents. - **Table extraction preserves reading order**: `sort_rtl_cell_items` groups by baseline and handles embedded LTR runs within RTL columns. ## Frequently Asked Questions ### How does pdf‑inspector distinguish CJK from other Unicode scripts? pdf‑inspector uses the `is_cjk_char` function in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) to check if a character falls within defined Unicode blocks for Hangul, Hiragana, Katakana, and CJK ideographs. This explicit range checking operates without external libraries and runs in O(1) per character. ### Why does RTL detection ignore CJK characters when counting votes? The `is_rtl_text` function deliberately excludes CJK and combining marks from its direction tally because these characters are **directionally neutral** in Unicode's bidirectional algorithm. Counting them would skew the RTL‑versus‑LTR ratio, causing incorrect line direction classification in mixed documents. ### What happens when a PDF stores Arabic text in visual order? Many PDF generators—especially for scanned documents—render RTL glyphs left‑to‑right as they appear on the page. `fix_visual_order_rtl` detects this pattern by comparing geometric progression against logical expectations, then reverses affected runs using `reverse_visual_arabic` to restore proper reading sequence. ### Does CJK handling affect performance of Latin text extraction? No. The CJK fast‑path in `should_join_items` executes a single boundary character check before falling back to standard gap‑ratio heuristics. `collect_gap_ratios` explicitly excludes CJK pairs from threshold calculations, ensuring Latin text processing remains unaffected by CJK document presence.