How pdf‑inspector Handles CJK and RTL Text: Complete Implementation Guide

pdf‑inspector treats Chinese‑Japanese‑Korean (CJK) and right‑to‑left (RTL) scripts as first‑class citizens throughout the extraction pipeline, detecting scripts via Unicode range checks, applying direction‑aware sorting, and fixing visual‑order glyph runs.

pdf‑inspector, Firecrawl's Rust‑based PDF text extraction engine, implements dedicated handling for complex scripts that lack word separators or read in reverse order. The implementation spans character classification, directional analysis, layout processing, and visual‑order correction—all concentrated in src/text_utils.rs with orchestration from the layout and markdown modules.

Character Classification: is_cjk_char and is_rtl_char

The foundation of CJK and RTL support lies in two predicate functions that classify individual code points.

CJK Detection

is_cjk_char matches Unicode ranges for:

• Hangul Jamo (U+1100–U+11FF)
• CJK Symbols and Punctuation (U+3000–U+303F)
• Hiragana (U+3040–U+309F)
• Katakana (U+30A0–U+30FF)
• CJK Ideographs (U+4E00–U+9FFF)
• Hangul Syllables (U+AC00–U+D7AF)
• CJK Compatibility Ideographs (U+F900–U+FAFF)
• Halfwidth/Fullwidth forms (U+FF00–U+FFEF)

// src/text_utils.rs (lines 87-95)
pub fn is_cjk_char(c: char) -> bool {
    matches!(c,
        '\u{1100}'..='\u{11FF}' |   // Hangul Jamo
        '\u{3000}'..='\u{303F}' |   // CJK Symbols and Punctuation
        '\u{3040}'..='\u{309F}' |   // Hiragana
        '\u{30A0}'..='\u{30FF}' |   // Katakana
        '\u{4E00}'..='\u{9FFF}' |   // CJK Unified Ideographs
        '\u{AC00}'..='\u{D7AF}' |   // Hangul Syllables
        '\u{F900}'..='\u{FAFF}' |   // CJK Compatibility Ideographs
        '\u{FF00}'..='\u{FFEF}')    // Halfwidth and Fullwidth Forms
}

RTL Detection

is_rtl_char identifies bidirectional scripts including Hebrew, Arabic, Syriac, Thaana, NKo, Samaritan, Mandaic, and their presentation forms.

// src/text_utils.rs (lines 97-119)
pub fn is_rtl_char(c: char) -> bool {
    matches!(c,
        '\u{0590}'..='\u{05FF}' |   // Hebrew
        '\u{0600}'..='\u{06FF}' |   // Arabic
        '\u{0700}'..='\u{074F}' |   // Syriac
        '\u{0780}'..='\u{07BF}' |   // Thaana
        '\u{07C0}'..='\u{07FF}' |   // NKo
        '\u{0800}'..='\u{083F}' |   // Samaritan
        '\u{0840}'..='\u{085F}' |   // Mandaic
        '\u{08A0}'..='\u{08FF}' |   // Arabic Extended-A
        '\u{FB1D}'..='\u{FB4F}' |   // Hebrew presentation forms
        '\u{FB50}'..='\u{FDFF}' |   // Arabic presentation forms A
        '\u{FE70}'..='\u{FEFF}')    // Arabic presentation forms B
}

Directional Analysis: is_rtl_text and sort_line_items

For mixed or whole‑line directionality, pdf‑inspector implements statistical detection rather than relying on single‑character checks.

Line‑Level RTL Detection

is_rtl_text scans a collection of strings, counts strong RTL letters (ignoring combining marks and CJK characters), and returns true when RTL letters outnumber LTR letters. This prevents neutral characters or embedded CJK from skewing the determination.

// src/text_utils.rs (lines 127-146)
pub fn is_rtl_text<'a>(texts: impl Iterator<Item = &'a str>) -> bool {
    let mut rtl_count = 0;
    let mut ltr_count = 0;
    
    for text in texts {
        for c in text.chars() {
            if c.is_ascii_alphabetic() {
                ltr_count += 1;
            } else if is_rtl_char(c) {
                rtl_count += 1;
            }
            // CJK and combining marks are deliberately ignored
        }
    }
    
    rtl_count > ltr_count
}

Direction‑Aware Sorting

sort_line_items uses the result of is_rtl_text to determine the correct reading order for TextItem collections. RTL lines are sorted by descending x coordinate; embedded LTR runs within RTL lines receive corrective handling to preserve logical order.

// src/text_utils.rs (lines 148-162)
pub fn sort_line_items(items: &mut [TextItem]) {
    if items.is_empty() {
        return;
    }
    
    let rtl = is_rtl_text(items.iter().map(|i| &i.text));
    
    if rtl {
        // Sort right-to-left (descending x, then ascending y)
        items.sort_by(|a, b| {
            b.bbox.x1.partial_cmp(&a.bbox.x1)
                .unwrap()
                .then_with(|| a.bbox.y1.partial_cmp(&b.bbox.y1).unwrap())
        });
    } else {
        // Standard left-to-right sort
        items.sort_by(|a, b| {
            a.bbox.x1.partial_cmp(&b.bbox.x1)
                .unwrap()
                .then_with(|| a.bbox.y1.partial_cmp(&b.bbox.y1).unwrap())
        });
    }
}

Layout‑Aware Processing for CJK and RTL

pdf‑inspector adapts spacing, table detection, and cell ordering based on script classification.

CJK Word Joining: should_join_items

CJK languages do not use spaces between words. The should_join_items function contains a fast‑path that always joins adjacent items when either item contains a CJK character, bypassing normal gap‑ratio heuristics entirely.

// src/text_utils.rs (lines 87-95, fallback block in should_join_items)
fn should_join_items(prev: &TextItem, curr: &TextItem, threshold: f64) -> bool {
    // ... gap calculation logic ...
    
    // Fast-path: always join CJK characters
    if prev.text.ends_with(|c| is_cjk_char(c)) 
        || curr.text.starts_with(|c| is_cjk_char(c)) {
        return true;
    }
    
    // Normal Otsu-based threshold comparison for non-CJK text
    gap_ratio < threshold
}

This prevents insertion of spurious spaces in Japanese, Chinese, or Korean documents.

RTL Table Cell Ordering: sort_rtl_cell_items

For table structures with RTL content, sort_rtl_cell_items groups items by baseline, sorts each horizontal band right‑to‑left, then restores embedded LTR runs to ensure correct reading order.

// src/text_utils.rs (lines 162-185)
pub fn sort_rtl_cell_items(items: &mut [TextItem]) {
    // Group by baseline (y-coordinate)
    let mut bands: HashMap<OrderedFloat<f64>, Vec<&mut TextItem>> = HashMap::new();
    
    for item in items.iter_mut() {
        let y_key = OrderedFloat(item.bbox.y1);
        bands.entry(y_key).or_default().push(item);
    }
    
    // Sort each band right-to-left
    for (_, band) in &mut bands {
        band.sort_by(|a, b| b.bbox.x1.partial_cmp(&a.bbox.x1).unwrap());
        
        // Restore LTR runs within RTL context
        restore_ltr_runs(band);
    }
}

Visual‑Order Correction: fix_visual_order_rtl

Many PDFs—especially scanned documents—store RTL glyphs in visual order (left‑to‑right as painted) rather than logical order. fix_visual_order_rtl detects this convention by counting geometric votes and reverses affected runs when visual order dominates.

// src/text_utils.rs (lines 556-584)
pub fn fix_visual_order_rtl(
    items: &mut [TextItem],
    visual_candidates: &[usize],
    logical_votes: usize,
) {
    // Count geometric direction votes from item positions
    let mut visual_votes = 0;
    for window in visual_candidates.windows(2) {
        let i = window[0];
        let j = window[1];
        if items[i].bbox.x1 < items[j].bbox.x1 {
            visual_votes += 1; // Rightward progression suggests visual order
        } else {
            logical_votes += 1; // Leftward progression suggests logical order
        }
    }
    
    // When visual order dominates, reverse each RTL item's text
    if visual_votes > logical_votes {
        for &idx in visual_candidates {
            if is_rtl_text(std::iter::once(&items[idx].text)) {
                items[idx].text = reverse_visual_arabic(&items[idx].text);
            }
        }
    }
}

The companion function reverse_visual_arabic handles Arabic‑specific shaping concerns during reversal.

Gap‑Ratio Calculations: CJK Exclusion from Heuristics

To prevent CJK text from distorting spacing thresholds used for Latin‑based content, collect_gap_ratios explicitly skips any item pair where either boundary character is CJK.

// src/text_utils.rs (lines 438-449)
fn collect_gap_ratios(items: &[TextItem]) -> Vec<f64> {
    let mut ratios = Vec::new();
    
    for window in items.windows(2) {
        let prev = &window[0];
        let curr = &window[1];
        
        // Skip if CJK present at boundaries
        if prev.text.ends_with(|c| is_cjk_char(c))
            || curr.text.starts_with(|c| is_cjk_char(c)) {
            continue;
        }
        
        let gap = curr.bbox.x0 - prev.bbox.x1;
        let font_size = prev.font_size.max(curr.font_size);
        ratios.push(gap / font_size);
    }
    
    ratios
}

This exclusion ensures that Otsu thresholding for word‑boundary detection remains accurate for non‑CJK scripts.

Complete Usage Example

use pdf_inspector::text_utils::{
    is_cjk_char, is_rtl_char, is_rtl_text, 
    sort_line_items, fix_visual_order_rtl
};

/// Detect script of a single character
fn classify_character(ch: char) -> &'static str {
    if is_cjk_char(ch) {
        "CJK"
    } else if is_rtl_char(ch) {
        "RTL"
    } else {
        "Other"
    }
}

/// Process a line of extracted text items
fn process_line(items: &mut [TextItem]) {
    // Determine if entire line is RTL
    let rtl = is_rtl_text(items.iter().map(|i| &i.text));
    println!("Line direction: {}", if rtl { "RTL" } else { "LTR" });
    
    // Apply appropriate sorting
    sort_line_items(items);
    
    // After initial extraction, check for visual-order PDF artifacts
    let candidates: Vec<usize> = items.iter()
        .enumerate()
        .filter(|(_, item)| is_rtl_text(std::iter::once(&item.text)))
        .map(|(i, _)| i)
        .collect();
    
    fix_visual_order_rtl(items, &candidates, 0);
}

Key Files in the CJK and RTL Pipeline

File	Responsibility
src/text_utils.rs	Core classification (is_cjk_char, is_rtl_char), direction analysis (is_rtl_text), sorting (sort_line_items, sort_rtl_cell_items), visual correction (fix_visual_order_rtl), and CJK‑aware gap handling
src/markdown/preprocess.rs	Invokes should_join_items with CJK fast‑path for word boundary detection
src/extractor/layout.rs	Orchestrates line extraction and applies RTL‑aware sorting per line
src/markdown/convert.rs	Final Markdown output preserving RTL directionality and CJK spacing

Summary

• Detection relies on explicit Unicode ranges: is_cjk_char and is_rtl_char in src/text_utils.rs identify scripts without external dependencies.

• Direction is determined statistically: is_rtl_text counts strong RTL letters while ignoring CJK and combining marks, preventing false positives.

• CJK receives special spacing treatment: The fast‑path in should_join_items always joins adjacent CJK items, eliminating spurious word breaks.

• RTL supports both logical and visual conventions: sort_line_items handles logical‑order PDFs; fix_visual_order_rtl corrects visual‑order artifacts common in scanned documents.

• Table extraction preserves reading order: sort_rtl_cell_items groups by baseline and handles embedded LTR runs within RTL columns.

Frequently Asked Questions

How does pdf‑inspector distinguish CJK from other Unicode scripts?

pdf‑inspector uses the is_cjk_char function in src/text_utils.rs to check if a character falls within defined Unicode blocks for Hangul, Hiragana, Katakana, and CJK ideographs. This explicit range checking operates without external libraries and runs in O(1) per character.

Why does RTL detection ignore CJK characters when counting votes?

The is_rtl_text function deliberately excludes CJK and combining marks from its direction tally because these characters are directionally neutral in Unicode's bidirectional algorithm. Counting them would skew the RTL‑versus‑LTR ratio, causing incorrect line direction classification in mixed documents.

What happens when a PDF stores Arabic text in visual order?

Many PDF generators—especially for scanned documents—render RTL glyphs left‑to‑right as they appear on the page. fix_visual_order_rtl detects this pattern by comparing geometric progression against logical expectations, then reverses affected runs using reverse_visual_arabic to restore proper reading sequence.

Does CJK handling affect performance of Latin text extraction?

No. The CJK fast‑path in should_join_items executes a single boundary character check before falling back to standard gap‑ratio heuristics. collect_gap_ratios explicitly excludes CJK pairs from threshold calculations, ensuring Latin text processing remains unaffected by CJK document presence.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}