How pdf-inspector Handles CJK and RTL Text: Ligature Expansion and NFKC Normalization Explained

pdf-inspector processes CJK and RTL text through character classification, NFKC Unicode normalization, ligature expansion, and direction-aware layout sorting to produce semantically correct, searchable output.

The firecrawl/pdf-inspector Rust library implements sophisticated text processing for non-Latin scripts. Its pipeline correctly identifies Chinese, Japanese, Korean (CJK), and right-to-left (RTL) scripts, applies targeted Unicode normalization, and preserves logical reading order—critical for accurate PDF-to-Markdown conversion.

Character Classification for CJK and RTL Scripts

pdf-inspector's detection system uses Unicode range matching to categorize characters before any transformation occurs.

CJK Detection with is_cjk_char

In src/text_utils.rs (lines 87-102), the is_cjk_char function identifies CJK scripts by checking against multiple Unicode blocks:

• Hangul Jamo and Hangul syllables
• Hiragana and Katakana
• CJK Unified Ideographs
• CJK Compatibility Ideographs
• Half-width forms (Japanese/Chinese punctuation)

use pdf_inspector::text_utils::is_cjk_char;

fn demo_cjk() {
    let zh = '中';      // CJK Unified Ideograph
    let ja = 'ひ';      // Hiragana
    let ko = '한';      // Hangul syllable
    
    println!("Chinese: {}", is_cjk_char(zh)); // true
    println!("Japanese: {}", is_cjk_char(ja)); // true
    println!("Korean: {}", is_cjk_char(ko)); // true
}

RTL Detection with is_rtl_char

The companion function is_rtl_char (lines 104-118) recognizes:

• Hebrew
• Arabic (including all presentation forms)
• Syriac
• Thaana
• NKo, Samaritan, Mandaic

use pdf_inspector::text_utils::is_rtl_char;

fn demo_rtl() {
    let he = 'ש';       // Hebrew
    let ar = 'م';       // Arabic
    let sy = 'ܐ';       // Syriac
    
    println!("Hebrew: {}", is_rtl_char(he)); // true
    println!("Arabic: {}", is_rtl_char(ar)); // true
    println!("Syriac: {}", is_rtl_char(sy)); // true
}

These classification helpers enable downstream decisions about word boundaries, spacing, and text direction.

Ligature Expansion and NFKC Normalization

The expand_ligatures function (lines 93-60 in src/text_utils.rs) handles Unicode standardization through a multi-stage pipeline.

Processing Pipeline

1. Strips control characters – removes invisible formatting codes
2. Detects Arabic Presentation Forms – checks for U+FB00–U+FDFF range
3. Conditionally applies NFKC – only when presentation forms are present, avoiding unnecessary computation
4. Expands Latin ligatures – converts ff → ff, fi → fi, fl → fl, etc.
5. Removes invisible characters – deletes zero-width joiners and non-joiners
6. Normalizes whitespace – collapses multiple spaces

use pdf_inspector::text_utils::expand_ligatures;

fn demo_normalization() {
    // Latin ligature expansion
    let with_ligature = "firstly";  // U+FB01 ligature
    println!("{}", expand_ligatures(with_ligature)); 
    // Output: "firstly"
    
    // Soft hyphen removal
    let with_hyphen = "invis\u{00AD}ible";
    println!("{}", expand_ligatures(with_hyphen));
    // Output: "invisible"
}

Arabic Presentation Forms and Visual-to-Logical Reordering

When Arabic Presentation Forms are detected, pdf-inspector performs two critical operations:

1. NFKC normalization – converts presentation forms (like U+FEE1, U+FEF3) to their base Arabic characters
2. Visual-to-logical reordering – calls reverse_visual_arabic (lines 62-70, 71-110) to restore correct reading order

use pdf_inspector::text_utils::{expand_ligatures, reverse_visual_arabic};

fn demo_arabic() {
    // Mixed visual-order Arabic with Latin ligature
    let raw = "\u{FB01}rst\u{00AD}ly \u{FEE1}\u{FEF3}";
    // first­ly (with ligature and soft-hyphen) + visual-order Arabic
    
    let cleaned = expand_ligatures(raw);
    // NFKC applied, ligature expanded, then reordered to logical RTL
    
    println!("cleaned: {}", cleaned);
    // Output: "firstly" followed by base Arabic in logical order
}

The reverse_visual_arabic function handles both pure RTL runs and mixed LTR/RTL bidirectional text.

Direction-Aware Layout and Line Building

RTL-Aware Sorting with sort_line_items

At lines 45-52 in src/text_utils.rs, sort_line_items determines text direction using is_rtl_text (which aggregates characters via is_rtl_char). The function then orders items either left-to-right or right-to-left accordingly:

// Called during Markdown conversion in src/markdown/convert.rs
pub fn sort_line_items(items: &mut [TextItem]) {
    let rtl = is_rtl_text(items);
    if rtl {
        items.sort_by(|a, b| b.bounds.x().partial_cmp(&a.bounds.x()).unwrap());
    } else {
        items.sort_by(|a, b| a.bounds.x().partial_cmp(&b.bounds.x()).unwrap());
    }
}

This ensures that extracted RTL text maintains correct logical order regardless of how the PDF stores positional data.

CJK-Aware Gap Collection and Joining

pdf-inspector's spacing logic special-cases CJK scripts at two points:

collect_gap_ratios (lines 69-81) – skips gap ratio calculation for CJK pairs entirely:

// Pseudocode from source analysis
if is_cjk_char(left) || is_cjk_char(right) {
    continue; // Skip gap collection for CJK
}

should_join_items (lines 103-108, fallback at 398-410) – applies CJK-specific joining rules:

• CJK items are always joined when gap < 0.8 × character width
• No space insertion between CJK characters (contextually appropriate, as CJK scripts don't use inter-word spaces)

// Fallback branch for CJK items
if is_cjk_char(left_last) || is_cjk_char(right_first) {
    return gap < char_width * 0.8; // Always join close CJK fragments
}

This prevents incorrect space insertion in Chinese, Japanese, or Korean text while preserving normal spacing rules for Latin scripts.

Complete Processing Flow

pdf-inspector's CJK and RTL handling operates across five stages:

Stage	Location	Key Function
Extraction	src/text_utils.rs	is_cjk_char, is_rtl_char classify characters
Preprocessing	src/markdown/preprocess.rs	expand_ligatures normalizes Unicode
Normalization	src/text_utils.rs	reverse_visual_arabic fixes RTL order
Line building	src/markdown/convert.rs	sort_line_items arranges by direction
Spacing	src/text_utils.rs	should_join_items applies CJK/RTL rules

The output is semantically correct Markdown with searchable text, proper logical order for RTL scripts, and appropriate word boundaries for all supported languages.

Summary

• Character classification via is_cjk_char and is_rtl_char enables script-specific processing decisions throughout the pipeline
• NFKC normalization is applied conditionally when Arabic Presentation Forms are detected, converting presentation forms to base characters
• Ligature expansion converts typographic ligatures (fi, ff, fl) to their component characters for searchability
• Visual-to-logical reordering via reverse_visual_arabic restores correct reading order for RTL text in visually-ordered PDFs
• CJK-aware spacing prevents incorrect word breaks in Chinese, Japanese, and Korean by skipping gap heuristics and always joining proximal CJK items
• Direction-aware sorting in sort_line_items ensures RTL text flows right-to-left in the final output

Frequently Asked Questions

When does pdf-inspector apply NFKC normalization?

NFKC normalization is applied only when Arabic Presentation Forms (U+FB00–U+FDFF) are detected in the input. This conditional approach, implemented in expand_ligatures at src/text_utils.rs lines 93-60, avoids unnecessary computation for pure Latin text while ensuring Arabic compatibility characters are properly normalized to their base forms.

Why does pdf-inspector skip gap ratio collection for CJK text?

CJK scripts do not use spaces between words, so gap-based heuristics for word boundary detection are inappropriate. The collect_gap_ratios function (lines 69-81) explicitly excludes CJK pairs from gap measurement, preventing the spacing algorithms from incorrectly splitting CJK text into words based on visual separation.

How does pdf-inspector handle mixed LTR and RTL text on the same line?

The reverse_visual_arabic function (lines 62-70, 71-110) handles bidirectional text by identifying pure RTL runs and mixed LTR/RTL segments. It reverses visual-order Arabic while preserving the relative position of embedded LTR text (like numbers or Latin words), producing logically-ordered output that matches standard Unicode bidirectional algorithm expectations.

What ligatures does pdf-inspector expand beyond Latin typographic forms?

In addition to common Latin ligatures (ff, fi, fl, ffi, ffl), the expand_ligatures function removes soft hyphens (\u{00AD}), zero-width joiners, and other invisible Unicode characters. The expansion is performed after NFKC normalization, ensuring that all text is both searchable and semantically correct before final Markdown generation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}