Does pdf-inspector Support CJK and RTL Text Extraction? A Technical Deep Dive

Yes, pdf-inspector fully supports CJK and RTL text extraction through dedicated Unicode detection functions, direction-aware layout heuristics, and Arabic normalization pipelines.

The firecrawl/pdf-inspector repository handles complex PDF text extraction for East Asian and right-to-left scripts by leveraging specific detection utilities and heuristics in its Rust codebase. The implementation preserves reading order and linguistic integrity across diverse writing systems without requiring manual configuration.

CJK and RTL Character Detection

The foundation of pdf-inspector's multilingual support lies in src/text_utils.rs, which provides precise Unicode-based character classification for complex scripts.

CJK Character Classification

The is_cjk_char function identifies East Asian scripts by checking Unicode ranges for Hangul, Hiragana, Katakana, CJK Unified Ideographs, CJK Compatibility Ideographs, and half-width/full-width forms. This detection enables the extractor to apply CJK-specific spacing rules during text assembly, ensuring that mixed-language documents maintain proper character boundaries.

RTL Character Identification

For right-to-left scripts, is_rtl_char recognizes Hebrew, Arabic, Syriac, Thaana, NKo, Samaritan, Mandaic, Arabic-Extended-A, and various Arabic presentation-form blocks. The is_rtl_text function extends this by scanning sequences of strings to determine overall text directionality, comparing RTL character counts against LTR alphabetic characters to decide column ordering and word-joining strategies.

Direction-Aware Layout Processing

During the extraction pipeline, pdf-inspector applies script-specific heuristics to maintain proper text flow and spacing when reconstructing lines from PDF content streams.

RTL Text Heuristics and Joining

The should_join_items function in src/text_utils.rs applies different thresholds for RTL runs during the line-joining phase. This prevents incorrect word splitting and ensures that RTL text flows naturally in the extracted output. The layout engine in src/extractor/layout.rs consumes these heuristics to arrange text lines in the correct reading order.

CJK Space Insertion Logic

The same should_join_items utility explicitly skips space-insertion for CJK characters, recognizing that East Asian scripts typically do not require whitespace between words or characters. This prevents the insertion of unwanted spaces that would corrupt the semantic structure of Chinese, Japanese, or Korean text during the reconstruction process.

Arabic Text Normalization

pdf-inspector includes specialized handling for Arabic presentation forms and visual ordering that goes beyond simple character extraction.

The expand_ligatures function normalizes Arabic presentation-form characters to their standard equivalents, while reverse_visual_arabic converts visually-ordered Arabic text to logical order. This combination ensures that extracted Arabic text reads correctly regardless of how the PDF internally stores the character data, handling ligatures and directional marks appropriately.

Using the pdf-inspector API for Multilingual PDFs

The public API exposes these capabilities through the process_pdf_with_options function in src/lib.rs. The extraction pipeline automatically applies CJK and RTL detection without requiring additional configuration flags.

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::options::PdfOptions;

fn main() {
    // Load a PDF that contains mixed English, Chinese, and Arabic text.
    let opts = PdfOptions::default();
    let result = process_pdf_with_options("sample.pdf", &opts).unwrap();

    // The extracted Markdown preserves the original direction.
    println!("{}", result.markdown);
}

Internally, this API orchestrates the following sequence through the Rust modules:

1. Text decoding via decode_text_string to handle PDF encoding quirks.
2. Ligature expansion and Arabic normalization via expand_ligatures.
3. Character detection using is_cjk_char and is_rtl_char to classify script types.
4. Smart joining via should_join_items, which applies CJK and RTL-specific logic to reconstruct words and lines correctly.

The final Markdown conversion in src/markdown/convert.rs preserves RTL flow while generating the structured output.

Summary

• pdf-inspector detects CJK characters using is_cjk_char in src/text_utils.rs to identify Hangul, Hiragana, Katakana, and CJK Ideographs.
• RTL scripts are recognized by is_rtl_char and is_rtl_text, which handle Hebrew, Arabic, Syriac, and related Unicode blocks.
• The should_join_items function applies script-specific spacing rules, skipping spaces for CJK and using RTL-aware thresholds for word joining.
• Arabic text undergoes normalization through expand_ligatures and reverse_visual_arabic to convert presentation forms and visual ordering to logical order.
• Unit tests including test_cjk_joining and reverse_visual_arabic_* verify correct handling across writing systems.

Frequently Asked Questions

How does pdf-inspector identify CJK characters in a PDF?

pdf-inspector uses the is_cjk_char utility in src/text_utils.rs to classify characters across Hangul, Hiragana, Katakana, CJK Unified Ideographs, CJK Compatibility Ideographs, and half-width/full-width forms. This detection triggers CJK-specific spacing rules during text assembly in the layout engine.

What RTL scripts does pdf-inspector support?

The is_rtl_char function recognizes Hebrew, Arabic, Syriac, Thaana, NKo, Samaritan, Mandaic, Arabic-Extended-A, and Arabic presentation-form blocks. The is_rtl_text heuristic then evaluates text sequences in src/text_utils.rs to determine overall directionality for proper column ordering and line layout.

How does pdf-inspector handle Arabic text normalization?

The extractor normalizes Arabic presentation-form characters through expand_ligatures and converts visual-order Arabic to logical order using reverse_visual_arabic, both defined in src/text_utils.rs. This ensures extracted Arabic text maintains correct reading order regardless of internal PDF storage methods.

Can pdf-inspector extract mixed-language documents automatically?

Yes, the process_pdf_with_options API in src/lib.rs automatically handles PDFs containing mixed English, CJK, and RTL text. The detection and normalization pipelines process each text segment according to its identified script type without manual intervention, preserving the original document's linguistic structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}