# Does pdf-inspector Support CJK and RTL Text Extraction? A Technical Deep Dive

> pdf-inspector excels at CJK and RTL text extraction. Discover how its Unicode detection and layout heuristics ensure accurate multilingual data extraction for your projects.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-08-13

---

**Yes, pdf-inspector fully supports CJK and RTL text extraction through dedicated Unicode detection functions, direction-aware layout heuristics, and Arabic normalization pipelines.**

The `firecrawl/pdf-inspector` repository handles complex PDF text extraction for East Asian and right-to-left scripts by leveraging specific detection utilities and heuristics in its Rust codebase. The implementation preserves reading order and linguistic integrity across diverse writing systems without requiring manual configuration.

## CJK and RTL Character Detection

The foundation of pdf-inspector's multilingual support lies in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs), which provides precise Unicode-based character classification for complex scripts.

### CJK Character Classification

The **`is_cjk_char`** function identifies East Asian scripts by checking Unicode ranges for Hangul, Hiragana, Katakana, CJK Unified Ideographs, CJK Compatibility Ideographs, and half-width/full-width forms. This detection enables the extractor to apply CJK-specific spacing rules during text assembly, ensuring that mixed-language documents maintain proper character boundaries.

### RTL Character Identification

For right-to-left scripts, **`is_rtl_char`** recognizes Hebrew, Arabic, Syriac, Thaana, NKo, Samaritan, Mandaic, Arabic-Extended-A, and various Arabic presentation-form blocks. The **`is_rtl_text`** function extends this by scanning sequences of strings to determine overall text directionality, comparing RTL character counts against LTR alphabetic characters to decide column ordering and word-joining strategies.

## Direction-Aware Layout Processing

During the extraction pipeline, pdf-inspector applies script-specific heuristics to maintain proper text flow and spacing when reconstructing lines from PDF content streams.

### RTL Text Heuristics and Joining

The **`should_join_items`** function in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) applies different thresholds for RTL runs during the line-joining phase. This prevents incorrect word splitting and ensures that RTL text flows naturally in the extracted output. The layout engine in [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) consumes these heuristics to arrange text lines in the correct reading order.

### CJK Space Insertion Logic

The same `should_join_items` utility explicitly **skips space-insertion for CJK characters**, recognizing that East Asian scripts typically do not require whitespace between words or characters. This prevents the insertion of unwanted spaces that would corrupt the semantic structure of Chinese, Japanese, or Korean text during the reconstruction process.

## Arabic Text Normalization

pdf-inspector includes specialized handling for Arabic presentation forms and visual ordering that goes beyond simple character extraction.

The **`expand_ligatures`** function normalizes Arabic presentation-form characters to their standard equivalents, while **`reverse_visual_arabic`** converts visually-ordered Arabic text to logical order. This combination ensures that extracted Arabic text reads correctly regardless of how the PDF internally stores the character data, handling ligatures and directional marks appropriately.

## Using the pdf-inspector API for Multilingual PDFs

The public API exposes these capabilities through the **`process_pdf_with_options`** function in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs). The extraction pipeline automatically applies CJK and RTL detection without requiring additional configuration flags.

```rust
use pdf_inspector::process_pdf_with_options;
use pdf_inspector::options::PdfOptions;

fn main() {
    // Load a PDF that contains mixed English, Chinese, and Arabic text.
    let opts = PdfOptions::default();
    let result = process_pdf_with_options("sample.pdf", &opts).unwrap();

    // The extracted Markdown preserves the original direction.
    println!("{}", result.markdown);
}

```

Internally, this API orchestrates the following sequence through the Rust modules:

1. **Text decoding** via `decode_text_string` to handle PDF encoding quirks.
2. **Ligature expansion** and Arabic normalization via `expand_ligatures`.
3. **Character detection** using `is_cjk_char` and `is_rtl_char` to classify script types.
4. **Smart joining** via `should_join_items`, which applies CJK and RTL-specific logic to reconstruct words and lines correctly.

The final Markdown conversion in [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) preserves RTL flow while generating the structured output.

## Summary

- pdf-inspector detects CJK characters using `is_cjk_char` in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) to identify Hangul, Hiragana, Katakana, and CJK Ideographs.
- RTL scripts are recognized by `is_rtl_char` and `is_rtl_text`, which handle Hebrew, Arabic, Syriac, and related Unicode blocks.
- The `should_join_items` function applies script-specific spacing rules, skipping spaces for CJK and using RTL-aware thresholds for word joining.
- Arabic text undergoes normalization through `expand_ligatures` and `reverse_visual_arabic` to convert presentation forms and visual ordering to logical order.
- Unit tests including `test_cjk_joining` and `reverse_visual_arabic_*` verify correct handling across writing systems.

## Frequently Asked Questions

### How does pdf-inspector identify CJK characters in a PDF?

pdf-inspector uses the `is_cjk_char` utility in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) to classify characters across Hangul, Hiragana, Katakana, CJK Unified Ideographs, CJK Compatibility Ideographs, and half-width/full-width forms. This detection triggers CJK-specific spacing rules during text assembly in the layout engine.

### What RTL scripts does pdf-inspector support?

The `is_rtl_char` function recognizes Hebrew, Arabic, Syriac, Thaana, NKo, Samaritan, Mandaic, Arabic-Extended-A, and Arabic presentation-form blocks. The `is_rtl_text` heuristic then evaluates text sequences in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) to determine overall directionality for proper column ordering and line layout.

### How does pdf-inspector handle Arabic text normalization?

The extractor normalizes Arabic presentation-form characters through `expand_ligatures` and converts visual-order Arabic to logical order using `reverse_visual_arabic`, both defined in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs). This ensures extracted Arabic text maintains correct reading order regardless of internal PDF storage methods.

### Can pdf-inspector extract mixed-language documents automatically?

Yes, the `process_pdf_with_options` API in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) automatically handles PDFs containing mixed English, CJK, and RTL text. The detection and normalization pipelines process each text segment according to its identified script type without manual intervention, preserving the original document's linguistic structure.