How PDF‑Inspector's NFKC Normalization Performs Ligature Expansion

PDF‑Inspector performs ligature expansion through expand_ligatures in src/text_utils.rs, which applies conditional NFKC normalization only when Arabic presentation forms are detected, followed by explicit matching for Latin ligatures that NFKC misses.

PDF‑Inspector, the Rust‑based PDF text extraction engine developed by Firecrawl, needs to convert typographic ligatures into their component characters for downstream NLP tasks. Unlike simpler tools that apply Unicode normalization blindly, the library uses a sophisticated three‑stage pipeline that preserves spacing semantics while handling both Latin and Arabic text correctly.

The expand_ligatures Function Architecture

The core implementation resides in src/text_utils.rs (lines 39‑96). This single routine combines sanitization, conditional normalization, manual ligature expansion, and Arabic visual‑order correction.

Step 1: Sanitization of Control Characters

Before any normalization occurs, the function strips low‑control characters that commonly leak through from malformed PDFs:

// Lines 39-48: Remove control chars except \n, \r, \t
let cleaned: String = text
    .chars()
    .filter(|c| !is_control_except_whitespace(*c))
    .collect();

This prevents stray bytes from interfering with the Unicode normalization crate's expectations.

Conditional NFKC Normalization

The NFKC normalization in PDF‑Inspector is deliberately conditional rather than universal. This design choice protects critical spacing characters.

Why Conditionality Matters

Applying nfkc() globally would transform non‑breaking spaces (U+00A0) into regular spaces, breaking the downstream spacing heuristics that rely on NBSP distinctions. The code therefore checks for a specific signal before normalizing.

Arabic Presentation Form Detection

The detection logic scans for Arabic presentation‑form characters (U+FB50‑FDFF, U+FE70‑FEFF):

// Lines 51-53
let had_presentation_forms = text.chars().any(is_arabic_presentation_form);

These code points indicate that the PDF stored Arabic glyphs in visual (LTR) order—a common PDF generator behavior that also requires NFKC to map presentation forms back to base Arabic code points.

The Normalization Trigger

When presentation forms are detected, NFKC runs via the unicode‑normalization crate (lines 60‑62):

if had_presentation_forms {
    text = text.nfkc().collect::<String>();
}

This single call simultaneously:

• Decomposes Latin ligatures (e.g., "fi" → "fi")
• Converts Arabic presentation forms to base letters (e.g., "ﻼ" → "لا")

Explicit Ligature Expansion for Edge Cases

Even after NFKC processing, some fonts expose ligatures through private‑use or custom ToUnicode mappings that escape standard normalization. The function handles these with an explicit match block (lines 69‑76):

'\u{FB00}' => result.push_str("ff"),
'\u{FB01}' => result.push_str("fi"),
'\u{FB02}' => result.push_str("fl"),
'\u{FB03}' => result.push_str("ffi"),
'\u{FB04}' => result.push_str("ffl"),
'\u{FB05}' | '\u{FB06}' => result.push_str("st"),

This fallback mechanism covers the Latin Presentation Forms‑A block (U+FB00‑U+FB4B) that NFKC should theoretically handle but sometimes misses due to PDF‑specific encoding quirks.

Complete Latin Ligature Coverage

The explicit mapping handles six common ligature code points:

Code Point	Expansion	Description
U+FB00	"ff"	Double f
U+FB01	"fi"	fi ligature
U+FB02	"fl"	fl ligature
U+FB03	"ffi"	ffi ligature
U+FB04	"ffl"	ffl ligature
U+FB05/U+FB06	"st"	Long s variants

Arabic Visual‑Order Correction

When Arabic presentation forms triggered NFKC normalization, the resulting text remains in visual LTR order. The routine restores logical RTL order through reverse_visual_arabic (lines 94‑96):

if had_presentation_forms {
    result = reverse_visual_arabic(&result);
}

This helper correctly handles:

• Mixed LTR runs within Arabic text
• Numeric separators
• Bidirectional boundary conditions

Invoking the Ligature Expander in Extraction Pipelines

PDF‑Inspector calls expand_ligatures automatically at strategic points in the extraction flow:

Content Stream Processing

In src/extractor/content_stream.rs (line 595), the function processes text immediately after decoding:

// After TJ/Tj operator processing
let expanded = expand_ligatures(&text_fragment);

XObject Text Extraction

In src/extractor/xobjects.rs, the same normalization applies to embedded text objects (lines 676 and 880), ensuring consistent handling across:

• Form XObjects
• Image alternate text
• Embedded font subsets

Practical Usage Example

You can invoke the ligature expander directly in custom tooling:

use pdf_inspector::text_utils::expand_ligatures;

fn main() {
    // Contains: Latin ligature, Arabic presentation forms, 
    // soft hyphen (U+00AD), zero-width space (U+200B)
    let raw = "first ﺍﻠﻤﺘﺤﻞ\u{00AD}\u{200B}";
    let cleaned = expand_ligatures(raw);
    println!("{}", cleaned);
    // Output: "first المتحل"
}

The function's sanitization step removes the soft hyphen and zero‑width space, NFKC handles the Arabic forms, and the explicit mapping expands "fi" to "fi".

Design Rationale and Trade‑offs

The conditional NFKC normalization in PDF‑Inspector reflects careful engineering around real‑world PDF constraints:

• Precision over convenience: Skipping NFKC for Latin‑only text preserves NBSP semantics that spacing heuristics depend on
• Unicode compliance: Arabic presentation forms correctly map to base characters only when needed
• Defensive programming: Explicit ligature matching catches edge cases the normalization crate misses
• Bidirectional integrity: Visual‑to‑logical conversion occurs only for the text that actually requires it

Summary

• expand_ligatures in src/text_utils.rs (lines 39‑96) is the central NFKC normalization and ligature expansion routine
• Conditional execution: NFKC runs only when Arabic presentation forms (U+FB50‑FDFF, U+FE70‑FEFF) are detected, protecting NBSP and other spacing‑sensitive characters
• Dual expansion strategy: Unicode normalization handles standard decompositions; explicit match arms cover private‑use ligatures
• Arabic handling: Visual‑order text receives NFKC normalization followed by reverse_visual_arabic for correct RTL output
• Pipeline integration: Called automatically from content_stream.rs (line 595) and xobjects.rs (lines 676, 880)

Frequently Asked Questions

What is NFKC normalization and why does PDF‑Inspector use it conditionally?

NFKC (Normalization Form KC) applies compatibility decomposition followed by canonical composition. It expands ligatures and converts stylistic variants to base forms. PDF‑Inspector uses it conditionally because global NFKC would convert non‑breaking spaces to regular spaces, breaking the spacing heuristics that detect word boundaries in extracted text. The code only triggers NFKC when Arabic presentation forms are present, since those always require normalization for correct extraction.

Which ligatures does PDF‑Inspector explicitly expand beyond NFKC?

The explicit match block in src/text_utils.rs (lines 69‑76) expands six Latin Presentation Forms‑A code points: U+FB00 ("ff"), U+FB01 ("fi"), U+FB02 ("fl"), U+FB03 ("ffi"), U+FB04 ("ffl"), and U+FB05/U+FB06 ("st"). These catch edge cases where PDF fonts use custom ToUnicode mappings that bypass standard Unicode normalization.

Why does Arabic text need special handling in ligature expansion?

PDF generators often store Arabic text in visual (LTR) order using presentation‑form code points rather than logical RTL order with base Arabic characters. PDF‑Inspector detects these presentation forms to trigger NFKC normalization, which maps visual forms back to base code points. It then calls reverse_visual_arabic to restore the correct logical reading order for downstream processing.

How can I use PDF‑Inspector's ligature expansion in my own code?

Import expand_ligatures from pdf_inspector::text_utils and pass any string potentially containing ligatures. The function returns a cleaned String with expanded ligatures, removed control characters, and corrected Arabic ordering. It is called automatically during PDF extraction, but available for standalone use when preprocessing text from other sources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}