# How PDF‑Inspector's NFKC Normalization Performs Ligature Expansion

> Discover how PDF-Inspector expands ligatures using NFKC normalization and explicit Latin ligature matching. Learn more about this essential text processing feature.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-09-02

---

**PDF‑Inspector performs ligature expansion through `expand_ligatures` in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs), which applies conditional NFKC normalization only when Arabic presentation forms are detected, followed by explicit matching for Latin ligatures that NFKC misses.**

PDF‑Inspector, the Rust‑based PDF text extraction engine developed by Firecrawl, needs to convert typographic ligatures into their component characters for downstream NLP tasks. Unlike simpler tools that apply Unicode normalization blindly, the library uses a sophisticated three‑stage pipeline that preserves spacing semantics while handling both Latin and Arabic text correctly.

## The `expand_ligatures` Function Architecture

The core implementation resides in **[`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs)** (lines 39‑96). This single routine combines sanitization, conditional normalization, manual ligature expansion, and Arabic visual‑order correction.

### Step 1: Sanitization of Control Characters

Before any normalization occurs, the function strips low‑control characters that commonly leak through from malformed PDFs:

```rust
// Lines 39-48: Remove control chars except \n, \r, \t
let cleaned: String = text
    .chars()
    .filter(|c| !is_control_except_whitespace(*c))
    .collect();

```

This prevents stray bytes from interfering with the Unicode normalization crate's expectations.

## Conditional NFKC Normalization

The **NFKC normalization** in PDF‑Inspector is deliberately **conditional** rather than universal. This design choice protects critical spacing characters.

### Why Conditionality Matters

Applying `nfkc()` globally would transform non‑breaking spaces (U+00A0) into regular spaces, breaking the downstream spacing heuristics that rely on NBSP distinctions. The code therefore checks for a specific signal before normalizing.

### Arabic Presentation Form Detection

The detection logic scans for **Arabic presentation‑form characters** (U+FB50‑FDFF, U+FE70‑FEFF):

```rust
// Lines 51-53
let had_presentation_forms = text.chars().any(is_arabic_presentation_form);

```

These code points indicate that the PDF stored Arabic glyphs in **visual (LTR) order**—a common PDF generator behavior that also requires NFKC to map presentation forms back to base Arabic code points.

### The Normalization Trigger

When presentation forms are detected, NFKC runs via the `unicode‑normalization` crate (lines 60‑62):

```rust
if had_presentation_forms {
    text = text.nfkc().collect::<String>();
}

```

This single call simultaneously:
- Decomposes Latin ligatures (e.g., "fi" → "fi")
- Converts Arabic presentation forms to base letters (e.g., "ﻼ" → "لا")

## Explicit Ligature Expansion for Edge Cases

Even after NFKC processing, some fonts expose ligatures through **private‑use or custom ToUnicode mappings** that escape standard normalization. The function handles these with an explicit match block (lines 69‑76):

```rust
'\u{FB00}' => result.push_str("ff"),
'\u{FB01}' => result.push_str("fi"),
'\u{FB02}' => result.push_str("fl"),
'\u{FB03}' => result.push_str("ffi"),
'\u{FB04}' => result.push_str("ffl"),
'\u{FB05}' | '\u{FB06}' => result.push_str("st"),

```

This fallback mechanism covers the **Latin Presentation Forms‑A** block (U+FB00‑U+FB4B) that NFKC should theoretically handle but sometimes misses due to PDF‑specific encoding quirks.

### Complete Latin Ligature Coverage

The explicit mapping handles six common ligature code points:

| Code Point | Expansion | Description |
|------------|-----------|-------------|
| U+FB00 | "ff" | Double f |
| U+FB01 | "fi" | fi ligature |
| U+FB02 | "fl" | fl ligature |
| U+FB03 | "ffi" | ffi ligature |
| U+FB04 | "ffl" | ffl ligature |
| U+FB05/U+FB06 | "st" | Long s variants |

## Arabic Visual‑Order Correction

When Arabic presentation forms triggered NFKC normalization, the resulting text remains in **visual LTR order**. The routine restores logical RTL order through `reverse_visual_arabic` (lines 94‑96):

```rust
if had_presentation_forms {
    result = reverse_visual_arabic(&result);
}

```

This helper correctly handles:
- Mixed LTR runs within Arabic text
- Numeric separators
- Bidirectional boundary conditions

## Invoking the Ligature Expander in Extraction Pipelines

PDF‑Inspector calls `expand_ligatures` automatically at strategic points in the extraction flow:

### Content Stream Processing

In **[`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs)** (line 595), the function processes text immediately after decoding:

```rust
// After TJ/Tj operator processing
let expanded = expand_ligatures(&text_fragment);

```

### XObject Text Extraction

In **[`src/extractor/xobjects.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/xobjects.rs)**, the same normalization applies to embedded text objects (lines 676 and 880), ensuring consistent handling across:

- Form XObjects
- Image alternate text
- Embedded font subsets

## Practical Usage Example

You can invoke the ligature expander directly in custom tooling:

```rust
use pdf_inspector::text_utils::expand_ligatures;

fn main() {
    // Contains: Latin ligature, Arabic presentation forms, 
    // soft hyphen (U+00AD), zero-width space (U+200B)
    let raw = "first ﺍﻠﻤﺘﺤﻞ\u{00AD}\u{200B}";
    let cleaned = expand_ligatures(raw);
    println!("{}", cleaned);
    // Output: "first المتحل"
}

```

The function's sanitization step removes the soft hyphen and zero‑width space, NFKC handles the Arabic forms, and the explicit mapping expands "fi" to "fi".

## Design Rationale and Trade‑offs

The **conditional NFKC normalization** in PDF‑Inspector reflects careful engineering around real‑world PDF constraints:

- **Precision over convenience**: Skipping NFKC for Latin‑only text preserves NBSP semantics that spacing heuristics depend on
- **Unicode compliance**: Arabic presentation forms correctly map to base characters only when needed
- **Defensive programming**: Explicit ligature matching catches edge cases the normalization crate misses
- **Bidirectional integrity**: Visual‑to‑logical conversion occurs only for the text that actually requires it

## Summary

- **`expand_ligatures`** in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) (lines 39‑96) is the central NFKC normalization and ligature expansion routine
- **Conditional execution**: NFKC runs only when Arabic presentation forms (U+FB50‑FDFF, U+FE70‑FEFF) are detected, protecting NBSP and other spacing‑sensitive characters
- **Dual expansion strategy**: Unicode normalization handles standard decompositions; explicit match arms cover private‑use ligatures
- **Arabic handling**: Visual‑order text receives NFKC normalization followed by `reverse_visual_arabic` for correct RTL output
- **Pipeline integration**: Called automatically from [`content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/content_stream.rs) (line 595) and [`xobjects.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/xobjects.rs) (lines 676, 880)

## Frequently Asked Questions

### What is NFKC normalization and why does PDF‑Inspector use it conditionally?

**NFKC** (Normalization Form KC) applies compatibility decomposition followed by canonical composition. It expands ligatures and converts stylistic variants to base forms. PDF‑Inspector uses it **conditionally** because global NFKC would convert non‑breaking spaces to regular spaces, breaking the spacing heuristics that detect word boundaries in extracted text. The code only triggers NFKC when Arabic presentation forms are present, since those always require normalization for correct extraction.

### Which ligatures does PDF‑Inspector explicitly expand beyond NFKC?

The explicit match block in [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) (lines 69‑76) expands six **Latin Presentation Forms‑A** code points: U+FB00 ("ff"), U+FB01 ("fi"), U+FB02 ("fl"), U+FB03 ("ffi"), U+FB04 ("ffl"), and U+FB05/U+FB06 ("st"). These catch edge cases where PDF fonts use custom ToUnicode mappings that bypass standard Unicode normalization.

### Why does Arabic text need special handling in ligature expansion?

PDF generators often store Arabic text in **visual (LTR) order** using presentation‑form code points rather than logical RTL order with base Arabic characters. PDF‑Inspector detects these presentation forms to trigger NFKC normalization, which maps visual forms back to base code points. It then calls `reverse_visual_arabic` to restore the correct logical reading order for downstream processing.

### How can I use PDF‑Inspector's ligature expansion in my own code?

Import `expand_ligatures` from `pdf_inspector::text_utils` and pass any string potentially containing ligatures. The function returns a cleaned `String` with expanded ligatures, removed control characters, and corrected Arabic ordering. It is called automatically during PDF extraction, but available for standalone use when preprocessing text from other sources.