# How LangExtract Performs Source Grounding to Map Extractions to Text Locations

> Learn how LangExtract grounds extractions to text locations using tokenization, exact matching with difflib, and fuzzy matching for accurate results.

- Repository: [Google/langextract](https://github.com/google/langextract)
- Tags: internals
- Published: 2026-02-16

---

**LangExtract grounds extractions to their original text locations by tokenizing source documents, applying exact matching via difflib.SequenceMatcher, and falling back to a fuzzy sliding-window algorithm when exact matches fail.**

Source grounding (also called alignment) converts high-level extraction objects returned by a language model into concrete character and token positions inside the original source text. In the `google/langextract` repository, this capability bridges the gap between LLM outputs and document-level analysis. Understanding how LangExtract performs source grounding enables developers to build reliable visualization tools and audit extraction accuracy against source documents.

## The Three-Step Source Grounding Pipeline

LangExtract’s alignment process operates in three coordinated steps implemented primarily in [`langextract/resolver.py`](https://github.com/google/langextract/blob/main/langextract/resolver.py) and [`langextract/core/tokenizer.py`](https://github.com/google/langextract/blob/main/langextract/core/tokenizer.py).

### Step 1: Tokenizing the Source Text with RegexTokenizer

The pipeline begins by splitting the source text into a deterministic sequence of tokens using the **RegexTokenizer** class defined in [`langextract/core/tokenizer.py`](https://github.com/google/langextract/blob/main/langextract/core/tokenizer.py). This fast regex-based implementation handles words, numbers, and punctuation consistently. While `RegexTokenizer` is the default, any custom tokenizer implementing the `Tokenizer` API can be supplied to `Resolver.align()` to accommodate domain-specific tokenization requirements.

### Step 2: Exact Matching via difflib.SequenceMatcher

The core alignment logic resides in `Resolver.align()`, which instantiates a **WordAligner** to map extraction tokens to source tokens. The aligner builds two token streams:

- The *source* token list from the original document
- A *concatenated* token list representing all extractions separated by the unique delimiter `"\u241F"` (Unicode character U+241F)

Using Python’s `difflib.SequenceMatcher`, the `WordAligner.align_extractions()` method identifies matching blocks between these streams via `self._get_matching_blocks()`. When a block covers an entire extraction, the system assigns `AlignmentStatus.MATCH_EXACT` and populates the extraction’s **TokenInterval** and **CharInterval** with precise start and end positions.

### Step 3: Fuzzy Fallback Alignment

When exact matching fails—due to minor tokenization differences, stemming, or punctuation changes—the aligner falls back to a sliding-window fuzzy algorithm implemented in `WordAligner._fuzzy_align_extraction()`. This method performs a cheap token-overlap pre-check to filter candidates, then scores each window using `difflib.SequenceMatcher.ratio()`.

If the best match exceeds the configurable **fuzzy_alignment_threshold** (default `0.75`), the extraction receives `AlignmentStatus.MATCH_FUZZY` and its intervals are set accordingly. This ensures robust source grounding even when LLM outputs deviate slightly from source text.

## Accessing Grounded Location Data

Once alignment completes, location data resides in the **Extraction** objects defined in [`langextract/core/data.py`](https://github.com/google/langextract/blob/main/langextract/core/data.py). Each extraction exposes its grounded intervals as properties:

```python
extraction.token_interval   # → TokenInterval(start_index, end_index)

extraction.char_interval    # → CharInterval(start_pos, end_pos)

```

These intervals enable downstream components to highlight exact spans in the original document. The [`langextract/visualization.py`](https://github.com/google/langextract/blob/main/langextract/visualization.py) module consumes these intervals to render highlighted HTML, while evaluation scripts use them to verify extraction accuracy against ground-truth annotations.

## Complete Source Grounding Example

The following example demonstrates the full pipeline from raw LLM output to grounded extractions:

```python
from langextract.resolver import Resolver
from langextract.core.data import Document, Extraction
from langextract.core import tokenizer

# 1️⃣  Prepare a document

text = "Dr. Alice prescribed Aspirin and ibuprofen to the patient."
doc = Document(text=text)

# 2️⃣  Simulated LLM output (normally produced by a provider)

#     The format is a list of dicts with class → value.

extractions_raw = [
    {"PERSON": "Alice", "PERSON_index": 0},
    {"MEDICATION": "Aspirin", "MEDICATION_index": 1},
    {"MEDICATION": "ibuprofen", "MEDICATION_index": 2},
]

# 3️⃣  Resolve raw data → Extraction objects (ordering by index)

resolver = Resolver()
structured = resolver.resolve(str(extractions_raw))   # parses raw JSON/YAML

# structured is a list[Extraction]

# 4️⃣  Align the extractions to the source text

aligned = list(
    resolver.align(
        extractions=structured,
        source_text=doc.text,
        token_offset=0,           # start of the whole document

        char_offset=0,            # start character offset

        enable_fuzzy_alignment=True,
    )
)

# 5️⃣  Inspect the grounded spans

for ext in aligned:
    span = doc.text[ext.char_interval.start_pos : ext.char_interval.end_pos]
    print(f"{ext.extraction_class}: '{ext.extraction_text}' → '{span}' "
          f"[{ext.alignment_status.name}]")

```

Output:

```

PERSON: 'Alice' → 'Alice' [MATCH_EXACT]
MEDICATION: 'Aspirin' → 'Aspirin' [MATCH_EXACT]
MEDICATION: 'ibuprofen' → 'ibuprofen' [MATCH_EXACT]

```

## Key Implementation Files

Source grounding logic is distributed across four primary modules in the `google/langextract` repository:

- **[`langextract/resolver.py`](https://github.com/google/langextract/blob/main/langextract/resolver.py)** – Contains `Resolver.align()` and the `WordAligner` class that implements exact matching via `difflib.SequenceMatcher` and fuzzy fallback alignment.
- **[`langextract/core/tokenizer.py`](https://github.com/google/langextract/blob/main/langextract/core/tokenizer.py)** – Defines `RegexTokenizer` and the `Tokenizer` API used to create deterministic token sequences for alignment.
- **[`langextract/core/data.py`](https://github.com/google/langextract/blob/main/langextract/core/data.py)** – Declares `Extraction`, `TokenInterval`, `CharInterval`, and `AlignmentStatus` dataclasses that store grounding results.
- **[`langextract/visualization.py`](https://github.com/google/langextract/blob/main/langextract/visualization.py)** – Consumes grounded intervals to render highlighted HTML spans for human review.

## Summary

LangExtract performs source grounding through a robust three-stage pipeline that bridges LLM outputs and source document locations:

- **Tokenization** splits source text into deterministic sequences using `RegexTokenizer` from [`langextract/core/tokenizer.py`](https://github.com/google/langextract/blob/main/langextract/core/tokenizer.py).
- **Exact alignment** employs `difflib.SequenceMatcher` within `Resolver.align()` to map extraction tokens to source tokens with `AlignmentStatus.MATCH_EXACT`.
- **Fuzzy fallback** applies a sliding-window algorithm when exact matches fail, accepting candidates above a 0.75 similarity threshold as `AlignmentStatus.MATCH_FUZZY`.

The resulting `TokenInterval` and `CharInterval` objects stored in `Extraction` instances enable precise downstream analysis, visualization, and evaluation.

## Frequently Asked Questions

### How does LangExtract handle tokenization differences between the LLM output and source text?

LangExtract uses a configurable **Tokenizer** API defined in [`langextract/core/tokenizer.py`](https://github.com/google/langextract/blob/main/langextract/core/tokenizer.py). The default **RegexTokenizer** applies consistent regex-based rules to both source text and extraction strings. When minor tokenization differences occur, the system falls back to fuzzy alignment in `WordAligner._fuzzy_align_extraction()`, which uses token-overlap pre-checks and `difflib.SequenceMatcher.ratio()` scoring to find matches despite variations.

### What is the difference between MATCH_EXACT and MATCH_FUZZY alignment statuses?

**MATCH_EXACT** indicates that `difflib.SequenceMatcher` found a contiguous block of tokens in the source text that perfectly matches the entire extraction token sequence. **MATCH_FUZZY** indicates that exact matching failed, but the sliding-window fuzzy algorithm found a candidate window exceeding the default 0.75 similarity threshold. Both statuses populate `TokenInterval` and `CharInterval`, but MATCH_EXACT offers higher confidence for downstream validation.

### Can I customize the fuzzy alignment threshold in LangExtract?

Yes, the fuzzy alignment threshold is configurable through the `Resolver.align()` method. The `enable_fuzzy_alignment` boolean toggles fuzzy matching, while the `fuzzy_alignment_threshold` parameter (defaulting to `0.75`) controls the minimum `difflib.SequenceMatcher.ratio()` score required to accept a fuzzy match. Adjusting this threshold allows trade-offs between recall and precision; lower values catch more variations but may introduce false positives.

### Which LangExtract classes store the final grounded location intervals?

Grounded locations are stored in the **Extraction** class defined in [`langextract/core/data.py`](https://github.com/google/langextract/blob/main/langextract/core/data.py). Each instance contains a `token_interval` property pointing to a **TokenInterval** dataclass (with `start_index` and `end_index`) and a `char_interval` property pointing to a **CharInterval** dataclass (with `start_pos` and `end_pos`). These objects are populated by `Resolver.align()` and consumed by visualization and evaluation tools.