How LangExtract Performs Source Grounding to Map Extractions to Text Locations

LangExtract grounds extractions to their original text locations by tokenizing source documents, applying exact matching via difflib.SequenceMatcher, and falling back to a fuzzy sliding-window algorithm when exact matches fail.

Source grounding (also called alignment) converts high-level extraction objects returned by a language model into concrete character and token positions inside the original source text. In the google/langextract repository, this capability bridges the gap between LLM outputs and document-level analysis. Understanding how LangExtract performs source grounding enables developers to build reliable visualization tools and audit extraction accuracy against source documents.

The Three-Step Source Grounding Pipeline

LangExtract’s alignment process operates in three coordinated steps implemented primarily in langextract/resolver.py and langextract/core/tokenizer.py.

Step 1: Tokenizing the Source Text with RegexTokenizer

The pipeline begins by splitting the source text into a deterministic sequence of tokens using the RegexTokenizer class defined in langextract/core/tokenizer.py. This fast regex-based implementation handles words, numbers, and punctuation consistently. While RegexTokenizer is the default, any custom tokenizer implementing the Tokenizer API can be supplied to Resolver.align() to accommodate domain-specific tokenization requirements.

Step 2: Exact Matching via difflib.SequenceMatcher

The core alignment logic resides in Resolver.align(), which instantiates a WordAligner to map extraction tokens to source tokens. The aligner builds two token streams:

• The source token list from the original document
• A concatenated token list representing all extractions separated by the unique delimiter "\u241F" (Unicode character U+241F)

Using Python’s difflib.SequenceMatcher, the WordAligner.align_extractions() method identifies matching blocks between these streams via self._get_matching_blocks(). When a block covers an entire extraction, the system assigns AlignmentStatus.MATCH_EXACT and populates the extraction’s TokenInterval and CharInterval with precise start and end positions.

Step 3: Fuzzy Fallback Alignment

When exact matching fails—due to minor tokenization differences, stemming, or punctuation changes—the aligner falls back to a sliding-window fuzzy algorithm implemented in WordAligner._fuzzy_align_extraction(). This method performs a cheap token-overlap pre-check to filter candidates, then scores each window using difflib.SequenceMatcher.ratio().

If the best match exceeds the configurable fuzzy_alignment_threshold (default 0.75), the extraction receives AlignmentStatus.MATCH_FUZZY and its intervals are set accordingly. This ensures robust source grounding even when LLM outputs deviate slightly from source text.

Accessing Grounded Location Data

Once alignment completes, location data resides in the Extraction objects defined in langextract/core/data.py. Each extraction exposes its grounded intervals as properties:

extraction.token_interval   # → TokenInterval(start_index, end_index)

extraction.char_interval    # → CharInterval(start_pos, end_pos)

These intervals enable downstream components to highlight exact spans in the original document. The langextract/visualization.py module consumes these intervals to render highlighted HTML, while evaluation scripts use them to verify extraction accuracy against ground-truth annotations.

Complete Source Grounding Example

The following example demonstrates the full pipeline from raw LLM output to grounded extractions:

from langextract.resolver import Resolver
from langextract.core.data import Document, Extraction
from langextract.core import tokenizer

# 1️⃣  Prepare a document

text = "Dr. Alice prescribed Aspirin and ibuprofen to the patient."
doc = Document(text=text)

# 2️⃣  Simulated LLM output (normally produced by a provider)

#     The format is a list of dicts with class → value.

extractions_raw = [
    {"PERSON": "Alice", "PERSON_index": 0},
    {"MEDICATION": "Aspirin", "MEDICATION_index": 1},
    {"MEDICATION": "ibuprofen", "MEDICATION_index": 2},
]

# 3️⃣  Resolve raw data → Extraction objects (ordering by index)

resolver = Resolver()
structured = resolver.resolve(str(extractions_raw))   # parses raw JSON/YAML

# structured is a list[Extraction]

# 4️⃣  Align the extractions to the source text

aligned = list(
    resolver.align(
        extractions=structured,
        source_text=doc.text,
        token_offset=0,           # start of the whole document

        char_offset=0,            # start character offset

        enable_fuzzy_alignment=True,
    )
)

# 5️⃣  Inspect the grounded spans

for ext in aligned:
    span = doc.text[ext.char_interval.start_pos : ext.char_interval.end_pos]
    print(f"{ext.extraction_class}: '{ext.extraction_text}' → '{span}' "
          f"[{ext.alignment_status.name}]")

Output:

PERSON: 'Alice' → 'Alice' [MATCH_EXACT]
MEDICATION: 'Aspirin' → 'Aspirin' [MATCH_EXACT]
MEDICATION: 'ibuprofen' → 'ibuprofen' [MATCH_EXACT]

Key Implementation Files

Source grounding logic is distributed across four primary modules in the google/langextract repository:

• langextract/resolver.py – Contains Resolver.align() and the WordAligner class that implements exact matching via difflib.SequenceMatcher and fuzzy fallback alignment.
• langextract/core/tokenizer.py – Defines RegexTokenizer and the Tokenizer API used to create deterministic token sequences for alignment.
• langextract/core/data.py – Declares Extraction, TokenInterval, CharInterval, and AlignmentStatus dataclasses that store grounding results.
• langextract/visualization.py – Consumes grounded intervals to render highlighted HTML spans for human review.

Summary

LangExtract performs source grounding through a robust three-stage pipeline that bridges LLM outputs and source document locations:

• Tokenization splits source text into deterministic sequences using RegexTokenizer from langextract/core/tokenizer.py.
• Exact alignment employs difflib.SequenceMatcher within Resolver.align() to map extraction tokens to source tokens with AlignmentStatus.MATCH_EXACT.
• Fuzzy fallback applies a sliding-window algorithm when exact matches fail, accepting candidates above a 0.75 similarity threshold as AlignmentStatus.MATCH_FUZZY.

The resulting TokenInterval and CharInterval objects stored in Extraction instances enable precise downstream analysis, visualization, and evaluation.

Frequently Asked Questions

How does LangExtract handle tokenization differences between the LLM output and source text?

LangExtract uses a configurable Tokenizer API defined in langextract/core/tokenizer.py. The default RegexTokenizer applies consistent regex-based rules to both source text and extraction strings. When minor tokenization differences occur, the system falls back to fuzzy alignment in WordAligner._fuzzy_align_extraction(), which uses token-overlap pre-checks and difflib.SequenceMatcher.ratio() scoring to find matches despite variations.

What is the difference between MATCH_EXACT and MATCH_FUZZY alignment statuses?

MATCH_EXACT indicates that difflib.SequenceMatcher found a contiguous block of tokens in the source text that perfectly matches the entire extraction token sequence. MATCH_FUZZY indicates that exact matching failed, but the sliding-window fuzzy algorithm found a candidate window exceeding the default 0.75 similarity threshold. Both statuses populate TokenInterval and CharInterval, but MATCH_EXACT offers higher confidence for downstream validation.

Can I customize the fuzzy alignment threshold in LangExtract?

Yes, the fuzzy alignment threshold is configurable through the Resolver.align() method. The enable_fuzzy_alignment boolean toggles fuzzy matching, while the fuzzy_alignment_threshold parameter (defaulting to 0.75) controls the minimum difflib.SequenceMatcher.ratio() score required to accept a fuzzy match. Adjusting this threshold allows trade-offs between recall and precision; lower values catch more variations but may introduce false positives.

Which LangExtract classes store the final grounded location intervals?

Grounded locations are stored in the Extraction class defined in langextract/core/data.py. Each instance contains a token_interval property pointing to a TokenInterval dataclass (with start_index and end_index) and a char_interval property pointing to a CharInterval dataclass (with start_pos and end_pos). These objects are populated by Resolver.align() and consumed by visualization and evaluation tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}