How LangExtract Handles Source Grounding and Text Alignment: A Technical Deep Dive

LangExtract grounds LLM extractions to source text by assigning character and token intervals through a three-stage pipeline that uses exact token matching with difflib.SequenceMatcher and optional fuzzy alignment, storing results in the Extraction object's char_interval, token_interval, and alignment_status fields.

LangExtract is an open-source Python library developed by Google that transforms unstructured LLM outputs into structured data while maintaining traceability to the original source text. Understanding how LangExtract handles source grounding and text alignment is essential for building reliable extraction pipelines that can verify model outputs against their documentary origins.

The Three-Stage Grounding Pipeline

LangExtract implements source grounding and text alignment through a systematic three-stage workflow defined in the Resolver class within langextract/resolver.py.

Stage 1: Parsing Raw LLM Output

The process begins in the Resolver.resolve() method, which delegates to a FormatHandler to parse JSON or YAML strings into a list of Extraction objects. At this initial stage, extractions contain semantic content but lack positional metadata.

Stage 2: Token-Level Alignment Strategies

The Resolver.align() method triggers the WordAligner.align_extractions() implementation, which performs source grounding and text alignment using two complementary strategies:

• Exact Token Matching: The aligner first attempts exact token-level matching using difflib.SequenceMatcher to compare token sequences between the extraction and source text.
• Fuzzy Alignment Fallback: When exact matching fails, the system optionally falls back to fuzzy alignment that scans sliding windows of source tokens, selecting the best overlap above a configurable threshold.

Stage 3: Recording Character Intervals and Alignment Status

For every successful match, the aligner populates three critical fields on each Extraction object defined in langextract/core/data.py:

• token_interval: A TokenInterval containing start/end token indices relative to the chunk.
• char_interval: A CharInterval containing start/end character offsets in the original string.
• alignment_status: An AlignmentStatus enum value describing the match quality.

Understanding Alignment Statuses

The AlignmentStatus enum in langextract/core/data.py defines four possible outcomes for source grounding and text alignment:

• MATCH_EXACT: Tokens line up perfectly between extraction and source.
• MATCH_GREATER: Extraction matches a longer span than expected.
• MATCH_LESSER: Extraction is longer than the matched source span.
• MATCH_FUZZY: Overlap ratio meets or exceeds the configurable fuzzy threshold.

When no interval can be determined, both char_interval and alignment_status remain None, marking the extraction as ungrounded. This distinction is critical for downstream validation, as demonstrated in benchmarks/benchmark.py where grounded versus ungrounded entities are counted by checking extraction.char_interval.

Practical Code Examples

Example 1: Basic Grounding with Exact Matching

The following example demonstrates standard source grounding and text alignment using the default resolver:

from langextract import Resolver, data

# Model-generated JSON (normally from an LLM)

model_output = """
{
  "person": "Ada Lovelace",
  "occupation": "mathematician"
}
"""

# Resolve to Extraction objects (no positions yet)

resolver = Resolver()
extractions: list[data.Extraction] = resolver.resolve(model_output)

# Align to the original source paragraph

source = "Ada Lovelace was a pioneering mathematician and writer."
aligned = list(
    resolver.align(
        extractions,
        source_text=source,
        token_offset=0,
        char_offset=0,
        enable_fuzzy_alignment=True,
    )
)

# Inspect grounding information

for ext in aligned:
    print(f"{ext.extraction_class!r}: {ext.extraction_text!r}")
    print(f"  Char interval: {ext.char_interval}")
    print(f"  Token interval: {ext.token_interval}")
    print(f"  Alignment status: {ext.alignment_status}\n")

Typical output:

'person': 'Ada Lovelace'
  Char interval: CharInterval(start_pos=0, end_pos=12)
  Token interval: TokenInterval(start_index=0, end_index=2)
  Alignment status: AlignmentStatus.MATCH_EXACT

'occupation': 'mathematician'
  Char interval: CharInterval(start_pos=27, end_pos=39)
  Token interval: TokenInterval(start_index=5, end_index=6)
  Alignment status: AlignmentStatus.MATCH_EXACT

Example 2: Fuzzy Alignment for Sub-Phrases

When extractions contain partial matches, fuzzy alignment bridges the gap:

source = "The quick brown fox jumps over the lazy dog."

# Extraction missing the article "the"

extractions = [
    data.Extraction(
        extraction_class="animal",
        extraction_text="lazy dog",
    )
]

resolver = Resolver()
aligned = list(
    resolver.align(
        extractions,
        source_text=source,
        token_offset=0,
        char_offset=0,
        enable_fuzzy_alignment=True,
        fuzzy_alignment_threshold=0.7,
    )
)

print(aligned[0].char_interval)      # CharInterval(start_pos=35, end_pos=43)

print(aligned[0].alignment_status)   # AlignmentStatus.MATCH_FUZZY

Example 3: Detecting Ungrounded Extractions

Identify hallucinations or failed matches by checking for None intervals:

from langextract import Resolver, data

source = "Paris is the capital of France."
extractions = [
    data.Extraction(extraction_class="city", extraction_text="Berlin"),
]

resolver = Resolver()
aligned = list(
    resolver.align(
        extractions,
        source_text=source,
        token_offset=0,
        char_offset=0,
        enable_fuzzy_alignment=False,
    )
)

print(aligned[0].char_interval)      # None

print(aligned[0].alignment_status)   # None (ungrounded)

Key Implementation Files

The source grounding and text alignment system spans several critical files in the google/langextract repository:

File	Purpose
langextract/core/data.py	Defines Extraction, CharInterval, TokenInterval, and the AlignmentStatus enum.
langextract/resolver.py	Contains the Resolver class with the align() method and the inner WordAligner class implementing exact and fuzzy matching.
langextract/core/tokenizer.py	Provides tokenization utilities and TokenInterval definitions used during alignment.
tests/resolver_test.py	Validates grounding accuracy across exact, lesser, fuzzy, and missing alignment scenarios.
benchmarks/benchmark.py	Demonstrates production usage by distinguishing grounded from ungrounded extractions.

Summary

LangExtract implements a robust source grounding and text alignment pipeline that bridges the gap between unstructured LLM outputs and traceable data extractions:

• Three-stage process: Parse model output into Extraction objects, align tokens using exact or fuzzy matching via WordAligner, and record positional metadata.
• Dual alignment strategies: Exact token matching with difflib.SequenceMatcher provides precision, while fuzzy sliding-window matching handles partial phrases and minor variations.
• Comprehensive metadata: Each extraction receives char_interval, token_interval, and alignment_status fields, enabling downstream validation and source slicing.
• Ungrounded detection: Failed alignments result in None values for intervals and status, allowing applications to filter hallucinations or mismatched extractions as demonstrated in the benchmark suite.

Frequently Asked Questions

What is the difference between exact and fuzzy alignment in LangExtract?

Exact alignment uses difflib.SequenceMatcher to find perfect token-level matches between the extraction text and source text, resulting in MATCH_EXACT status. Fuzzy alignment activates when exact matching fails, scanning sliding windows of source tokens to find the best partial overlap above a configurable threshold, resulting in MATCH_FUZZY status. Fuzzy alignment accommodates extractions that omit articles or minor words present in the source text.

How does LangExtract represent ungrounded extractions?

When the WordAligner cannot locate the extraction text in the source through either exact or fuzzy matching, it leaves the char_interval and token_interval fields as None and sets alignment_status to None. These ungrounded extractions indicate potential hallucinations or entities drawn from the model's training data rather than the provided context, allowing downstream applications to filter or flag them accordingly.

What files contain the core alignment logic?

The primary alignment implementation resides in langextract/resolver.py, specifically within the Resolver.align() method and the inner WordAligner class. Data structures supporting grounding—including Extraction, CharInterval, TokenInterval, and AlignmentStatus—are defined in langextract/core/data.py. Tokenization utilities used during alignment are located in langextract/core/tokenizer.py. Comprehensive tests validating these behaviors exist in tests/resolver_test.py.

Can I adjust the fuzzy alignment threshold?

Yes, the fuzzy alignment threshold is configurable through the fuzzy_alignment_threshold parameter in the Resolver.align() method. By default, LangExtract requires a high overlap ratio for fuzzy matches, but you can lower this threshold to accommodate more aggressive partial matching. However, reducing the threshold increases the risk of false positives where extractions align to semantically incorrect portions of the source text.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}