LangExtract Prompt Validation and Alignment: How Google Validates Few-Shot Prompts

LangExtract validates few-shot prompt alignment by verifying that extraction annotations exactly match their source text using exact and fuzzy matching algorithms, with configurable validation levels that can trigger warnings or raise PromptAlignmentError exceptions.

LangExtract is an open-source extraction framework developed by Google that enables structured data extraction from unstructured text using few-shot prompting. A critical challenge in few-shot learning is ensuring that the annotated examples in your prompt actually align with the text they claim to extract. This article examines how LangExtract implements robust prompt validation and alignment verification to prevent runtime extraction errors.

Core Components of LangExtract Prompt Validation

The validation system in langextract/prompt_validation.py centers around three primary components that work together to verify prompt integrity.

PromptValidationLevel Enum

The PromptValidationLevel enum (lines 44-50 in langextract/prompt_validation.py) defines three validation behaviors:

• OFF – Disables validation entirely
• WARNING – Logs alignment issues but continues execution
• ERROR – Raises PromptAlignmentError when misalignments are detected

validate_prompt_alignment Function

The validate_prompt_alignment() function (lines 22-39) serves as the entry point for validation. It accepts a list of ExampleData objects and iterates through each example, using a WordAligner instance to verify that every extraction in the example aligns with the example's source text.

The function returns a ValidationReport object containing detailed information about any alignment failures or non-exact matches.

handle_alignment_report Function

The handle_alignment_report() function (lines 112-128) consumes the ValidationReport and implements the policy defined by PromptValidationLevel. When the level is set to ERROR, it raises PromptAlignmentError for failed alignments. If strict_non_exact=True, it also raises errors for fuzzy or partial matches that would otherwise be accepted.

How LangExtract Aligns Extractions with Source Text

The alignment engine resides in langextract/resolver.py and uses a hybrid approach combining exact token matching with fuzzy fallback mechanisms.

Exact Matching with SequenceMatcher

The WordAligner.align_extractions() method (lines 663-728) first attempts exact matching using Python's difflib.SequenceMatcher. The algorithm:

1. Tokenizes the source text and extraction texts using _tokenize_with_lowercase()
2. Builds token sequences with delimiters separating multiple extractions
3. Identifies matching blocks where tokens are identical
4. Assigns AlignmentStatus.MATCH_EXACT when a block covers the full extraction
5. Assigns AlignmentStatus.MATCH_LESSER when accept_match_lesser=True and the block is shorter than the extraction

Fuzzy Matching Algorithm

When exact matching fails, the aligner optionally executes fuzzy alignment (lines implemented in _fuzzy_align_extraction):

1. Normalizes tokens via _normalize_token() (lowercasing and light stemming)
2. Slides a window over the source tokens
3. Measures overlap using difflib.SequenceMatcher.ratio()
4. Assigns AlignmentStatus.MATCH_FUZZY if the best ratio exceeds the configurable threshold (default 0.75)

Alignment Status Classification

The AlignmentStatus enum in langextract/core/data.py defines four states:

• MATCH_EXACT – Perfect token alignment
• MATCH_FUZZY – Fuzzy match above threshold
• MATCH_LESSER – Partial exact match (shorter than extraction)
• NO_MATCH – No alignment found

Building and Acting on Validation Reports

The validation system generates structured reports that enable precise debugging of prompt issues.

ValidationIssue and ValidationReport Structure

Each alignment problem is captured as a ValidationIssue (lines 59-71 in prompt_validation.py) containing:

• example_index: Position in the few-shot list
• example_id: Optional identifier for the example
• extraction_class: The entity type being extracted
• extraction_text_preview: Truncated text of the extraction
• alignment_status: The actual status returned by the aligner
• issue_kind: Either FAILED (no match) or NON_EXACT (fuzzy/partial)

The ValidationReport wrapper (lines 86-101) provides has_failed and has_non_exact boolean properties for quick policy checks.

Error Handling Based on Validation Level

The handle_alignment_report() function implements a three-tier response system:

1. Silent mode (OFF): Returns immediately without inspection
2. Warning mode (WARNING): Iterates through all issues and logs warnings using the short_msg() method of each ValidationIssue
3. Strict mode (ERROR): Raises PromptAlignmentError with a detailed message if any FAILED issues exist; additionally raises if strict_non_exact=True and NON_EXACT issues are present

Configuring Alignment Policy

The AlignmentPolicy dataclass (referenced in prompt_validation.py lines 107-113) allows fine-tuning of the alignment behavior:

@dataclass
class AlignmentPolicy:
    enable_fuzzy_alignment: bool = True
    fuzzy_alignment_threshold: float = 0.75
    accept_match_lesser: bool = False

• enable_fuzzy_alignment: Toggle fuzzy fallback when exact matching fails
• fuzzy_alignment_threshold: Minimum similarity ratio (0.0-1.0) for fuzzy acceptance
• accept_match_lesser: Whether to accept partial exact matches as valid

When omitted, validate_prompt_alignment() uses default AlignmentPolicy() values.

Practical Implementation Examples

Basic Validation Pipeline

This example demonstrates the complete validation workflow using the high-level API:

from langextract import resolver, prompt_validation as pv

# List of few-shot examples

examples = [...]  # List[data.ExampleData]

# Generate validation report

report = pv.validate_prompt_alignment(
    examples,
    aligner=resolver.WordAligner(),
    policy=pv.AlignmentPolicy(),
)

# Handle based on severity level

pv.handle_alignment_report(
    report,
    level=pv.PromptValidationLevel.ERROR,
    strict_non_exact=True,
)

When PromptValidationLevel.ERROR is specified, the function raises PromptAlignmentError if any extraction fails alignment or produces a non-exact match.

Custom Fuzzy Threshold Configuration

For applications requiring stricter alignment verification:

policy = pv.AlignmentPolicy(
    enable_fuzzy_alignment=True,
    fuzzy_alignment_threshold=0.90,  # Require 90% token overlap

    accept_match_lesser=False,
)

report = pv.validate_prompt_alignment(examples, policy=policy)
pv.handle_alignment_report(report, level=pv.PromptValidationLevel.WARNING)

Low-Level WordAligner Usage

Direct access to the alignment engine for custom validation logic:

from langextract.resolver import WordAligner, data

aligner = WordAligner()
source_text = "Alice went to Paris in 2022."

extractions = [
    data.Extraction(
        extraction_class="PERSON",
        extraction_text="Alice",
    ),
    data.Extraction(
        extraction_class="LOCATION",
        extraction_text="Paris",
    ),
]

aligned = aligner.align_extractions(
    extraction_groups=[extractions],
    source_text=source_text,
    token_offset=0,
    char_offset=0,
)

# Each result contains token_interval, char_interval, and alignment_status

for group in aligned:
    for extraction in group:
        print(f"{extraction.extraction_class}: {extraction.alignment_status}")

Summary

• LangExtract validates few-shot prompts through a three-tier system controlled by PromptValidationLevel (OFF, WARNING, ERROR).
• The validate_prompt_alignment() function in langextract/prompt_validation.py orchestrates validation by iterating through examples and using WordAligner to verify extraction alignment.
• Alignment uses hybrid exact/fuzzy matching via difflib.SequenceMatcher, with fuzzy thresholds defaulting to 0.75 and configurable through AlignmentPolicy.
• Validation reports aggregate issues as ValidationIssue objects containing example indices, extraction previews, and AlignmentStatus classifications (MATCH_EXACT, MATCH_FUZZY, MATCH_LESSER, NO_MATCH).
• Error handling via handle_alignment_report() raises PromptAlignmentError for failed alignments when configured strictly, or logs warnings based on the selected validation level.

Frequently Asked Questions

What happens when LangExtract detects a misaligned extraction in strict mode?

When PromptValidationLevel.ERROR is enabled and handle_alignment_report() encounters a FAILED alignment issue, it raises PromptAlignmentError with a detailed message indicating which example index and extraction class failed to align. If strict_non_exact=True, it also raises errors for fuzzy or partial (MATCH_LESSER) matches that would otherwise be accepted as valid alignments.

How does LangExtract handle case sensitivity and pluralization during alignment?

The WordAligner in langextract/resolver.py normalizes tokens using _tokenize_with_lowercase() and _normalize_token() helpers (lines 744-770 and 803-810). These functions convert tokens to lowercase and apply light stemming to handle pluralization, ensuring that "Paris" and "paris" or "companies" and "company" can match during the fuzzy alignment phase.

Can I disable fuzzy matching and require only exact alignments?

Yes, you can disable fuzzy matching by setting enable_fuzzy_alignment=False in the AlignmentPolicy dataclass. When disabled, the WordAligner will only attempt exact matching using difflib.SequenceMatcher. If an extraction does not produce an exact match block, it will be marked with AlignmentStatus.NO_MATCH regardless of how similar the text is to the source.

What is the difference between MATCH_LESSER and MATCH_FUZZY in LangExtract?

MATCH_LESSER indicates that an exact token match was found, but the matching block is shorter than the full extraction text (partial match), and this is only accepted if accept_match_lesser=True in the policy. MATCH_FUZZY indicates that no exact match was found, but a fuzzy search using normalized token sliding windows achieved a similarity ratio above the configured threshold (default 0.75), indicating the extraction text is present in the source with minor variations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}