How LangExtract Handles Overlapping Extractions Across Multiple Passes

LangExtract resolves overlapping extractions across multiple passes using a deterministic "first-pass wins" policy, where extractions from earlier passes take precedence and later overlapping spans are automatically discarded.

When running multiple extraction passes to improve recall, the open-source google/langextract library must reconcile cases where the same character span is identified in more than one pass. Understanding how LangExtract manages these overlapping extractions is critical for configuring multi-pass pipelines and interpreting extraction results.

The First-Pass Wins Policy for Overlapping Extractions

LangExtract implements a strict precedence system when merging results from multiple extraction passes. Each pass generates a list of data.Extraction objects containing character intervals. When these intervals overlap across passes, the extraction from the earlier pass is retained, and the later extraction is discarded.

How Passes Are Ordered and Merged

The merge process occurs in annotation._merge_non_overlapping_extractions, which processes passes sequentially in their numeric order (pass 0, pass 1, pass 2, etc.). The function iterates through each pass's extractions and checks for interval collisions against the already-accepted extractions from previous passes.

This logic is invoked by _annotate_documents_sequential_passes in langextract/annotation.py when extraction_passes > 1 is specified in the Annotator.annotate_text call.

The Interval Overlap Detection Logic

The actual collision detection happens in annotation._extractions_overlap, which compares the char_interval attributes of two Extraction objects. Two intervals are considered overlapping only when:

start1 < end2 and start2 < end1

This mathematical definition means that adjacent intervals—such as [0, 5) and [5, 10)—are not treated as overlapping. Only true intersections trigger the first-pass wins exclusion rule.

Core Implementation in langextract/annotation.py

The overlap resolution logic is centralized in langextract/annotation.py, with supporting data structures defined in langextract/core/data.py.

The _merge_non_overlapping_extractions Function

Located at lines 46-84 in annotation.py, this function implements the core merging algorithm:

def _merge_non_overlapping_extractions(
    all_pass_extractions: Sequence[Sequence[data.Extraction]],
) -> list[data.Extraction]:
    """Merges extractions from multiple passes, keeping first-pass wins."""
    merged: list[data.Extraction] = []
    
    for pass_extractions in all_pass_extractions:
        for extraction in pass_extractions:
            # Check against all already-accepted extractions

            if not any(
                _extractions_overlap(extraction, existing)
                for existing in merged
            ):
                merged.append(extraction)
    return merged

The _extractions_overlap Helper

This utility function (lines 87-115 in annotation.py) performs the precise interval arithmetic:

def _extractions_overlap(
    extraction1: data.Extraction,
    extraction2: data.Extraction,
) -> bool:
    """Returns True if char_intervals overlap (adjacent is not overlap)."""
    start1, end1 = extraction1.char_interval.start, extraction1.char_interval.end
    start2, end2 = extraction2.char_interval.start, extraction2.char_interval.end
    
    return start1 < end2 and start2 < end1

Code Examples: Managing Overlapping Extractions

Multi-Pass Extraction with Automatic Deduplication

When configuring multiple extraction passes, overlapping spans are automatically handled according to the first-pass wins policy:

from langextract import annotation, prompting, resolver as resolver_lib
from langextract.core import data

annotator = annotation.Annotator(
    language_model=my_gemini_model,
    prompt_template=prompting.PromptTemplateStructured(description="Extract entities"),
)

my_resolver = resolver_lib.Resolver(format_type=data.FormatType.YAML)

text = "Dr. Smith prescribed aspirin."

# Run two passes - first pass extracts doctor, second pass might extract overlapping patient

result = annotator.annotate_text(
    text,
    resolver=my_resolver,
    extraction_passes=2,
    debug=False,
)

print([(e.extraction_class, e.extraction_text) for e in result.extractions])

# Output: [('doctor', 'Dr. Smith'), ('medication', 'aspirin')]

# The overlapping 'patient' extraction from pass 2 is discarded

Direct Merge Function Demonstration

You can observe the overlap resolution directly using the internal merge function:

from langextract.annotation import _merge_non_overlapping_extractions
from langextract.core import data

# Pass 1 extracts "Dr. Smith" (0-10)

pass1 = [
    data.Extraction("doctor", "Dr. Smith",
                    char_interval=data.CharInterval(0, 10)),
]

# Pass 2 extracts overlapping "Smith" (4-10) and non-overlapping "aspirin" (21-28)

pass2 = [
    data.Extraction("patient", "Smith",
                    char_interval=data.CharInterval(4, 10)),
    data.Extraction("medication", "aspirin",
                    char_interval=data.CharInterval(21, 28)),
]

merged = _merge_non_overlapping_extractions([pass1, pass2])
print([e.extraction_class for e in merged])

# → ['doctor', 'medication']

# 'patient' is dropped due to interval overlap with 'doctor'

Unit Test Verification

The library's test suite validates this behavior in tests/annotation_test.py:

from langextract import annotation, core
from langextract.core import data

def test_first_pass_wins():
    all_extractions = [
        [data.Extraction("class1", "text1", char_interval=data.CharInterval(0, 10))],
        [data.Extraction("class2", "text2", char_interval=data.CharInterval(5, 15)),
         data.Extraction("class3", "text3", char_interval=data.CharInterval(20, 25))],
    ]
    merged = annotation._merge_non_overlapping_extractions(all_extractions)
    assert len(merged) == 2
    assert {e.extraction_class for e in merged} == {"class1", "class3"}

Summary

• First-pass wins policy: When the same character span appears in multiple extraction passes, LangExtract keeps the extraction from the earliest pass and discards later overlaps.
• Precise interval logic: The overlap detector in _extractions_overlap uses strict interval intersection (start1 < end2 and start2 < end1), treating adjacent but non-intersecting spans as distinct.
• Centralized implementation: All overlap resolution logic resides in langextract/annotation.py, specifically within _merge_non_overlapping_extractions and _extractions_overlap.
• Automatic deduplication: When using Annotator.annotate_text with extraction_passes > 1, the merging happens automatically without requiring manual intervention.

Frequently Asked Questions

What happens when two extraction passes identify the same text span?

LangExtract applies a "first-pass wins" rule. The extraction from the earlier pass is retained, while the extraction from the later pass is discarded. This ensures deterministic results when running multiple passes to improve recall.

Does LangExtract consider adjacent text intervals as overlapping?

No. The _extractions_overlap function in langextract/annotation.py treats intervals as overlapping only when they truly intersect using the logic start1 < end2 and start2 < end1. Adjacent intervals such as [0, 5) and [5, 10) are considered non-overlapping and both would be retained.

How can I configure the number of extraction passes in LangExtract?

Set the extraction_passes parameter when calling Annotator.annotate_text(). The default value is 1. When you specify a value greater than 1, LangExtract automatically runs sequential passes and merges the results using the first-pass wins overlap resolution strategy.

Where is the overlap detection logic implemented in the codebase?

The overlap detection logic is implemented in langextract/annotation.py. The key functions are _extractions_overlap, which performs the mathematical interval comparison, and _merge_non_overlapping_extractions, which orchestrates the first-pass wins merging across multiple passes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}