# How LangExtract Processes Long Documents with Chunking and Parallel Workers

> LangExtract processes long documents via chunking and parallel workers. Learn how text splitting and batch processing maximize throughput while respecting LLM context limits.

- Repository: [Google/langextract](https://github.com/google/langextract)
- Tags: internals
- Published: 2026-02-16

---

**LangExtract processes long documents by splitting text into token-aware chunks, grouping them into batches, and executing parallel inference via ThreadPoolExecutor to maximize throughput while respecting LLM context limits.**

LangExtract is an open-source Python library developed by Google that extracts structured data from unstructured text using large language models. When processing long documents such as books or lengthy reports, the library implements a sophisticated three-stage pipeline that combines **chunking and parallel workers** to efficiently handle texts exceeding typical LLM token limits.

## The Three-Stage Processing Pipeline

### Stage 1: Token-Aware Chunking

The pipeline begins in [`langextract/chunking.py`](https://github.com/google/langextract/blob/main/langextract/chunking.py), where the `ChunkIterator` class greedily constructs text chunks that respect both sentence boundaries and the `max_char_buffer` limit. The algorithm iterates through tokenized sentences, accumulating text until adding another sentence would exceed the character buffer, then yields a `TextChunk` containing the accumulated content plus optional context from the previous chunk.

Key implementation details from [`langextract/chunking.py`](https://github.com/google/langextract/blob/main/langextract/chunking.py) lines 43-84:

- Uses `RegexTokenizer` by default to identify sentence boundaries
- Handles edge cases where a single token exceeds `max_char_buffer` by creating a single-token chunk
- Supports `context_window_chars` to prepend characters from the previous chunk for cross-chunk coherence

### Stage 2: Batch Formation

Once chunks are generated, the `Annotator` class in [`langextract/annotation.py`](https://github.com/google/langextract/blob/main/langextract/annotation.py) groups consecutive chunks into batches. The `_annotate_documents_single_pass` method (lines 58-84) collects chunks until reaching `batch_length`, then converts each chunk into a structured prompt using `PromptTemplateStructured`.

Batch formation serves two purposes:

1. **API Efficiency**: Grouping prompts allows providers to use batch APIs when available
2. **Memory Management**: Processing batches sequentially prevents memory exhaustion on extremely large documents

### Stage 3: Parallel Inference

The final stage executes prompts concurrently using worker pools. When a batch contains multiple prompts and `max_workers > 1`, providers spawn a `ThreadPoolExecutor` to parallelize inference.

Implementation in [`langextract/providers/gemini.py`](https://github.com/google/langextract/blob/main/langextract/providers/gemini.py) (lines 7-14):

```python

# Simplified representation of the parallel execution logic

with ThreadPoolExecutor(max_workers=min(self.max_workers, len(batch_prompts))) as executor:
    futures = [executor.submit(self._process_single_prompt, prompt) for prompt in batch_prompts]
    results = [f.result() for f in futures]

```

The OpenAI provider follows an identical pattern in [`langextract/providers/openai.py`](https://github.com/google/langextract/blob/main/langextract/providers/openai.py) (lines 7-14), ensuring consistent parallelism across different LLM backends.

## Configuration Parameters for Chunking and Parallelism

LangExtract exposes several parameters in the `extract()` function (defined in [`langextract/extraction.py`](https://github.com/google/langextract/blob/main/langextract/extraction.py)) to tune the pipeline:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `max_char_buffer` | 1000 | Maximum characters per chunk before splitting |
| `batch_length` | 10 | Number of chunks to group into a single batch |
| `max_workers` | 10 | Maximum parallel threads for inference |
| `context_window_chars` | 0 | Characters from previous chunk to prepend for context |
| `extraction_passes` | 1 | Number of times to run the entire pipeline over the text |

Note: When `batch_length < max_workers`, [`langextract/extraction.py`](https://github.com/google/langextract/blob/main/langextract/extraction.py) emits a warning (lines 3-8) because the configuration underutilizes available worker capacity.

## Practical Code Examples

### Basic Extraction with Explicit Chunking and Parallelism

```python
import langextract as lx
import textwrap

prompt = textwrap.dedent(
    """\
    Extract all character names and their emotions from the story.
    Return each extraction as a JSON object with fields "character" and "emotion".
    """
)

examples = [
    lx.data.ExampleData(
        text="ROMEO: But soft! What light through yonder window breaks?",
        extractions=[
            lx.data.Extraction(
                extraction_class="character",
                extraction_text="ROMEO",
                attributes={"emotion": "wonder"},
            )
        ],
    )
]

result = lx.extract(
    text_or_documents="https://www.gutenberg.org/files/1513/1513-0.txt",  # full Romeo & Juliet

    prompt_description=prompt,
    examples=examples,
    max_char_buffer=1200,   # each LLM call sees ≤ 1200 chars

    batch_length=20,        # group 20 chunks per batch

    max_workers=8,          # run up to 8 prompts in parallel

    extraction_passes=1,    # single pass (increase for higher recall)

    show_progress=False,
)

```

Under the hood, this configuration processes the entire play by downloading the text via `io.download_text_from_url`, creating approximately 10,000 chunks using `ChunkIterator` with 1200-character buffers, grouping chunks into batches of 20, and executing each batch with 8 parallel workers via `ThreadPoolExecutor`.

### Multi-Pass Extraction for Higher Recall

```python
result = lx.extract(
    text_or_documents=long_text,
    prompt_description=prompt,
    examples=examples,
    max_char_buffer=800,
    batch_length=15,
    max_workers=5,
    extraction_passes=3,   # run the whole pipeline three times

)

```

Each pass reprocesses the same chunks independently, allowing the model to discover entities missed in earlier iterations. The `Resolver` merges results across passes, preserving the first-found annotation for overlapping entities.

### Adding Cross-Chunk Context

```python
result = lx.extract(
    text_or_documents=long_text,
    prompt_description=prompt,
    examples=examples,
    max_char_buffer=1000,
    batch_length=10,
    max_workers=4,
    context_window_chars=200,  # prepend last 200 chars from previous chunk

)

```

The `Annotator` copies the final 200 characters from the preceding chunk into the current prompt, helping the model resolve anaphoric references like "she" or "it" that depend on earlier context.

## Key Source Files and Implementation Details

Understanding the internal architecture helps debug performance bottlenecks:

| File | Primary Responsibility | Notable Sections |
|------|------------------------|------------------|
| **[`langextract/chunking.py`](https://github.com/google/langextract/blob/main/langextract/chunking.py)** | Token-aware text segmentation | `ChunkIterator` class (lines 43-84) implements the greedy chunking algorithm with sentence boundary detection |
| **[`langextract/annotation.py`](https://github.com/google/langextract/blob/main/langextract/annotation.py)** | Batch orchestration and prompt assembly | `_annotate_documents_single_pass` (lines 58-84) groups chunks and manages the inference loop |
| **[`langextract/providers/gemini.py`](https://github.com/google/langextract/blob/main/langextract/providers/gemini.py)** | Gemini-specific parallel execution | Lines 7-14 contain the `ThreadPoolExecutor` logic that caps workers at `min(max_workers, len(batch_prompts))` |
| **[`langextract/providers/openai.py`](https://github.com/google/langextract/blob/main/langextract/providers/openai.py)** | OpenAI-specific parallel execution | Lines 7-14 mirror the Gemini pattern for consistent cross-provider parallelism |
| **[`langextract/extraction.py`](https://github.com/google/langextract/blob/main/langextract/extraction.py)** | Public API and configuration validation | Lines 3-8 emit warnings when `batch_length < max_workers`; lines 34-47 handle model instantiation |

These files implement the **"chunk → batch → parallel inference"** workflow that enables LangExtract to efficiently process megabytes of text while staying within LLM token limits.

## Summary

LangExtract efficiently processes long documents through a three-stage pipeline that balances memory constraints with parallel throughput:

- **Token-aware chunking** in [`langextract/chunking.py`](https://github.com/google/langextract/blob/main/langextract/chunking.py) splits documents into `max_char_buffer`-sized segments while preserving sentence boundaries, with optional `context_window_chars` for cross-chunk coherence.
- **Batch formation** in [`langextract/annotation.py`](https://github.com/google/langextract/blob/main/langextract/annotation.py) groups consecutive chunks into `batch_length` sized collections to optimize API usage and memory consumption.
- **Parallel inference** via `ThreadPoolExecutor` in both [`langextract/providers/gemini.py`](https://github.com/google/langextract/blob/main/langextract/providers/gemini.py) and [`langextract/providers/openai.py`](https://github.com/google/langextract/blob/main/langextract/providers/openai.py) executes up to `max_workers` prompts concurrently, automatically scaling down when batches are smaller than the worker limit.

Configure the pipeline using `max_char_buffer`, `batch_length`, `max_workers`, and `extraction_passes` to tune the trade-off between latency, cost, and recall when processing large-scale text corpora.

## Frequently Asked Questions

### What is the optimal batch_length to max_workers ratio for LangExtract?

For maximum throughput, set `batch_length` equal to or greater than `max_workers`. When `batch_length` is smaller than `max_workers`, the system emits a warning in [`langextract/extraction.py`](https://github.com/google/langextract/blob/main/langextract/extraction.py) because you have idle threads that cannot be utilized within a single batch. However, if memory constraints force small batches, the pipeline still functions correctly but underutilizes available parallelism.

### How does LangExtract handle chunks that exceed max_char_buffer?

The `ChunkIterator` in [`langextract/chunking.py`](https://github.com/google/langextract/blob/main/langextract/chunking.py) implements a greedy algorithm that checks token sizes before adding sentences to the current chunk. If a single token (sentence) exceeds `max_char_buffer`, the system creates a chunk containing only that token rather than splitting it mid-sentence, ensuring semantic coherence is preserved even with aggressive size constraints.

### Can I use different chunking strategies with LangExtract?

Currently, LangExtract uses the built-in `RegexTokenizer` and `ChunkIterator` defined in [`langextract/chunking.py`](https://github.com/google/langextract/blob/main/langextract/chunking.py). While the library does not expose pluggable chunking strategies in the public API, you can influence behavior through `max_char_buffer` and `context_window_chars` parameters. The `context_window_chars` option prepends previous context to each chunk, effectively creating overlapping sliding windows without changing the underlying segmentation logic.

### Does parallel execution work with both Gemini and OpenAI providers?

Yes, both providers implement identical parallel execution patterns. The `Gemini` class in [`langextract/providers/gemini.py`](https://github.com/google/langextract/blob/main/langextract/providers/gemini.py) and the `OpenAI` class in [`langextract/providers/openai.py`](https://github.com/google/langextract/blob/main/langextract/providers/openai.py) both use `ThreadPoolExecutor` to run `_process_single_prompt` across multiple threads when `max_workers > 1` and the batch contains multiple prompts. The executor size is automatically capped at `min(self.max_workers, len(batch_prompts))` to prevent spawning unnecessary threads.