# How ThreadPoolExecutor in Chandra's vLLM Module Manages Concurrent Page Processing

> Discover how Chandra's vLLM module uses ThreadPoolExecutor to boost concurrent page processing Up to 64 worker threads optimize I/O bound HTTP requests while maintaining batch order

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: internals
- Published: 2026-03-27

---

**Chandra's vLLM integration processes multiple page images concurrently using Python's ThreadPoolExecutor, automatically scaling worker threads up to 64 to optimize I/O-bound HTTP requests while preserving batch order.**

Chandra, an open-source OCR-to-LLM pipeline hosted at `datalab-to/chandra`, leverages the `ThreadPoolExecutor` class to parallelize page processing when generating text from images via remote vLLM services. The implementation in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) creates a dedicated thread pool for each batch request, allowing multiple HTTP API calls to overlap while maintaining strict result ordering and implementing robust per-thread retry logic.

## Batch Preparation and Worker Configuration

The concurrency management begins in the `generate_vllm` function, which receives a `List[BatchInputItem]` where each item represents a page image and optional prompt (defined in [`chandra/model/schema.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/schema.py)).

Before processing, the system calculates the optimal number of worker threads. If the caller does not specify `max_workers`, the code automatically caps concurrency at 64 threads or the batch size, whichever is smaller:

```python
max_workers = min(64, len(batch))

```

The `ThreadPoolExecutor` is then instantiated as a context manager for the duration of the batch processing:

```python
with ThreadPoolExecutor(max_workers=max_workers) as executor:
    # Processing logic executes here

```

This design ensures that resources are cleaned up immediately after the batch completes, preventing thread leakage across multiple API calls.

## Per-Item Processing with ThreadPoolExecutor.map

Inside the executor context, the `process_item` helper function (defined in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py)) wraps the core generation logic for individual pages. This function calls `_generate` to build multipart HTTP requests, send them to the vLLM API, and parse JSON responses.

To distribute work across threads while passing consistent retry parameters, the implementation uses `executor.map` combined with `itertools.repeat`:

```python
from itertools import repeat

results = list(
    executor.map(
        process_item,
        batch,
        repeat(max_retries),
        repeat(max_failure_retries),
    )
)

```

The `map` method yields results **in the same order as the input batch**, ensuring that the first page in the input list corresponds to the first result in the output list. This ordering guarantee eliminates the need for manual index tracking when correlating images with their generated text.

## Thread-Level Retry and Robustness

Each thread executes independent retry logic within the `process_item` function, ensuring that a single problematic page does not block the entire batch. The implementation incorporates two distinct recovery mechanisms:

### Repeat-Token Detection

If the vLLM response contains repetitive or degenerate tokens, the thread automatically retries the request with an adjusted temperature parameter to increase response diversity.

### Network Error Recovery

For transient network failures or server errors, the thread executes exponential back-off pauses using `time.sleep(2 * (retries + 1))` before retrying. This delay increases with each attempt, preventing thundering herd scenarios against the remote vLLM service.

Both mechanisms execute entirely within the worker thread, isolating failures and allowing other pages to continue processing uninterrupted.

## Why ThreadPoolExecutor for I/O-Bound Workloads

The Chandra vLLM module chooses `ThreadPoolExecutor` over process-based parallelism because the workload is fundamentally **I/O-bound**. The dominant cost is the HTTP round-trip latency to the vLLM server, not CPU-intensive computation.

Threads efficiently overlap network waiting periods without incurring the serialization overhead or memory duplication associated with separate processes. The 64-thread cap provides sufficient parallelism to saturate network capacity while preventing resource exhaustion on either the client machine or the remote vLLM service.

## Practical Code Examples

### Basic Concurrent Processing

The following example demonstrates processing three page images concurrently without manual thread management:

```python
from chandra.model.vllm import generate_vllm
from chandra.model.schema import BatchInputItem
from PIL import Image

# Prepare a batch of images

batch = [
    BatchInputItem(image=Image.open("page1.png")),
    BatchInputItem(image=Image.open("page2.png")),
    BatchInputItem(image=Image.open("page3.png")),
]

# Process pages concurrently

results = generate_vllm(batch)

for i, res in enumerate(results, 1):
    print(f"--- Page {i} ---")
    print(res.raw)

```

### Customizing Concurrency and Retries

Control thread count and retry behavior by passing explicit parameters:

```python
results = generate_vllm(
    batch,
    max_workers=8,          # Limit to 8 concurrent threads

    max_retries=4,          # Retry degenerate responses up to 4 times

    max_failure_retries=2,  # Retry network errors up to 2 times

    temperature=0.0,
    top_p=0.1,
)

```

### Inspecting Per-Page Results

Each `GenerationResult` object (defined in [`chandra/model/schema.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/schema.py)) contains metadata about the generation attempt:

```python
for result in results:
    if result.error:
        print(f"Generation failed: {result.error}")
    else:
        print(f"Tokens used: {result.token_count}")
        print(result.raw[:200])  # Preview first 200 characters

```

## Summary

- **ThreadPoolExecutor Initialization**: Created once per `generate_vllm` call in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) with `max_workers` capped at `min(64, len(batch))`.
- **Ordered Results**: `executor.map` guarantees output order matches input batch order, simplifying result correlation.
- **I/O Optimization**: Thread-based concurrency efficiently handles HTTP latency to vLLM APIs without process overhead.
- **Isolated Retry Logic**: Per-thread retry mechanisms handle degenerate tokens and network errors independently, using exponential back-off strategies.
- **Resource Safety**: The executor context manager ensures threads are properly cleaned up after batch completion.

## Frequently Asked Questions

### How does Chandra maintain result order when processing pages concurrently?

Chandra uses `executor.map` rather than `executor.submit` to process the batch. According to the Python standard library documentation and the implementation in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py), `map` yields results in the same sequence as the input iterable, regardless of which thread finishes first. This allows the system to correlate each `GenerationResult` with its corresponding `BatchInputItem` by position alone.

### What is the maximum number of concurrent threads Chandra uses for vLLM processing?

By default, the system calculates `max_workers` as `min(64, len(batch))`, meaning it never spawns more than 64 threads regardless of batch size. Users can override this by passing a custom `max_workers` value to `generate_vllm`, but the 64-thread default protects both client resources and the remote vLLM service from excessive connection loads.

### How does retry logic work when a single page fails?

Each worker thread executes the `process_item` function, which contains isolated retry logic. If a page generates repetitive tokens, the thread retries with modified temperature settings. For network errors, the thread pauses execution using `time.sleep(2 * (retries + 1))` before attempting again. These retries occur within the individual thread, allowing other pages in the batch to continue processing without interruption.

### Why does Chandra use ThreadPoolExecutor instead of ProcessPoolExecutor?

The vLLM integration is **I/O-bound** rather than CPU-bound, as the primary bottleneck is waiting for HTTP responses from the remote API. Threads provide sufficient concurrency for overlapping network requests without the memory overhead and serialization costs associated with process pools. This approach maximizes throughput while minimizing resource consumption on the OCR processing machine.