How ThreadPoolExecutor in Chandra's vLLM Module Manages Concurrent Page Processing

Chandra's vLLM integration processes multiple page images concurrently using Python's ThreadPoolExecutor, automatically scaling worker threads up to 64 to optimize I/O-bound HTTP requests while preserving batch order.

Chandra, an open-source OCR-to-LLM pipeline hosted at datalab-to/chandra, leverages the ThreadPoolExecutor class to parallelize page processing when generating text from images via remote vLLM services. The implementation in chandra/model/vllm.py creates a dedicated thread pool for each batch request, allowing multiple HTTP API calls to overlap while maintaining strict result ordering and implementing robust per-thread retry logic.

Batch Preparation and Worker Configuration

The concurrency management begins in the generate_vllm function, which receives a List[BatchInputItem] where each item represents a page image and optional prompt (defined in chandra/model/schema.py).

Before processing, the system calculates the optimal number of worker threads. If the caller does not specify max_workers, the code automatically caps concurrency at 64 threads or the batch size, whichever is smaller:

max_workers = min(64, len(batch))

The ThreadPoolExecutor is then instantiated as a context manager for the duration of the batch processing:

with ThreadPoolExecutor(max_workers=max_workers) as executor:
    # Processing logic executes here

This design ensures that resources are cleaned up immediately after the batch completes, preventing thread leakage across multiple API calls.

Per-Item Processing with ThreadPoolExecutor.map

Inside the executor context, the process_item helper function (defined in chandra/model/vllm.py) wraps the core generation logic for individual pages. This function calls _generate to build multipart HTTP requests, send them to the vLLM API, and parse JSON responses.

To distribute work across threads while passing consistent retry parameters, the implementation uses executor.map combined with itertools.repeat:

from itertools import repeat

results = list(
    executor.map(
        process_item,
        batch,
        repeat(max_retries),
        repeat(max_failure_retries),
    )
)

The map method yields results in the same order as the input batch, ensuring that the first page in the input list corresponds to the first result in the output list. This ordering guarantee eliminates the need for manual index tracking when correlating images with their generated text.

Thread-Level Retry and Robustness

Each thread executes independent retry logic within the process_item function, ensuring that a single problematic page does not block the entire batch. The implementation incorporates two distinct recovery mechanisms:

Repeat-Token Detection

If the vLLM response contains repetitive or degenerate tokens, the thread automatically retries the request with an adjusted temperature parameter to increase response diversity.

Network Error Recovery

For transient network failures or server errors, the thread executes exponential back-off pauses using time.sleep(2 * (retries + 1)) before retrying. This delay increases with each attempt, preventing thundering herd scenarios against the remote vLLM service.

Both mechanisms execute entirely within the worker thread, isolating failures and allowing other pages to continue processing uninterrupted.

Why ThreadPoolExecutor for I/O-Bound Workloads

The Chandra vLLM module chooses ThreadPoolExecutor over process-based parallelism because the workload is fundamentally I/O-bound. The dominant cost is the HTTP round-trip latency to the vLLM server, not CPU-intensive computation.

Threads efficiently overlap network waiting periods without incurring the serialization overhead or memory duplication associated with separate processes. The 64-thread cap provides sufficient parallelism to saturate network capacity while preventing resource exhaustion on either the client machine or the remote vLLM service.

Practical Code Examples

Basic Concurrent Processing

The following example demonstrates processing three page images concurrently without manual thread management:

from chandra.model.vllm import generate_vllm
from chandra.model.schema import BatchInputItem
from PIL import Image

# Prepare a batch of images

batch = [
    BatchInputItem(image=Image.open("page1.png")),
    BatchInputItem(image=Image.open("page2.png")),
    BatchInputItem(image=Image.open("page3.png")),
]

# Process pages concurrently

results = generate_vllm(batch)

for i, res in enumerate(results, 1):
    print(f"--- Page {i} ---")
    print(res.raw)

Customizing Concurrency and Retries

Control thread count and retry behavior by passing explicit parameters:

results = generate_vllm(
    batch,
    max_workers=8,          # Limit to 8 concurrent threads

    max_retries=4,          # Retry degenerate responses up to 4 times

    max_failure_retries=2,  # Retry network errors up to 2 times

    temperature=0.0,
    top_p=0.1,
)

Inspecting Per-Page Results

Each GenerationResult object (defined in chandra/model/schema.py) contains metadata about the generation attempt:

for result in results:
    if result.error:
        print(f"Generation failed: {result.error}")
    else:
        print(f"Tokens used: {result.token_count}")
        print(result.raw[:200])  # Preview first 200 characters

Summary

• ThreadPoolExecutor Initialization: Created once per generate_vllm call in chandra/model/vllm.py with max_workers capped at min(64, len(batch)).
• Ordered Results: executor.map guarantees output order matches input batch order, simplifying result correlation.
• I/O Optimization: Thread-based concurrency efficiently handles HTTP latency to vLLM APIs without process overhead.
• Isolated Retry Logic: Per-thread retry mechanisms handle degenerate tokens and network errors independently, using exponential back-off strategies.
• Resource Safety: The executor context manager ensures threads are properly cleaned up after batch completion.

Frequently Asked Questions

How does Chandra maintain result order when processing pages concurrently?

Chandra uses executor.map rather than executor.submit to process the batch. According to the Python standard library documentation and the implementation in chandra/model/vllm.py, map yields results in the same sequence as the input iterable, regardless of which thread finishes first. This allows the system to correlate each GenerationResult with its corresponding BatchInputItem by position alone.

What is the maximum number of concurrent threads Chandra uses for vLLM processing?

By default, the system calculates max_workers as min(64, len(batch)), meaning it never spawns more than 64 threads regardless of batch size. Users can override this by passing a custom max_workers value to generate_vllm, but the 64-thread default protects both client resources and the remote vLLM service from excessive connection loads.

How does retry logic work when a single page fails?

Each worker thread executes the process_item function, which contains isolated retry logic. If a page generates repetitive tokens, the thread retries with modified temperature settings. For network errors, the thread pauses execution using time.sleep(2 * (retries + 1)) before attempting again. These retries occur within the individual thread, allowing other pages in the batch to continue processing without interruption.

Why does Chandra use ThreadPoolExecutor instead of ProcessPoolExecutor?

The vLLM integration is I/O-bound rather than CPU-bound, as the primary bottleneck is waiting for HTTP responses from the remote API. Threads provide sufficient concurrency for overlapping network requests without the memory overhead and serialization costs associated with process pools. This approach maximizes throughput while minimizing resource consumption on the OCR processing machine.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}