Chandra OCR --method vllm vs hf: Remote Server vs Local Model Inference

The --method vllm option sends OCR requests to a remote vLLM server using an OpenAI-compatible API, while --method hf loads a HuggingFace model locally and runs inference in-process.

The datalab-to/chandra repository provides a command-line interface for OCR that supports two distinct inference backends. Understanding the functional difference between the --method vllm and --method hf CLI options is essential for choosing the right deployment strategy based on your infrastructure and performance requirements.

Execution Model Architecture

The primary distinction lies in where the neural network computation occurs.

Remote vLLM Server Inference

When using --method vllm, Chandra acts as a client to a remote inference server. The InferenceManager class initializes with self.model = None (see lines 15-18 in [chandra/model/__init__.py](https://github.com/datalab-to/chandra/blob/master/chandra/model/__init__.py)) and forwards all generation requests to generate_vllm in [chandra/model/vllm.py](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py). This implementation creates an OpenAI-compatible client using settings.VLLM_API_KEY and settings.VLLM_API_BASE (defined in [chandra/settings.py](https://github.com/datalab-to/chandra/blob/master/chandra/settings.py)), sending image-prompt pairs via HTTP requests.

Local HuggingFace Model Inference

Conversely, --method hf triggers local model loading via load_model() (lines 73-101) in [chandra/model/hf.py](https://github.com/datalab-to/chandra/blob/master/chandra/model/hf.py). The InferenceManager downloads and instantiates the checkpoint defined in settings.MODEL_CHECKPOINT directly into the Python process. Generation occurs through generate_hf, which constructs chat templates and executes model.generate() on local GPU or CPU hardware.

Code Path and Dispatch Logic

The InferenceManager.generate method in [chandra/model/__init__.py](https://github.com/datalab-to/chandra/blob/master/chandra/model/__init__.py) serves as the central router.

For vLLM requests (lines 33-41), the manager calls generate_vllm, which implements a ThreadPoolExecutor for concurrent requests, exponential back-off retry logic, and repeat-token detection via _should_retry and detect_repeat_token.

For HF requests (lines 42-48), the manager calls generate_hf, which runs a single forward pass without internal retry mechanisms. Any error handling or retry logic must be implemented by the calling code.

Configuration Differences and CLI Options

The [chandra/scripts/cli.py](https://github.com/datalab-to/chandra/blob/master/chandra/scripts/cli.py) file handles method-specific defaults and available flags.

Batch Size Defaults:

• --method vllm: Defaults to 28 pages per batch (lines 97-103)
• --method hf: Defaults to 1 page per batch

Method-Specific Flags:

• vllm supports --max-workers and --max-retries for controlling parallel execution and fault tolerance
• hf ignores these concurrency flags; it respects only max_output_tokens and image-related processing options

Environment Requirements:

• vllm requires a reachable vLLM endpoint and valid API credentials (VLLM_API_KEY)
• hf requires the optional chandra-ocr[hf] extra dependency (PyTorch and Transformers libraries)

Practical Usage Examples

Connecting to a Remote vLLM Server

chandra run /data/images output_dir --method vllm \
    --max-workers 8 --max-retries 3 \
    --include-images --save-html

Internally, this creates an InferenceManager with method="vllm", leaving self.model as None. The generate_vllm function builds an OpenAI client and dispatches images concurrently, gathering results into BatchOutputItem objects.

Running Local HuggingFace Inference

chandra run /data/pdfs output_dir --method hf \
    --include-images --save-html

This executes load_model() to instantiate the local checkpoint, then processes batches sequentially through generate_hf. The chat template is applied, inputs are tokenized, and model.generate produces outputs without network overhead.

Summary

• Remote vs Local: --method vllm uses HTTP API calls to external servers; --method hf performs computation within the Python process
• Concurrency: vLLM supports thread-pool parallelism with configurable workers; HF runs single-threaded batch processing
• Resilience: vLLM includes exponential back-off and retry logic; HF requires external error handling
• Dependencies: vLLM needs only API credentials; HF requires PyTorch, Transformers, and sufficient local compute resources
• Throughput: vLLM defaults to 28-page batches suitable for server deployment; HF defaults to single-page processing optimized for local development

Frequently Asked Questions

Can I use --max-workers with the hf method?

No. The --max-workers and --max-retries flags are specific to the vLLM backend and are ignored when using --method hf. Local HuggingFace inference processes batches sequentially without internal parallelization, though you could implement external parallelization by running multiple CLI processes.

What happens if the vLLM server is unreachable?

The generate_vllm implementation in chandra/model/vllm.py includes robust retry logic with exponential back-off. It detects transient failures and repeat-token anomalies through the _should_retry helper, automatically resubmitting requests up to the limit specified by --max-retries before raising a final exception.

Do I need to download models before using --method hf?

Yes. The first time you run with --method hf, the load_model() function (lines 73-101) in chandra/model/hf.py downloads the checkpoint defined in settings.MODEL_CHECKPOINT from the HuggingFace Hub if not already cached locally. Subsequent runs use the cached model weights.

Which method offers better performance for large-scale OCR?

For production workloads, --method vllm typically provides superior throughput due to its default 28-page batching, parallel worker threads, and the ability to leverage optimized inference servers. The --method hf option is better suited for development, offline processing, or environments without network access to external APIs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}