# Chandra OCR --method vllm vs hf: Remote Server vs Local Model Inference

> Understand Chandra's -vllm and -hf methods. Discover remote vLLM inference versus local HuggingFace model execution for your OCR tasks.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: deep-dive
- Published: 2026-03-27

---

**The `--method vllm` option sends OCR requests to a remote vLLM server using an OpenAI-compatible API, while `--method hf` loads a HuggingFace model locally and runs inference in-process.**

The `datalab-to/chandra` repository provides a command-line interface for OCR that supports two distinct inference backends. Understanding the functional difference between the `--method vllm` and `--method hf` CLI options is essential for choosing the right deployment strategy based on your infrastructure and performance requirements.

## Execution Model Architecture

The primary distinction lies in where the neural network computation occurs.

### Remote vLLM Server Inference

When using `--method vllm`, Chandra acts as a client to a remote inference server. The `InferenceManager` class initializes with `self.model = None` (see lines 15-18 in [[`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py)](https://github.com/datalab-to/chandra/blob/master/chandra/model/__init__.py)) and forwards all generation requests to `generate_vllm` in [[`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py)](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py). This implementation creates an OpenAI-compatible client using `settings.VLLM_API_KEY` and `settings.VLLM_API_BASE` (defined in [[`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py)](https://github.com/datalab-to/chandra/blob/master/chandra/settings.py)), sending image-prompt pairs via HTTP requests.

### Local HuggingFace Model Inference

Conversely, `--method hf` triggers local model loading via `load_model()` (lines 73-101) in [[`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py)](https://github.com/datalab-to/chandra/blob/master/chandra/model/hf.py). The `InferenceManager` downloads and instantiates the checkpoint defined in `settings.MODEL_CHECKPOINT` directly into the Python process. Generation occurs through `generate_hf`, which constructs chat templates and executes `model.generate()` on local GPU or CPU hardware.

## Code Path and Dispatch Logic

The `InferenceManager.generate` method in [[`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py)](https://github.com/datalab-to/chandra/blob/master/chandra/model/__init__.py) serves as the central router.

For vLLM requests (lines 33-41), the manager calls `generate_vllm`, which implements a `ThreadPoolExecutor` for concurrent requests, exponential back-off retry logic, and repeat-token detection via `_should_retry` and `detect_repeat_token`.

For HF requests (lines 42-48), the manager calls `generate_hf`, which runs a single forward pass without internal retry mechanisms. Any error handling or retry logic must be implemented by the calling code.

## Configuration Differences and CLI Options

The [[`chandra/scripts/cli.py`](https://github.com/datalab-to/chandra/blob/main/chandra/scripts/cli.py)](https://github.com/datalab-to/chandra/blob/master/chandra/scripts/cli.py) file handles method-specific defaults and available flags.

**Batch Size Defaults:**

- `--method vllm`: Defaults to 28 pages per batch (lines 97-103)
- `--method hf`: Defaults to 1 page per batch

**Method-Specific Flags:**

- `vllm` supports `--max-workers` and `--max-retries` for controlling parallel execution and fault tolerance
- `hf` ignores these concurrency flags; it respects only `max_output_tokens` and image-related processing options

**Environment Requirements:**

- `vllm` requires a reachable vLLM endpoint and valid API credentials (`VLLM_API_KEY`)
- `hf` requires the optional `chandra-ocr[hf]` extra dependency (PyTorch and Transformers libraries)

## Practical Usage Examples

### Connecting to a Remote vLLM Server

```bash
chandra run /data/images output_dir --method vllm \
    --max-workers 8 --max-retries 3 \
    --include-images --save-html

```

Internally, this creates an `InferenceManager` with `method="vllm"`, leaving `self.model` as `None`. The `generate_vllm` function builds an OpenAI client and dispatches images concurrently, gathering results into `BatchOutputItem` objects.

### Running Local HuggingFace Inference

```bash
chandra run /data/pdfs output_dir --method hf \
    --include-images --save-html

```

This executes `load_model()` to instantiate the local checkpoint, then processes batches sequentially through `generate_hf`. The chat template is applied, inputs are tokenized, and `model.generate` produces outputs without network overhead.

## Summary

- **Remote vs Local**: `--method vllm` uses HTTP API calls to external servers; `--method hf` performs computation within the Python process
- **Concurrency**: vLLM supports thread-pool parallelism with configurable workers; HF runs single-threaded batch processing
- **Resilience**: vLLM includes exponential back-off and retry logic; HF requires external error handling
- **Dependencies**: vLLM needs only API credentials; HF requires PyTorch, Transformers, and sufficient local compute resources
- **Throughput**: vLLM defaults to 28-page batches suitable for server deployment; HF defaults to single-page processing optimized for local development

## Frequently Asked Questions

### Can I use --max-workers with the hf method?

No. The `--max-workers` and `--max-retries` flags are specific to the vLLM backend and are ignored when using `--method hf`. Local HuggingFace inference processes batches sequentially without internal parallelization, though you could implement external parallelization by running multiple CLI processes.

### What happens if the vLLM server is unreachable?

The `generate_vllm` implementation in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) includes robust retry logic with exponential back-off. It detects transient failures and repeat-token anomalies through the `_should_retry` helper, automatically resubmitting requests up to the limit specified by `--max-retries` before raising a final exception.

### Do I need to download models before using --method hf?

Yes. The first time you run with `--method hf`, the `load_model()` function (lines 73-101) in [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py) downloads the checkpoint defined in `settings.MODEL_CHECKPOINT` from the HuggingFace Hub if not already cached locally. Subsequent runs use the cached model weights.

### Which method offers better performance for large-scale OCR?

For production workloads, `--method vllm` typically provides superior throughput due to its default 28-page batching, parallel worker threads, and the ability to leverage optimized inference servers. The `--method hf` option is better suited for development, offline processing, or environments without network access to external APIs.