# How to Choose Between vLLM and HuggingFace Inference Modes in Chandra for Production

> Choose between vLLM and HuggingFace inference in Chandra for production. Use the method parameter to select vLLM for scalable servers or HF for local inference.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: best-practices
- Published: 2026-03-27

---

**Use the `method` parameter in `InferenceManager` to select `"vllm"` for scalable, stateless production servers or `"hf"` for local, single-node inference.**

The `datalab-to/chandra` repository provides dual inference back-ends that allow you to choose between vLLM and HuggingFace (HF) generation modes depending on your deployment requirements. Whether you need horizontal scaling for high-throughput OCR workloads or prefer a self-contained local deployment, understanding how to configure these modes is essential for production optimization.

## Understanding the Two Inference Back-Ends in Chandra

Chandra implements an abstraction layer that lets you switch between two distinct inference strategies without changing your application code. The choice centers on the **InferenceManager** class located in [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py), which accepts a `method` argument of either `"vllm"` or `"hf"` during instantiation.

**vLLM mode** dispatches requests to a remote OpenAI-compatible HTTP server. When you select this method, Chandra uses `generate_vllm` from [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) to send image and prompt data over HTTP using the `OpenAI` client. This approach decouples the model binaries from your application host and enables GPU sharing across multiple clients.

**HuggingFace mode** loads the model locally using `transformers` and `torch`. When `"hf"` is specified, `load_model()` in [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py) loads the checkpoint defined by `settings.MODEL_CHECKPOINT` and attaches a processor, running generation in-process on your local hardware.

## How to Configure vLLM and HuggingFace Modes

### Programmatic Configuration via InferenceManager

To switch modes programmatically, instantiate `InferenceManager` with the appropriate `method` string. The constructor validates this parameter at lines 10-18 of [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py).

```python
from chandra.model import InferenceManager

# Production: remote vLLM server

manager = InferenceManager(method="vllm")

# Local: in-process HuggingFace

manager = InferenceManager(method="hf")

```

The manager then dispatches to either `generate_vllm` or the HF equivalent based on this selection when you call `manager.generate()` (see lines 33-48 of the same file).

### CLI Configuration

The command-line interface in [`chandra/scripts/cli.py`](https://github.com/datalab-to/chandra/blob/main/chandra/scripts/cli.py) exposes the `--method` flag with a default value of `vllm`. The `main` function at lines 24-31 passes this value directly to the `InferenceManager` constructor.

```bash

# Production deployment with vLLM

chandra-cli /data/documents ./output --method vllm --max-workers 32

# Local debugging with HuggingFace

chandra-cli /data/documents ./output --method hf --batch-size 1

```

### Environment Variables and Settings

Configuration for both modes flows through [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py). For vLLM deployments, critical variables include `VLLM_API_BASE`, `VLLM_API_KEY`, and `VLLM_MODEL_NAME` (lines 18-24). For HF mode, the system relies on `MODEL_CHECKPOINT`, `TORCH_DEVICE`, and `TORCH_ATTN`.

```bash

# .env configuration for vLLM production

VLLM_API_BASE="http://vllm-server:8000/v1"
VLLM_API_KEY="my-secret-token"
VLLM_MODEL_NAME="chandra"
VLLM_GPUS="0,1"
MAX_VLLM_RETRIES=6

```

## Production Deployment Considerations

When choosing between **vLLM** and **HuggingFace** modes for production workloads, evaluate these factors:

**Scalability:** vLLM enables horizontal scaling by adding more vLLM workers or replica servers; your Chandra client remains lightweight and stateless. HF mode is limited to the resources of the host process and requires container replication or process-level parallelism to scale.

**Latency:** HuggingFace mode offers lower latency through direct in-process calls, though performance may degrade under GPU memory pressure. vLLM introduces one extra network hop (typically <10ms on LAN) plus server queue time.

**Resource Isolation:** With vLLM, model binaries reside on dedicated inference servers, allowing your application containers to run on CPU-only hosts. HF mode requires the model and application to share GPU/CPU resources, complicating resource budgeting.

**Fault Tolerance:** The vLLM client includes built-in retry logic with `max_retries` and `max_failure_retries` parameters. The `_should_retry` method at lines 104-132 of [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) automatically re-attempts requests on repeat token detection or server errors. HF generation fails with the raw exception from `model.generate`, requiring you to implement custom retry handling.

**Deployment Complexity:** vLLM requires a running endpoint (Docker container, Kubernetes pod, or dedicated server), while HF mode only needs a Python environment with `torch` and `transformers` installed.

## Code Examples for Both Modes

### Complete Python API Example

```python
from pathlib import Path
from chandra.input import load_file
from chandra.model import InferenceManager
from chandra.model.schema import BatchInputItem

# Initialize with your chosen method

manager = InferenceManager(method="vllm")  # Change to "hf" for local inference

# Load document (returns list of PIL images)

images = load_file("document.pdf", {})

# Prepare batch items

batch = [
    BatchInputItem(image=img, prompt_type="ocr_layout")
    for img in images
]

# Execute inference

results = manager.generate(
    batch,
    include_images=True,
    include_headers_footers=False,
    max_output_tokens=8000,
)

```

### Environment Configuration for Production vLLM

```bash
export VLLM_API_BASE="https://vllm.internal.company.com/v1"
export VLLM_API_KEY="sk-production-key"
export VLLM_MODEL_NAME="chandra-v1"
export MAX_VLLM_RETRIES=6
export VLLM_TIMEOUT=120

```

### Local HuggingFace Setup

```python

# For HF mode, ensure your settings are configured

from chandra import settings
settings.MODEL_CHECKPOINT = "datalab-to/chandra-base"
settings.TORCH_DEVICE = "cuda:0"

manager = InferenceManager(method="hf")

```

## Summary

- **Use `method="vllm"`** in `InferenceManager` (located in [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py)) for production deployments requiring scalability, resource isolation, and built-in fault tolerance through automatic retries.
- **Use `method="hf"`** for single-node deployments, debugging, or when you need to avoid managing a separate vLLM server infrastructure.
- Configure vLLM endpoints via environment variables in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) (`VLLM_API_BASE`, `VLLM_API_KEY`), while HF mode uses `MODEL_CHECKPOINT` and `TORCH_DEVICE`.
- The CLI exposes this choice through the `--method` flag defined in [`chandra/scripts/cli.py`](https://github.com/datalab-to/chandra/blob/main/chandra/scripts/cli.py).

## Frequently Asked Questions

### How do I switch between vLLM and HuggingFace modes in Chandra?

Pass the `method` parameter when instantiating `InferenceManager` with either `"vllm"` or `"hf"` as the value. This selection determines whether the system uses remote HTTP inference via [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) or local generation via [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py).

### What are the performance differences between vLLM and HuggingFace modes in production?

HuggingFace mode provides lower baseline latency for single requests because it avoids network overhead, but it limits you to the GPU resources of a single host. vLLM mode adds a network hop but enables horizontal scaling across multiple GPU workers and keeps model weights off your application servers.

### Does Chandra provide automatic retry logic for inference failures?

The vLLM client includes automatic retry mechanisms defined in the `_should_retry` method of [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) (lines 104-132), which handles server errors and repeat token detection. The HuggingFace mode does not include built-in retries; you must implement exception handling around `model.generate` calls yourself.

### Which mode should I choose for high-throughput OCR processing?

Select **vLLM mode** for high-throughput production environments. It allows you to scale inference capacity independently from your application layer, supports concurrent request batching, and maintains fault isolation between the model server and your Chandra client processes.