How to Choose Between vLLM and HuggingFace Inference Modes in Chandra for Production

Use the method parameter in InferenceManager to select "vllm" for scalable, stateless production servers or "hf" for local, single-node inference.

The datalab-to/chandra repository provides dual inference back-ends that allow you to choose between vLLM and HuggingFace (HF) generation modes depending on your deployment requirements. Whether you need horizontal scaling for high-throughput OCR workloads or prefer a self-contained local deployment, understanding how to configure these modes is essential for production optimization.

Understanding the Two Inference Back-Ends in Chandra

Chandra implements an abstraction layer that lets you switch between two distinct inference strategies without changing your application code. The choice centers on the InferenceManager class located in chandra/model/__init__.py, which accepts a method argument of either "vllm" or "hf" during instantiation.

vLLM mode dispatches requests to a remote OpenAI-compatible HTTP server. When you select this method, Chandra uses generate_vllm from chandra/model/vllm.py to send image and prompt data over HTTP using the OpenAI client. This approach decouples the model binaries from your application host and enables GPU sharing across multiple clients.

HuggingFace mode loads the model locally using transformers and torch. When "hf" is specified, load_model() in chandra/model/hf.py loads the checkpoint defined by settings.MODEL_CHECKPOINT and attaches a processor, running generation in-process on your local hardware.

How to Configure vLLM and HuggingFace Modes

Programmatic Configuration via InferenceManager

To switch modes programmatically, instantiate InferenceManager with the appropriate method string. The constructor validates this parameter at lines 10-18 of chandra/model/__init__.py.

from chandra.model import InferenceManager

# Production: remote vLLM server

manager = InferenceManager(method="vllm")

# Local: in-process HuggingFace

manager = InferenceManager(method="hf")

The manager then dispatches to either generate_vllm or the HF equivalent based on this selection when you call manager.generate() (see lines 33-48 of the same file).

CLI Configuration

The command-line interface in chandra/scripts/cli.py exposes the --method flag with a default value of vllm. The main function at lines 24-31 passes this value directly to the InferenceManager constructor.

# Production deployment with vLLM

chandra-cli /data/documents ./output --method vllm --max-workers 32

# Local debugging with HuggingFace

chandra-cli /data/documents ./output --method hf --batch-size 1

Environment Variables and Settings

Configuration for both modes flows through chandra/settings.py. For vLLM deployments, critical variables include VLLM_API_BASE, VLLM_API_KEY, and VLLM_MODEL_NAME (lines 18-24). For HF mode, the system relies on MODEL_CHECKPOINT, TORCH_DEVICE, and TORCH_ATTN.

# .env configuration for vLLM production

VLLM_API_BASE="http://vllm-server:8000/v1"
VLLM_API_KEY="my-secret-token"
VLLM_MODEL_NAME="chandra"
VLLM_GPUS="0,1"
MAX_VLLM_RETRIES=6

Production Deployment Considerations

When choosing between vLLM and HuggingFace modes for production workloads, evaluate these factors:

Scalability: vLLM enables horizontal scaling by adding more vLLM workers or replica servers; your Chandra client remains lightweight and stateless. HF mode is limited to the resources of the host process and requires container replication or process-level parallelism to scale.

Latency: HuggingFace mode offers lower latency through direct in-process calls, though performance may degrade under GPU memory pressure. vLLM introduces one extra network hop (typically <10ms on LAN) plus server queue time.

Resource Isolation: With vLLM, model binaries reside on dedicated inference servers, allowing your application containers to run on CPU-only hosts. HF mode requires the model and application to share GPU/CPU resources, complicating resource budgeting.

Fault Tolerance: The vLLM client includes built-in retry logic with max_retries and max_failure_retries parameters. The _should_retry method at lines 104-132 of chandra/model/vllm.py automatically re-attempts requests on repeat token detection or server errors. HF generation fails with the raw exception from model.generate, requiring you to implement custom retry handling.

Deployment Complexity: vLLM requires a running endpoint (Docker container, Kubernetes pod, or dedicated server), while HF mode only needs a Python environment with torch and transformers installed.

Code Examples for Both Modes

Complete Python API Example

from pathlib import Path
from chandra.input import load_file
from chandra.model import InferenceManager
from chandra.model.schema import BatchInputItem

# Initialize with your chosen method

manager = InferenceManager(method="vllm")  # Change to "hf" for local inference

# Load document (returns list of PIL images)

images = load_file("document.pdf", {})

# Prepare batch items

batch = [
    BatchInputItem(image=img, prompt_type="ocr_layout")
    for img in images
]

# Execute inference

results = manager.generate(
    batch,
    include_images=True,
    include_headers_footers=False,
    max_output_tokens=8000,
)

Environment Configuration for Production vLLM

export VLLM_API_BASE="https://vllm.internal.company.com/v1"
export VLLM_API_KEY="sk-production-key"
export VLLM_MODEL_NAME="chandra-v1"
export MAX_VLLM_RETRIES=6
export VLLM_TIMEOUT=120

Local HuggingFace Setup

# For HF mode, ensure your settings are configured

from chandra import settings
settings.MODEL_CHECKPOINT = "datalab-to/chandra-base"
settings.TORCH_DEVICE = "cuda:0"

manager = InferenceManager(method="hf")

Summary

• Use method="vllm" in InferenceManager (located in chandra/model/__init__.py) for production deployments requiring scalability, resource isolation, and built-in fault tolerance through automatic retries.
• Use method="hf" for single-node deployments, debugging, or when you need to avoid managing a separate vLLM server infrastructure.
• Configure vLLM endpoints via environment variables in chandra/settings.py (VLLM_API_BASE, VLLM_API_KEY), while HF mode uses MODEL_CHECKPOINT and TORCH_DEVICE.
• The CLI exposes this choice through the --method flag defined in chandra/scripts/cli.py.

Frequently Asked Questions

How do I switch between vLLM and HuggingFace modes in Chandra?

Pass the method parameter when instantiating InferenceManager with either "vllm" or "hf" as the value. This selection determines whether the system uses remote HTTP inference via chandra/model/vllm.py or local generation via chandra/model/hf.py.

What are the performance differences between vLLM and HuggingFace modes in production?

HuggingFace mode provides lower baseline latency for single requests because it avoids network overhead, but it limits you to the GPU resources of a single host. vLLM mode adds a network hop but enables horizontal scaling across multiple GPU workers and keeps model weights off your application servers.

Does Chandra provide automatic retry logic for inference failures?

The vLLM client includes automatic retry mechanisms defined in the _should_retry method of chandra/model/vllm.py (lines 104-132), which handles server errors and repeat token detection. The HuggingFace mode does not include built-in retries; you must implement exception handling around model.generate calls yourself.

Which mode should I choose for high-throughput OCR processing?

Select vLLM mode for high-throughput production environments. It allows you to scale inference capacity independently from your application layer, supports concurrent request batching, and maintains fault isolation between the model server and your Chandra client processes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}