# How the InferenceManager Class in Chandra Routes Requests to LLM Backends

> Discover how Chandra's InferenceManager class routes requests to VLLM or Hugging Face LLM backends using its generate method. Learn about conditional branching and backend delegation.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: internals
- Published: 2026-03-27

---

**The InferenceManager class routes requests to either VLLM or Hugging Face backends based on the `method` parameter, using a conditional branch in the `generate` method to delegate to specialized handlers in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) or [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py).**

The `InferenceManager` in the [datalab-to/chandra](https://github.com/datalab-to/chandra) repository serves as a unified façade for document understanding tasks, abstracting the complexity of different LLM providers behind a single Python interface. This architectural pattern allows developers to switch between local Hugging Face inference and remote VLLM services without modifying downstream code.

## Architecture Overview

The routing mechanism relies on a simple string identifier set during instantiation. The class supports two distinct backends: `"vllm"` for remote inference via HTTP APIs and `"hf"` for local PyTorch execution through the Transformers library.

### Constructor Logic

In [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py) (lines 10-18), the `__init__` method validates the `method` argument and initializes the appropriate backend:

- When `method="hf"`, the manager eagerly loads the Hugging Face model by calling `load_model()` from `chandra.model.hf`, binding the model weights to GPU memory during construction.
- When `method="vllm"` (the default), the manager remains stateless regarding model weights, deferring all inference to an external HTTP service accessed at generation time.

This design means the Hugging Face backend incurs an upfront loading cost during instantiation, while the VLLM backend initializes instantly but requires network latency per request.

## Runtime Routing Logic

The core routing decision occurs in the `generate` method implemented in [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py). This method accepts a list of `BatchInputItem` objects and optional configuration overrides.

### The generate Method

According to the source in [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py) (lines 33-48), the method extracts common parameters such as `include_images`, `include_headers_footers`, and `bbox_scale` from the input kwargs. It then evaluates `self.method` to determine which backend handler to invoke.

### VLLM Path

When `self.method == "vllm"`, the manager calls `generate_vllm` (defined in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py), referenced at lines 33-40 in the init file). This function:

1. Constructs an HTTP request to the VLLM OpenAI-compatible REST API
2. Uses `settings.VLLM_API_BASE` as the default endpoint (overridable via the `vllm_api_base` parameter)
3. Returns a list of `GenerationResult` objects parsed from the JSON response

The VLLM backend requires no local GPU memory for model weights, making it suitable for distributed deployments where inference clusters are separate from the application server.

### Hugging Face Path

When `self.method == "hf"`, the manager invokes `generate_hf` (implemented in [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py), referenced at lines 42-48 in the init file). This path:

1. Executes the previously loaded Transformers pipeline locally in PyTorch
2. Processes images through the local GPU or CPU using the model loaded during `__init__`
3. Returns `GenerationResult` objects with identical schema to the VLLM path

Both paths converge on identical post-processing utilities from `chandra.output`, ensuring that `BatchOutputItem` results contain consistent markdown, HTML, and OCR chunk formats regardless of which backend performed the inference.

## Configuration and Overrides

Global defaults reside in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py), including `VLLM_API_BASE`, `VLLM_MODEL_NAME`, and `BBOX_SCALE`. The `generate` method accepts these as optional kwargs, allowing per-request overrides without modifying global state or creating new manager instances.

## Usage Examples

### Using the Default VLLM Backend

```python
from chandra.model import InferenceManager, BatchInputItem
from PIL import Image

# Prepare a single-page batch

batch = [
    BatchInputItem(
        image=Image.open("sample_page.png"),
        prompt=None,                # Use default prompt for the image type

        prompt_type="ocr",         # Or any supported prompt_type

    )
]

# Instantiate the manager (default method = "vllm")

manager = InferenceManager()          # equivalent to InferenceManager(method="vllm")

# Run inference

output = manager.generate(batch)

print(output[0].markdown)             # Markdown OCR result

```

### Switching to the Hugging Face Backend

```python
from chandra.model import InferenceManager, BatchInputItem
from PIL import Image

batch = [
    BatchInputItem(image=Image.open("sample_page.png"), prompt_type="ocr")
]

# Explicitly request the HF backend

manager = InferenceManager(method="hf")

# The HF model will be loaded once at construction

output = manager.generate(batch, include_images=True)

print(output[0].images)               # Dictionary of extracted images

```

### Overriding VLLM Configuration Per Request

```python
manager = InferenceManager(method="vllm")
output = manager.generate(
    batch,
    vllm_api_base="https://my-custom-vllm.example.com/v1",
    temperature=0.2,
)

```

## Summary

- **Instantiation-time routing**: The `method` parameter in `InferenceManager.__init__` determines whether to load Hugging Face models locally (lines 10-18 in [`chandra/model/__init__.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/__init__.py)) or prepare for VLLM HTTP calls.
- **Runtime delegation**: The `generate` method uses a conditional branch (lines 33-48) to call either `generate_vllm` or `generate_hf` based on the stored `self.method` value.
- **Uniform interface**: Both backends return identical `GenerationResult` objects processed by `chandra.output` utilities into `BatchOutputItem` instances with consistent markdown and extraction formats.
- **Configuration flexibility**: Global settings from [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) can be overridden per request via kwargs like `vllm_api_base` and `bbox_scale`.

## Frequently Asked Questions

### Can I switch backends after instantiating InferenceManager?

No, the backend selection is immutable after construction. The `method` attribute is set during `__init__` and stored as an instance variable, determining which generation path is used for all subsequent `generate` calls. To use a different backend, you must create a new `InferenceManager` instance with the desired `method` parameter.

### Where does the VLLM backend send HTTP requests?

By default, it uses the endpoint defined in `settings.VLLM_API_BASE` from [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py). You can override this default for individual requests by passing the `vllm_api_base` keyword argument to the `generate` method, allowing you to route different requests to different VLLM clusters without restarting your application.

### What is the performance difference between the two backends?

The Hugging Face backend eagerly loads the full model into GPU memory during initialization, incurring a significant startup delay but offering lower latency per request for batch processing. The VLLM backend initializes instantly but introduces network latency for each API call, making it suitable for scenarios where local GPU resources are limited or when leveraging optimized VLLM deployments with queuing capabilities.

### How does InferenceManager handle output formatting consistently across backends?

Regardless of which backend generates the raw text, both `generate_vllm` and `generate_hf` return standardized `GenerationResult` objects defined in [`chandra/model/schema.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/schema.py). The `InferenceManager` then processes these through shared utilities in `chandra/output` to produce `BatchOutputItem` instances containing markdown, HTML, and image extractions in a uniform structure, ensuring downstream code remains agnostic to the inference provider.