How the InferenceManager Class in Chandra Routes Requests to LLM Backends

The InferenceManager class routes requests to either VLLM or Hugging Face backends based on the method parameter, using a conditional branch in the generate method to delegate to specialized handlers in chandra/model/vllm.py or chandra/model/hf.py.

The InferenceManager in the datalab-to/chandra repository serves as a unified façade for document understanding tasks, abstracting the complexity of different LLM providers behind a single Python interface. This architectural pattern allows developers to switch between local Hugging Face inference and remote VLLM services without modifying downstream code.

Architecture Overview

The routing mechanism relies on a simple string identifier set during instantiation. The class supports two distinct backends: "vllm" for remote inference via HTTP APIs and "hf" for local PyTorch execution through the Transformers library.

Constructor Logic

In chandra/model/__init__.py (lines 10-18), the __init__ method validates the method argument and initializes the appropriate backend:

• When method="hf", the manager eagerly loads the Hugging Face model by calling load_model() from chandra.model.hf, binding the model weights to GPU memory during construction.
• When method="vllm" (the default), the manager remains stateless regarding model weights, deferring all inference to an external HTTP service accessed at generation time.

This design means the Hugging Face backend incurs an upfront loading cost during instantiation, while the VLLM backend initializes instantly but requires network latency per request.

Runtime Routing Logic

The core routing decision occurs in the generate method implemented in chandra/model/__init__.py. This method accepts a list of BatchInputItem objects and optional configuration overrides.

The generate Method

According to the source in chandra/model/__init__.py (lines 33-48), the method extracts common parameters such as include_images, include_headers_footers, and bbox_scale from the input kwargs. It then evaluates self.method to determine which backend handler to invoke.

VLLM Path

When self.method == "vllm", the manager calls generate_vllm (defined in chandra/model/vllm.py, referenced at lines 33-40 in the init file). This function:

1. Constructs an HTTP request to the VLLM OpenAI-compatible REST API
2. Uses settings.VLLM_API_BASE as the default endpoint (overridable via the vllm_api_base parameter)
3. Returns a list of GenerationResult objects parsed from the JSON response

The VLLM backend requires no local GPU memory for model weights, making it suitable for distributed deployments where inference clusters are separate from the application server.

Hugging Face Path

When self.method == "hf", the manager invokes generate_hf (implemented in chandra/model/hf.py, referenced at lines 42-48 in the init file). This path:

1. Executes the previously loaded Transformers pipeline locally in PyTorch
2. Processes images through the local GPU or CPU using the model loaded during __init__
3. Returns GenerationResult objects with identical schema to the VLLM path

Both paths converge on identical post-processing utilities from chandra.output, ensuring that BatchOutputItem results contain consistent markdown, HTML, and OCR chunk formats regardless of which backend performed the inference.

Configuration and Overrides

Global defaults reside in chandra/settings.py, including VLLM_API_BASE, VLLM_MODEL_NAME, and BBOX_SCALE. The generate method accepts these as optional kwargs, allowing per-request overrides without modifying global state or creating new manager instances.

Usage Examples

Using the Default VLLM Backend

from chandra.model import InferenceManager, BatchInputItem
from PIL import Image

# Prepare a single-page batch

batch = [
    BatchInputItem(
        image=Image.open("sample_page.png"),
        prompt=None,                # Use default prompt for the image type

        prompt_type="ocr",         # Or any supported prompt_type

    )
]

# Instantiate the manager (default method = "vllm")

manager = InferenceManager()          # equivalent to InferenceManager(method="vllm")

# Run inference

output = manager.generate(batch)

print(output[0].markdown)             # Markdown OCR result

Switching to the Hugging Face Backend

from chandra.model import InferenceManager, BatchInputItem
from PIL import Image

batch = [
    BatchInputItem(image=Image.open("sample_page.png"), prompt_type="ocr")
]

# Explicitly request the HF backend

manager = InferenceManager(method="hf")

# The HF model will be loaded once at construction

output = manager.generate(batch, include_images=True)

print(output[0].images)               # Dictionary of extracted images

Overriding VLLM Configuration Per Request

manager = InferenceManager(method="vllm")
output = manager.generate(
    batch,
    vllm_api_base="https://my-custom-vllm.example.com/v1",
    temperature=0.2,
)

Summary

• Instantiation-time routing: The method parameter in InferenceManager.__init__ determines whether to load Hugging Face models locally (lines 10-18 in chandra/model/__init__.py) or prepare for VLLM HTTP calls.
• Runtime delegation: The generate method uses a conditional branch (lines 33-48) to call either generate_vllm or generate_hf based on the stored self.method value.
• Uniform interface: Both backends return identical GenerationResult objects processed by chandra.output utilities into BatchOutputItem instances with consistent markdown and extraction formats.
• Configuration flexibility: Global settings from chandra/settings.py can be overridden per request via kwargs like vllm_api_base and bbox_scale.

Frequently Asked Questions

Can I switch backends after instantiating InferenceManager?

No, the backend selection is immutable after construction. The method attribute is set during __init__ and stored as an instance variable, determining which generation path is used for all subsequent generate calls. To use a different backend, you must create a new InferenceManager instance with the desired method parameter.

Where does the VLLM backend send HTTP requests?

By default, it uses the endpoint defined in settings.VLLM_API_BASE from chandra/settings.py. You can override this default for individual requests by passing the vllm_api_base keyword argument to the generate method, allowing you to route different requests to different VLLM clusters without restarting your application.

What is the performance difference between the two backends?

The Hugging Face backend eagerly loads the full model into GPU memory during initialization, incurring a significant startup delay but offering lower latency per request for batch processing. The VLLM backend initializes instantly but introduces network latency for each API call, making it suitable for scenarios where local GPU resources are limited or when leveraging optimized VLLM deployments with queuing capabilities.

How does InferenceManager handle output formatting consistently across backends?

Regardless of which backend generates the raw text, both generate_vllm and generate_hf return standardized GenerationResult objects defined in chandra/model/schema.py. The InferenceManager then processes these through shared utilities in chandra/output to produce BatchOutputItem instances containing markdown, HTML, and image extractions in a uniform structure, ensuring downstream code remains agnostic to the inference provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}