Supported Prompt Types in Chandra: How PROMPT_MAPPING Configures OCR Modes

Chandra supports two prompt types—ocr_layout and ocr—defined in chandra/prompts.py where the PROMPT_MAPPING dictionary maps each identifier to a specific prompt template for LLM inference.

The datalab-to/chandra repository provides an OCR framework that uses large language models to convert images to HTML. Understanding the supported prompt types in Chandra and how PROMPT_MAPPING is configured is essential for selecting the correct inference mode for your use case, whether you need simple text extraction or layout-aware HTML generation.

Supported Prompt Types in Chandra

Chandra currently defines two supported prompt identifiers in the PROMPT_MAPPING dictionary:

• ocr_layout: Performs OCR with layout analysis, returning HTML <div> blocks that include bounding-box metadata for precise document structure preservation.
• ocr: Performs plain OCR conversion, transforming an image directly into HTML markup without explicit layout metadata.

These identifiers are the only valid keys accepted by the inference backends when constructing a BatchInputItem.

How PROMPT_MAPPING Is Configured

The mapping between prompt identifiers and their corresponding templates lives in chandra/prompts.py. According to the datalab-to/chandra source code, the configuration is implemented as a simple dictionary that references prompt strings defined earlier in the same module:

# chandra/prompts.py

PROMPT_MAPPING = {
    "ocr_layout": OCR_LAYOUT_PROMPT,
    "ocr": OCR_PROMPT,
}

Both OCR_LAYOUT_PROMPT and OCR_PROMPT are string templates defined immediately above the mapping in chandra/prompts.py. These templates contain the specific instructions and formatting requirements sent to the underlying language model, including allowed HTML tags and attributes for the output structure.

How Inference Backends Use PROMPT_MAPPING

The PROMPT_MAPPING dictionary serves as the single source of truth for prompt resolution across Chandra's model implementations.

VLLM Backend (chandra/model/vllm.py)

In the VLLM inference backend, the system retrieves the appropriate prompt text by indexing into PROMPT_MAPPING using the prompt_type attribute of the input item:

prompt = PROMPT_MAPPING[item.prompt_type]

This lookup occurs during the request preparation phase before sending the image and prompt to the VLLM engine.

HuggingFace Backend (chandra/model/hf.py)

Similarly, the HuggingFace backend resolves prompt strings using the same dictionary:

prompt = PROMPT_MAPPING[prompt_type]

This ensures consistent prompt behavior regardless of which backend executes the inference.

Practical Examples: Selecting Prompt Types

When building an inference request, you explicitly set the prompt_type parameter when constructing a BatchInputItem. Here are the two supported configurations:

Layout-Aware OCR Request

To extract text while preserving document layout information, use the ocr_layout prompt type as implemented in chandra/scripts/screenshot_app.py:

from chandra.input import BatchInputItem

# img is a PIL Image or similar

item = BatchInputItem(
    image=img,
    prompt_type="ocr_layout",  # Maps to OCR_LAYOUT_PROMPT

)

result = model.predict(item)

Standard OCR Request

For straightforward image-to-HTML conversion without layout metadata, specify the ocr prompt type:

from chandra.input import BatchInputItem

item = BatchInputItem(
    image=img,
    prompt_type="ocr",  # Maps to OCR_PROMPT

)

result = model.predict(item)

Both examples appear throughout the codebase, including in chandra/scripts/cli.py and the integration tests in tests/integration/test_image_inference.py.

Summary

• Chandra supports exactly two prompt types: ocr_layout for layout-aware HTML generation and ocr for plain text extraction.
• The PROMPT_MAPPING dictionary in chandra/prompts.py maps these identifiers to their respective prompt templates (OCR_LAYOUT_PROMPT and OCR_PROMPT).
• Both the VLLM backend (chandra/model/vllm.py) and HuggingFace backend (chandra/model/hf.py) resolve prompts using PROMPT_MAPPING[item.prompt_type].
• You select the prompt type by setting the prompt_type parameter when creating a BatchInputItem instance.

Frequently Asked Questions

What prompt types does Chandra currently support?

Chandra supports two prompt types: ocr_layout for OCR with layout analysis (returning HTML with bounding-box metadata) and ocr for plain OCR (returning HTML without explicit layout information). These are defined in the PROMPT_MAPPING dictionary in chandra/prompts.py.

Where is the PROMPT_MAPPING dictionary defined?

The PROMPT_MAPPING dictionary is defined in chandra/prompts.py at the module level. It maps string identifiers like "ocr_layout" and "ocr" to their corresponding prompt template variables (OCR_LAYOUT_PROMPT and OCR_PROMPT) that contain the actual LLM instructions.

How do I select a prompt type when using Chandra?

You select a prompt type by passing the identifier string to the prompt_type parameter when constructing a BatchInputItem. For example: BatchInputItem(image=img, prompt_type="ocr_layout"). The inference backend then uses this key to look up the full prompt template in PROMPT_MAPPING.

Can I add custom prompt types to Chandra?

Currently, Chandra only supports the two predefined prompt types defined in chandra/prompts.py. To add custom prompt types, you would need to modify the source code to add your new prompt template variable to the file and register it in the PROMPT_MAPPING dictionary with a new unique key.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}