MAX_OUTPUT_TOKENS in Chandra: Purpose, Default Value, and When to Adjust It

MAX_OUTPUT_TOKENS is a global configuration constant that caps the number of tokens the OCR model can generate per request, defaulting to 12,384 tokens to protect against context window overflows and resource exhaustion.

In the datalab-to/chandra OCR framework, controlling generation length is critical for stable production deployments. The MAX_OUTPUT_TOKENS setting acts as a safety guardrail across both vLLM and Hugging Face backends, ensuring that single inference requests cannot consume unbounded resources or exceed model context limits.

What Is MAX_OUTPUT_TOKENS?

Default Value and Global Configuration

Located in chandra/settings.py at line 14, MAX_OUTPUT_TOKENS is set to 12,384 tokens by default. This centralized constant provides a fallback value used whenever a caller does not explicitly specify a generation limit.

Backend Integration

Both inference implementations read this setting when initializing generation parameters. In chandra/model/vllm.py (lines 49-51), the code assigns max_output_tokens = settings.MAX_OUTPUT_TOKENS when no explicit value is provided. Similarly, chandra/model/hf.py (lines 16-18) applies the same fallback logic for the Hugging Face backend.

Why the Token Limit Matters

The token ceiling serves three protective functions:

• Context Window Protection: Most LLMs enforce a hard upper bound on total context length (input + output). Exceeding this bound results in truncated or failed responses.
• Resource Conservation: Longer generations consume disproportionate GPU memory, CPU time, and network bandwidth, directly increasing latency and operational cost.
• Safety and Predictability: Bounding output size prevents runaway generation loops (e.g., repetitive token sequences) that could otherwise hang the service indefinitely.

When to Adjust MAX_OUTPUT_TOKENS

You should modify this default when specific workload characteristics demand it:

1. Model Migration: When switching to a model with a larger (or smaller) context window, raise or lower the limit to match the new architecture's capabilities.
2. Complex Document Processing: Multi-page PDFs or dense textual images may require output sequences longer than 12,384 tokens. Increase the limit only when you consistently encounter generation truncation.
3. Performance Optimization: If observed outputs are consistently shorter than the current ceiling and you face memory pressure or high latency, lowering the limit safely conserves resources.
4. vLLM Server Alignment: Ensure MAX_OUTPUT_TOKENS does not exceed any server-side token policies enforced by your vLLM deployment infrastructure.

Practical Code Examples

The following examples demonstrate how to interact with the token limit using both backends. Both respect the global default unless explicitly overridden via the max_output_tokens parameter.

from chandra.model.hf import generate_hf
from chandra.model.vllm import generate_vllm
from chandra.input import BatchInputItem

# Construct a batch for OCR processing

batch = [
    BatchInputItem(
        image=my_pil_image,
        prompt_type="ocr",
        prompt=None,
    )
]

# 1. Use the default MAX_OUTPUT_TOKENS (12,384)

default_results = generate_hf(batch, model=my_hf_model)

# 2. Request a shorter response for quick previews

short_results = generate_hf(
    batch, 
    model=my_hf_model, 
    max_output_tokens=1024
)

# 3. Request a longer response for dense documents via vLLM

long_results = generate_vllm(
    batch, 
    max_output_tokens=15000  # ensure this fits your model's context window

)

Note that explicit arguments take precedence over the global MAX_OUTPUT_TOKENS value defined in chandra/settings.py.

Summary

• MAX_OUTPUT_TOKENS defaults to 12,384 tokens in chandra/settings.py and applies globally across vLLM and Hugging Face backends.
• The limit protects against context window overflow, excessive resource consumption, and runaway generation loops.
• Adjust the value when changing models, processing atypical document types, optimizing for performance, or aligning with external vLLM constraints.
• Per-request overrides via max_output_tokens parameters in generate_hf() and generate_vllm() provide fine-grained control without modifying global configuration.

Frequently Asked Questions

What happens if I set max_output_tokens higher than the model's context window?

The request will likely fail or return truncated content. Both the vLLM and Hugging Face backends will attempt to generate up to the requested limit, but underlying model constraints will either trigger an error or silently truncate the output. Always ensure your MAX_OUTPUT_TOKENS value remains within the specific model's documented context length.

Does lowering MAX_OUTPUT_TOKENS improve inference speed?

Yes, particularly for GPU-bound workloads. Shorter generation sequences reduce memory bandwidth requirements and decrease total generation time. If your OCR outputs consistently fall below 2,000 tokens, reducing the limit from 12,384 can yield measurable throughput gains without quality degradation.

Can I set MAX_OUTPUT_TOKENS globally instead of per-request?

Yes. Modify the constant in chandra/settings.py to change the default for all subsequent inference calls that do not specify an explicit max_output_tokens argument. However, per-request overrides remain the recommended approach for production environments requiring heterogeneous document processing.

Is there a minimum recommended value for MAX_OUTPUT_TOKENS?

There is no hardcoded minimum in the source, but practical OCR tasks rarely succeed below 256 tokens for single-line text or 512 tokens for structured documents. Setting the value too low risks truncating legitimate content before the model completes its description.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}