# MAX_OUTPUT_TOKENS in Chandra: Purpose, Default Value, and When to Adjust It

> Understand MAX_OUTPUT_TOKENS in Chandra. Learn its purpose, default value, and when to adjust this config constant to prevent context window overflows and resource exhaustion.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: deep-dive
- Published: 2026-03-27

---

**MAX_OUTPUT_TOKENS is a global configuration constant that caps the number of tokens the OCR model can generate per request, defaulting to 12,384 tokens to protect against context window overflows and resource exhaustion.**

In the `datalab-to/chandra` OCR framework, controlling generation length is critical for stable production deployments. The `MAX_OUTPUT_TOKENS` setting acts as a safety guardrail across both vLLM and Hugging Face backends, ensuring that single inference requests cannot consume unbounded resources or exceed model context limits.

## What Is MAX_OUTPUT_TOKENS?

### Default Value and Global Configuration

Located in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) at line 14, `MAX_OUTPUT_TOKENS` is set to **12,384 tokens** by default. This centralized constant provides a fallback value used whenever a caller does not explicitly specify a generation limit.

### Backend Integration

Both inference implementations read this setting when initializing generation parameters. In [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) (lines 49-51), the code assigns `max_output_tokens = settings.MAX_OUTPUT_TOKENS` when no explicit value is provided. Similarly, [`chandra/model/hf.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/hf.py) (lines 16-18) applies the same fallback logic for the Hugging Face backend.

## Why the Token Limit Matters

The token ceiling serves three protective functions:

- **Context Window Protection**: Most LLMs enforce a hard upper bound on total context length (input + output). Exceeding this bound results in truncated or failed responses.
- **Resource Conservation**: Longer generations consume disproportionate GPU memory, CPU time, and network bandwidth, directly increasing latency and operational cost.
- **Safety and Predictability**: Bounding output size prevents runaway generation loops (e.g., repetitive token sequences) that could otherwise hang the service indefinitely.

## When to Adjust MAX_OUTPUT_TOKENS

You should modify this default when specific workload characteristics demand it:

1. **Model Migration**: When switching to a model with a larger (or smaller) context window, raise or lower the limit to match the new architecture's capabilities.
2. **Complex Document Processing**: Multi-page PDFs or dense textual images may require output sequences longer than 12,384 tokens. Increase the limit only when you consistently encounter generation truncation.
3. **Performance Optimization**: If observed outputs are consistently shorter than the current ceiling and you face memory pressure or high latency, lowering the limit safely conserves resources.
4. **vLLM Server Alignment**: Ensure `MAX_OUTPUT_TOKENS` does not exceed any server-side token policies enforced by your vLLM deployment infrastructure.

## Practical Code Examples

The following examples demonstrate how to interact with the token limit using both backends. Both respect the global default unless explicitly overridden via the `max_output_tokens` parameter.

```python
from chandra.model.hf import generate_hf
from chandra.model.vllm import generate_vllm
from chandra.input import BatchInputItem

# Construct a batch for OCR processing

batch = [
    BatchInputItem(
        image=my_pil_image,
        prompt_type="ocr",
        prompt=None,
    )
]

# 1. Use the default MAX_OUTPUT_TOKENS (12,384)

default_results = generate_hf(batch, model=my_hf_model)

# 2. Request a shorter response for quick previews

short_results = generate_hf(
    batch, 
    model=my_hf_model, 
    max_output_tokens=1024
)

# 3. Request a longer response for dense documents via vLLM

long_results = generate_vllm(
    batch, 
    max_output_tokens=15000  # ensure this fits your model's context window

)

```

Note that explicit arguments take precedence over the global `MAX_OUTPUT_TOKENS` value defined in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py).

## Summary

- `MAX_OUTPUT_TOKENS` defaults to **12,384 tokens** in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) and applies globally across vLLM and Hugging Face backends.
- The limit protects against context window overflow, excessive resource consumption, and runaway generation loops.
- Adjust the value when changing models, processing atypical document types, optimizing for performance, or aligning with external vLLM constraints.
- Per-request overrides via `max_output_tokens` parameters in `generate_hf()` and `generate_vllm()` provide fine-grained control without modifying global configuration.

## Frequently Asked Questions

### What happens if I set max_output_tokens higher than the model's context window?

The request will likely fail or return truncated content. Both the vLLM and Hugging Face backends will attempt to generate up to the requested limit, but underlying model constraints will either trigger an error or silently truncate the output. Always ensure your `MAX_OUTPUT_TOKENS` value remains within the specific model's documented context length.

### Does lowering MAX_OUTPUT_TOKENS improve inference speed?

Yes, particularly for GPU-bound workloads. Shorter generation sequences reduce memory bandwidth requirements and decrease total generation time. If your OCR outputs consistently fall below 2,000 tokens, reducing the limit from 12,384 can yield measurable throughput gains without quality degradation.

### Can I set MAX_OUTPUT_TOKENS globally instead of per-request?

Yes. Modify the constant in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) to change the default for all subsequent inference calls that do not specify an explicit `max_output_tokens` argument. However, per-request overrides remain the recommended approach for production environments requiring heterogeneous document processing.

### Is there a minimum recommended value for MAX_OUTPUT_TOKENS?

There is no hardcoded minimum in the source, but practical OCR tasks rarely succeed below 256 tokens for single-line text or 512 tokens for structured documents. Setting the value too low risks truncating legitimate content before the model completes its description.