# What is MAX_VLLM_RETRIES in Chandra? Understanding vLLM Generation Retry Logic

> Discover MAX_VLLM_RETRIES in Chandra. Learn how this setting controls vLLM generation retries for repetitive output or service errors and optimize your LLM performance.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: internals
- Published: 2026-03-27

---

**MAX_VLLM_RETRIES** is a configurable integer (default = 6) that caps the number of automatic re‑invocations Chandra performs when its vLLM generation encounters repetitive output patterns or service errors.

In the `datalab-to/chandra` repository, **MAX_VLLM_RETRIES** governs the resilience of the document analysis pipeline. This setting ensures that transient failures and degenerate model loops do not block batch processing by allowing the system to back off and retry with adjusted sampling parameters.

## Global Configuration in settings.py

The retry limit is defined as a typed constant in the global settings module. In **[chandra/settings.py at line 23](https://github.com/datalab-to/chandra/blob/master/chandra/settings.py#L23)**, the value is declared:

```python
MAX_VLLM_RETRIES: int = 6

```

When the `generate_vllm` function is invoked without an explicit `max_retries` argument, it automatically falls back to this global default. The defaulting logic appears in **[chandra/model/vllm.py at lines 43‑45](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py#L43-L45)**:

```python
if max_retries is None:
    max_retries = settings.MAX_VLLM_RETRIES

```

## Retry Conditions and Logic

The actual decision to retry is encapsulated in the private helper `_should_retry`. According to the source code in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py), retries are triggered under two distinct conditions.

### Repeat Token Detection

The system scans the generated raw output for a special “repeat token” that signals the model is looping. The check is performed on the entire result and also on the final 50 characters to catch repeats that appear near the end of the generation. This logic resides in **[vllm.py at lines 104‑110](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py#L104-L110)**.

### vLLM Service Errors

Any exception raised during the OpenAI client call is caught, logged, and the resulting `GenerationResult` is marked with `error=True`. If the result carries this error flag, the system initiates a retry (subject to the remaining retry budget). This exception handling is implemented in **[vllm.py at lines 115‑121](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py#L115-L121)**.

## Advanced Retry Behavior

Each retry attempt modifies the sampling parameters to encourage more diverse completions and reduce the chance of re‑entering a loop:

- **Temperature** increases by `0.2` per attempt, capped at `0.8`.
- **Nucleus sampling** (`top_p`) is widened to `0.95`.

For error‑only retries, the code implements exponential back‑off. After each failed attempt, it sleeps for `2 × (attempt + 1)` seconds to reduce pressure on a potentially overloaded vLLM server, as seen in **[vllm.py at lines 118‑120](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py#L118-L120)**.

Additionally, callers may supply a separate `max_failure_retries` parameter. When provided, this value acts as a distinct ceiling for error‑only retries, allowing you to permit more repeat‑token retries than error retries if desired. This logic is handled in **[vllm.py at lines 122‑130](https://github.com/datalab-to/chandra/blob/master/chandra/model/vllm.py#L122-L130)**.

## Practical Usage Examples

Use the default retry limit of six without passing any extra arguments:

```python
from chandra.model.vllm import generate_vllm
from chandra.model.schema import BatchInputItem

batch = [
    BatchInputItem(
        image=my_pil_image,
        prompt="Extract the table data",
        prompt_type="TABLE"
    )
]

results = generate_vllm(batch)   # will retry up to 6 times if needed

print(results[0].raw)

```

Override the limits to allow more attempts for repeat‑token loops while restricting error retries:

```python
from chandra.model.vllm import generate_vllm

results = generate_vllm(
    batch,
    max_retries=10,            # up to 10 total attempts

    max_failure_retries=2      # at most 2 error‑only attempts

)

```

## Summary

- **MAX_VLLM_RETRIES** defaults to `6` in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) and serves as the global ceiling for generation retries.
- Retries trigger on **repeat‑token detection** in the model output or **exceptions** from the vLLM service.
- Each retry raises **temperature** by `0.2` (max `0.8`) and sets **top_p** to `0.95` to diversify output.
- Callers can override the default via the `max_retries` parameter and set a separate `max_failure_retries` limit for error‑specific handling.

## Frequently Asked Questions

### What is the default value of MAX_VLLM_RETRIES in Chandra?

The default value is `6`, defined as a typed integer constant in [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) at line 23. This value is imported by the generation pipeline and used when the caller does not supply an explicit `max_retries` argument.

### How does Chandra detect when to retry a vLLM generation?

The `_should_retry` helper in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) evaluates two conditions: it scans the raw output for a repeat token (checking both the full text and the last 50 characters), and it inspects the `GenerationResult` for an `error=True` flag set when the OpenAI client raises an exception.

### Can I configure different limits for error retries versus repeat token retries?

Yes. The `generate_vllm` function accepts both `max_retries` (total attempts) and `max_failure_retries` (error‑only attempts). By supplying both, you can allow, for example, up to ten attempts for repeat loops but only two attempts for service errors, as implemented in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) lines 122‑130.

### What sampling parameter adjustments occur during retries?

Each retry increments the **temperature** by `0.2` until it reaches a maximum of `0.8`, and it fixes **top_p** at `0.95`. These adjustments are designed to break repetitive generation patterns by encouraging more stochastic completions.