# How Chandra Handles vLLM Retries and Repeated Token Detection During Inference

> Learn how Chandra's vLLM wrapper handles inference retries and detects repeated tokens using a sliding window. It employs increasing temperature and back-off for robust generation.

- Repository: [Datalab/chandra](https://github.com/datalab-to/chandra)
- Tags: internals
- Published: 2026-03-27

---

**Chandra’s vLLM wrapper detects repeated tokens using a sliding-window heuristic in `detect_repeat_token()` and retries failed generations with linearly increasing temperature and sleep back-off until `max_retries` or `max_failure_retries` limits are reached.**

The `datalab-to/chandra` repository provides a robust Python inference pipeline for vision-language models built atop vLLM. When generating text from images, the system guards against both pathological repetition loops and transient API failures through a sophisticated retry mechanism. This article examines how Chandra handles vLLM retries and repeated token detection during inference, referencing the actual implementation in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) and [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py).

## Retry Logic Overview

Chandra implements a **dual-layer retry system** that operates inside the `generate_vllm` function. The wrapper distinguishes between two failure modes that trigger retries:

- **Content-based retries**: Triggered when `detect_repeat_token()` identifies repetitive patterns in the generated text
- **Error-based retries**: Triggered when the underlying OpenAI-compatible client raises an exception

Both paths converge in the `_should_retry` helper, which evaluates whether to attempt another generation based on current retry counts and configurable limits.

## Detecting Repeated Token Patterns

### The detect_repeat_token Heuristic

Located in [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py) (lines 68‑100), the `detect_repeat_token()` function implements a multi-stage algorithm to identify degenerate repetition loops before they contaminate the final output:

1. **Tail truncation**: Optionally removes a configurable number of characters from the string end (`cut_from_end`) to ignore benign trailing patterns
2. **Sliding window analysis**: Examines every possible sequence length up to `window_size / 2`
3. **Dynamic threshold calculation**: Computes allowed repeat counts using `base_max_repeats × (1 + scaling_factor / seq_len)`, making shorter sequences tolerable of fewer repetitions
4. **Consecutive counting**: Walks backward from the string end, counting how many times a candidate sequence occurs consecutively
5. **Early termination**: Returns `True` immediately upon detecting any sequence exceeding its calculated threshold

In [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) (lines 101‑108), the `_should_retry` method invokes this detector on `result.raw` after each generation attempt.

## Handling Transient API Errors

When the vLLM endpoint raises an exception, the `_generate` method (lines 87‑89) catches the failure and wraps it in a `GenerationResult` object with `error=True`. The `_should_retry` logic (lines 110‑127) then evaluates this flag independently of the repeat-detection path.

Error-driven retries implement a **linear back-off strategy**: `time.sleep(2 × (retries + 1))`. This ensures progressive cooling between attempts without exponential explosion, balancing responsiveness with API load management.

## Dynamic Sampling Adjustments on Retry

To break repetition cycles and avoid recurring errors, Chandra dynamically widens the sampling distribution on each retry attempt. Inside the `process_item` retry loop (lines 97‑101), the wrapper modifies generation parameters:

- **Temperature scaling**: Increases by `0.2 × (retries + 1)`, capped at a maximum of `0.8` to prevent complete randomness
- **Nucleus sampling**: Sets `top_p = 0.95` to encourage token diversity while maintaining coherence

These adjustments apply uniformly whether the retry was triggered by repeated tokens or API errors, ensuring the model explores alternative output paths.

## Configuration and Retry Limits

The `generate_vllm` function accepts two distinct retry ceilings defined at lines 43‑45:

- **`max_retries`**: The total retry budget for any reason (defaults to `settings.MAX_VLLM_RETRIES`)
- **`max_failure_retries`**: An optional, separate limit applied only to error-driven retries

The `_should_retry` method (lines 109‑131) evaluates both caps before permitting another attempt. Once either limit is exhausted, the system returns the current `GenerationResult` unchanged, preserving whatever partial or error-state output was obtained.

## Practical Implementation Example

The following example demonstrates how to invoke the inference pipeline with explicit retry configuration:

```python
from chandra.model.vllm import generate_vllm
from chandra.input import BatchInputItem
from PIL import Image

# Build a single-item batch

item = BatchInputItem(
    image=Image.open("sample.png"),
    prompt="Describe the content of the image.",
    prompt_type="default",   # resolves via PROMPT_MAPPING

)

# Run inference with custom retry limits

results = generate_vllm(
    batch=[item],
    max_output_tokens=256,
    max_retries=5,            # try up to 5 times on repeats

    max_failure_retries=2,    # extra attempts only for errors

    temperature=0.0,          # start deterministic

)

print(results[0].raw)   # Final (non-repeating) generation

```

Under the hood, this call executes the vLLM endpoint, inspects the output for repetition patterns, catches any exceptions, and automatically retries with adjusted sampling parameters until the constraints are satisfied or limits reached.

## Summary

- **Repeat detection** occurs via `detect_repeat_token()` in [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py), which scans output tails for consecutive repeating sequences using a dynamic threshold algorithm
- **API error handling** wraps exceptions in `GenerationResult` objects and applies linear sleep back-offs of `2 × (retries + 1)` seconds
- **Sampling adjustments** increase temperature by `0.2` per retry (capped at `0.8`) and fix `top_p` at `0.95` to diversify outputs
- **Retry budgets** are controlled by `max_retries` (general) and `max_failure_retries` (error-specific), evaluated in `_should_retry` at lines 109‑131 of [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py)
- **Configuration defaults** originate from [`chandra/settings.py`](https://github.com/datalab-to/chandra/blob/main/chandra/settings.py) via `MAX_VLLM_RETRIES`

## Frequently Asked Questions

### How does Chandra detect repeated tokens in vLLM output?

Chandra uses the `detect_repeat_token()` function in [`chandra/model/util.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/util.py) (lines 68‑100). The algorithm examines the tail of the generated text through a sliding window, calculating a dynamic repeat threshold based on sequence length, then counts consecutive occurrences walking backward from the end. If any pattern exceeds its allowed count, the function returns `True` and triggers a retry.

### What happens when the vLLM API throws an exception?

The `_generate` method in [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py) (lines 87‑89) catches the exception and returns a `GenerationResult` with `error=True`. The `_should_retry` logic (lines 110‑127) detects this flag and pauses execution for `2 × (retries + 1)` seconds before permitting another attempt, provided the `max_failure_retries` limit has not been exhausted.

### How does temperature scaling work during retries?

On each retry attempt, the temperature increases by `0.2 × (retries + 1)` with a hard ceiling of `0.8`, while `top_p` is set to `0.95`. This logic resides in the retry loop inside `process_item` (lines 97‑101 of [`chandra/model/vllm.py`](https://github.com/datalab-to/chandra/blob/main/chandra/model/vllm.py)), applying uniformly to both error-driven and repeat-detection retries.

### What is the difference between max_retries and max_failure_retries?

`max_retries` sets the absolute ceiling for all retry attempts regardless of cause, defaulting to `settings.MAX_VLLM_RETRIES`. `max_failure_retries` provides a secondary, optional cap that applies exclusively to error-driven retries, allowing stricter limits on API exception recovery while permitting more attempts to resolve repetition issues. Both are evaluated in `_should_retry` (lines 109‑131).