How Chandra Handles vLLM Retries and Repeated Token Detection During Inference

Chandra’s vLLM wrapper detects repeated tokens using a sliding-window heuristic in detect_repeat_token() and retries failed generations with linearly increasing temperature and sleep back-off until max_retries or max_failure_retries limits are reached.

The datalab-to/chandra repository provides a robust Python inference pipeline for vision-language models built atop vLLM. When generating text from images, the system guards against both pathological repetition loops and transient API failures through a sophisticated retry mechanism. This article examines how Chandra handles vLLM retries and repeated token detection during inference, referencing the actual implementation in chandra/model/vllm.py and chandra/model/util.py.

Retry Logic Overview

Chandra implements a dual-layer retry system that operates inside the generate_vllm function. The wrapper distinguishes between two failure modes that trigger retries:

• Content-based retries: Triggered when detect_repeat_token() identifies repetitive patterns in the generated text
• Error-based retries: Triggered when the underlying OpenAI-compatible client raises an exception

Both paths converge in the _should_retry helper, which evaluates whether to attempt another generation based on current retry counts and configurable limits.

Detecting Repeated Token Patterns

The detect_repeat_token Heuristic

Located in chandra/model/util.py (lines 68‑100), the detect_repeat_token() function implements a multi-stage algorithm to identify degenerate repetition loops before they contaminate the final output:

1. Tail truncation: Optionally removes a configurable number of characters from the string end (cut_from_end) to ignore benign trailing patterns
2. Sliding window analysis: Examines every possible sequence length up to window_size / 2
3. Dynamic threshold calculation: Computes allowed repeat counts using base_max_repeats × (1 + scaling_factor / seq_len), making shorter sequences tolerable of fewer repetitions
4. Consecutive counting: Walks backward from the string end, counting how many times a candidate sequence occurs consecutively
5. Early termination: Returns True immediately upon detecting any sequence exceeding its calculated threshold

In chandra/model/vllm.py (lines 101‑108), the _should_retry method invokes this detector on result.raw after each generation attempt.

Handling Transient API Errors

When the vLLM endpoint raises an exception, the _generate method (lines 87‑89) catches the failure and wraps it in a GenerationResult object with error=True. The _should_retry logic (lines 110‑127) then evaluates this flag independently of the repeat-detection path.

Error-driven retries implement a linear back-off strategy: time.sleep(2 × (retries + 1)). This ensures progressive cooling between attempts without exponential explosion, balancing responsiveness with API load management.

Dynamic Sampling Adjustments on Retry

To break repetition cycles and avoid recurring errors, Chandra dynamically widens the sampling distribution on each retry attempt. Inside the process_item retry loop (lines 97‑101), the wrapper modifies generation parameters:

• Temperature scaling: Increases by 0.2 × (retries + 1), capped at a maximum of 0.8 to prevent complete randomness
• Nucleus sampling: Sets top_p = 0.95 to encourage token diversity while maintaining coherence

These adjustments apply uniformly whether the retry was triggered by repeated tokens or API errors, ensuring the model explores alternative output paths.

Configuration and Retry Limits

The generate_vllm function accepts two distinct retry ceilings defined at lines 43‑45:

• max_retries: The total retry budget for any reason (defaults to settings.MAX_VLLM_RETRIES)
• max_failure_retries: An optional, separate limit applied only to error-driven retries

The _should_retry method (lines 109‑131) evaluates both caps before permitting another attempt. Once either limit is exhausted, the system returns the current GenerationResult unchanged, preserving whatever partial or error-state output was obtained.

Practical Implementation Example

The following example demonstrates how to invoke the inference pipeline with explicit retry configuration:

from chandra.model.vllm import generate_vllm
from chandra.input import BatchInputItem
from PIL import Image

# Build a single-item batch

item = BatchInputItem(
    image=Image.open("sample.png"),
    prompt="Describe the content of the image.",
    prompt_type="default",   # resolves via PROMPT_MAPPING

)

# Run inference with custom retry limits

results = generate_vllm(
    batch=[item],
    max_output_tokens=256,
    max_retries=5,            # try up to 5 times on repeats

    max_failure_retries=2,    # extra attempts only for errors

    temperature=0.0,          # start deterministic

)

print(results[0].raw)   # Final (non-repeating) generation

Under the hood, this call executes the vLLM endpoint, inspects the output for repetition patterns, catches any exceptions, and automatically retries with adjusted sampling parameters until the constraints are satisfied or limits reached.

Summary

• Repeat detection occurs via detect_repeat_token() in chandra/model/util.py, which scans output tails for consecutive repeating sequences using a dynamic threshold algorithm
• API error handling wraps exceptions in GenerationResult objects and applies linear sleep back-offs of 2 × (retries + 1) seconds
• Sampling adjustments increase temperature by 0.2 per retry (capped at 0.8) and fix top_p at 0.95 to diversify outputs
• Retry budgets are controlled by max_retries (general) and max_failure_retries (error-specific), evaluated in _should_retry at lines 109‑131 of chandra/model/vllm.py
• Configuration defaults originate from chandra/settings.py via MAX_VLLM_RETRIES

Frequently Asked Questions

How does Chandra detect repeated tokens in vLLM output?

Chandra uses the detect_repeat_token() function in chandra/model/util.py (lines 68‑100). The algorithm examines the tail of the generated text through a sliding window, calculating a dynamic repeat threshold based on sequence length, then counts consecutive occurrences walking backward from the end. If any pattern exceeds its allowed count, the function returns True and triggers a retry.

What happens when the vLLM API throws an exception?

The _generate method in chandra/model/vllm.py (lines 87‑89) catches the exception and returns a GenerationResult with error=True. The _should_retry logic (lines 110‑127) detects this flag and pauses execution for 2 × (retries + 1) seconds before permitting another attempt, provided the max_failure_retries limit has not been exhausted.

How does temperature scaling work during retries?

On each retry attempt, the temperature increases by 0.2 × (retries + 1) with a hard ceiling of 0.8, while top_p is set to 0.95. This logic resides in the retry loop inside process_item (lines 97‑101 of chandra/model/vllm.py), applying uniformly to both error-driven and repeat-detection retries.

What is the difference between max_retries and max_failure_retries?

max_retries sets the absolute ceiling for all retry attempts regardless of cause, defaulting to settings.MAX_VLLM_RETRIES. max_failure_retries provides a secondary, optional cap that applies exclusively to error-driven retries, allowing stricter limits on API exception recovery while permitting more attempts to resolve repetition issues. Both are evaluated in _should_retry (lines 109‑131).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}