What is MAX_VLLM_RETRIES in Chandra? Understanding vLLM Generation Retry Logic

MAX_VLLM_RETRIES is a configurable integer (default = 6) that caps the number of automatic re‑invocations Chandra performs when its vLLM generation encounters repetitive output patterns or service errors.

In the datalab-to/chandra repository, MAX_VLLM_RETRIES governs the resilience of the document analysis pipeline. This setting ensures that transient failures and degenerate model loops do not block batch processing by allowing the system to back off and retry with adjusted sampling parameters.

Global Configuration in settings.py

The retry limit is defined as a typed constant in the global settings module. In chandra/settings.py at line 23, the value is declared:

MAX_VLLM_RETRIES: int = 6

When the generate_vllm function is invoked without an explicit max_retries argument, it automatically falls back to this global default. The defaulting logic appears in chandra/model/vllm.py at lines 43‑45:

if max_retries is None:
    max_retries = settings.MAX_VLLM_RETRIES

Retry Conditions and Logic

The actual decision to retry is encapsulated in the private helper _should_retry. According to the source code in chandra/model/vllm.py, retries are triggered under two distinct conditions.

Repeat Token Detection

The system scans the generated raw output for a special “repeat token” that signals the model is looping. The check is performed on the entire result and also on the final 50 characters to catch repeats that appear near the end of the generation. This logic resides in vllm.py at lines 104‑110.

vLLM Service Errors

Any exception raised during the OpenAI client call is caught, logged, and the resulting GenerationResult is marked with error=True. If the result carries this error flag, the system initiates a retry (subject to the remaining retry budget). This exception handling is implemented in vllm.py at lines 115‑121.

Advanced Retry Behavior

Each retry attempt modifies the sampling parameters to encourage more diverse completions and reduce the chance of re‑entering a loop:

• Temperature increases by 0.2 per attempt, capped at 0.8.
• Nucleus sampling (top_p) is widened to 0.95.

For error‑only retries, the code implements exponential back‑off. After each failed attempt, it sleeps for 2 × (attempt + 1) seconds to reduce pressure on a potentially overloaded vLLM server, as seen in vllm.py at lines 118‑120.

Additionally, callers may supply a separate max_failure_retries parameter. When provided, this value acts as a distinct ceiling for error‑only retries, allowing you to permit more repeat‑token retries than error retries if desired. This logic is handled in vllm.py at lines 122‑130.

Practical Usage Examples

Use the default retry limit of six without passing any extra arguments:

from chandra.model.vllm import generate_vllm
from chandra.model.schema import BatchInputItem

batch = [
    BatchInputItem(
        image=my_pil_image,
        prompt="Extract the table data",
        prompt_type="TABLE"
    )
]

results = generate_vllm(batch)   # will retry up to 6 times if needed

print(results[0].raw)

Override the limits to allow more attempts for repeat‑token loops while restricting error retries:

from chandra.model.vllm import generate_vllm

results = generate_vllm(
    batch,
    max_retries=10,            # up to 10 total attempts

    max_failure_retries=2      # at most 2 error‑only attempts

)

Summary

• MAX_VLLM_RETRIES defaults to 6 in chandra/settings.py and serves as the global ceiling for generation retries.
• Retries trigger on repeat‑token detection in the model output or exceptions from the vLLM service.
• Each retry raises temperature by 0.2 (max 0.8) and sets top_p to 0.95 to diversify output.
• Callers can override the default via the max_retries parameter and set a separate max_failure_retries limit for error‑specific handling.

Frequently Asked Questions

What is the default value of MAX_VLLM_RETRIES in Chandra?

The default value is 6, defined as a typed integer constant in chandra/settings.py at line 23. This value is imported by the generation pipeline and used when the caller does not supply an explicit max_retries argument.

How does Chandra detect when to retry a vLLM generation?

The _should_retry helper in chandra/model/vllm.py evaluates two conditions: it scans the raw output for a repeat token (checking both the full text and the last 50 characters), and it inspects the GenerationResult for an error=True flag set when the OpenAI client raises an exception.

Can I configure different limits for error retries versus repeat token retries?

Yes. The generate_vllm function accepts both max_retries (total attempts) and max_failure_retries (error‑only attempts). By supplying both, you can allow, for example, up to ten attempts for repeat loops but only two attempts for service errors, as implemented in chandra/model/vllm.py lines 122‑130.

What sampling parameter adjustments occur during retries?

Each retry increments the temperature by 0.2 until it reaches a maximum of 0.8, and it fixes top_p at 0.95. These adjustments are designed to break repetitive generation patterns by encouraging more stochastic completions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}