How detect_repeat_token Identifies Generation Failures in Chandra

The detect_repeat_token utility detects potential generation failures by scanning the tail of model outputs for excessive consecutive repetitions of any token sequence, using a dynamic threshold that scales inversely with sequence length.

Chandra's generation pipeline integrates with vLLM to produce text outputs, but large language models can occasionally enter repetitive loops that stall meaningful generation. Located in chandra/model/util.py (lines 68-99), the detect_repeat_token function provides the core detection mechanism that identifies these failure patterns by analyzing the raw token string for abnormal repetition at the tail end of the output.

How Repetition Signals Generation Failure

When a language model gets stuck in a generation loop, it emits the same token or short token sequence repeatedly instead of progressing toward a coherent completion. This behavior indicates a soft generation failure—the model has lost context or confidence and is recycling safe, repetitive patterns. Chandra treats these loops as recoverable errors, triggering automatic retries with adjusted sampling parameters to break the cycle.

The detect_repeat_token Algorithm

The detection logic implements a sliding-window analysis that examines candidate sequences at the end of the generated text. The function accepts predicted_tokens as a single concatenated string and optionally trims trailing characters via the cut_from_end parameter to ignore partially formed tokens.

Sliding-Window Sequence Extraction

For every possible sequence length seq_len ranging from 1 to window_size // 2 (default 250), the function extracts a candidate sequence from the string's tail:

candidate_seq = predicted_tokens[-seq_len:]

Starting just before this candidate, the function walks backwards through the string, comparing each block of size seq_len against candidate_seq. The loop counts consecutive matches until encountering a mismatch, yielding a repeat_count for that specific sequence length.

Dynamic Repeat Thresholds

Rather than using a fixed repeat limit, the function calculates a length-dependent threshold that allows short sequences more repetitions while flagging longer sequences sooner:

max_repeats = int(base_max_repeats * (1 + scaling_factor / seq_len))

Short patterns naturally occur more frequently in valid text, so they require higher repeat counts to trigger a failure signal. Longer repetitive patterns indicate more severe looping and are flagged after fewer consecutive appearances. If repeat_count exceeds max_repeats for any sequence length, the function immediately returns True; otherwise, it completes the scan and returns False.

Integration with Chandra's vLLM Wrapper

The generation wrapper in chandra/model/vllm.py (lines 104-112) integrates detect_repeat_token into its retry logic through the _should_retry helper. This wrapper calls the detector twice to maximize accuracy:

has_repeat = detect_repeat_token(result.raw) or (
    len(result.raw) > 50 and detect_repeat_token(result.raw, cut_from_end=50)
)

The first call checks the full raw output for obvious loops. The second call—conditional on output length exceeding 50 characters—trims the final 50 characters via cut_from_end=50 to avoid false positives from partially formed tokens at the generation boundary (lines 5-8). When has_repeat evaluates to True and the retry budget remains available, the pipeline logs the detection and regenerates the prompt with a slightly higher temperature to perturb the model's sampling distribution.

Practical Implementation Examples

Detecting Loops in Raw Output

You can use detect_repeat_token directly to validate model outputs before further processing:

from chandra.model.util import detect_repeat_token

# Simulated stuck generation repeating "hello "

output = "The answer is: hello hello hello hello hello "

if detect_repeat_token(output):
    print("Repeat loop detected – consider retrying.")
else:
    print("No repeat loop.")

This returns True because the short token sequence exceeds the dynamic repeat threshold for its length.

Automatic Retry Logic

When using Chandra's batch generation interface, the repeat detection operates transparently behind the generate_batch function:

from chandra.model.vllm import generate_batch

results = generate_batch(
    batch=["Explain quantum computing", "Define machine learning"],
    temperature=0.6,
    top_p=0.9,
    max_retries=3,
    max_failure_retries=2,
)

for r in results:
    if r.error:
        print("Generation failed after retries.")
    elif r.was_retried_for_repeat:
        print("Succeeded after recovering from repeat loop.")
    else:
        print("Generated:", r.raw)

The wrapper automatically increments temperature and re-invokes the model when detect_repeat_token identifies a loop, up to the specified max_retries limit.

Summary

• detect_repeat_token in chandra/model/util.py scans the tail of generated text for consecutive repetitions of any sequence length between 1 and 250 characters.
• Dynamic thresholds scale inversely with sequence length (max_repeats = int(base_max_repeats * (1 + scaling_factor / seq_len))), allowing short patterns more occurrences than long ones.
• Sliding-window analysis extracts candidate sequences from the string's end and counts backward matches to determine repeat density.
• Dual-pass detection in chandra/model/vllm.py checks both full outputs and trimmed versions (minus 50 characters) to eliminate edge-case false positives.
• Automatic recovery triggers temperature-adjusted retries when loops are detected, treating repetition as a soft failure recoverable through sampling perturbation.

Frequently Asked Questions

What triggers detect_repeat_token to return True?

The function returns True when any sequence at the tail of the output repeats consecutively beyond its calculated max_repeats threshold. This threshold varies by sequence length—shorter sequences require more repeats to trigger failure than longer ones. For example, a 5-character sequence repeating 20 times might pass, while a 50-character sequence repeating 5 times would fail.

Why does the function use a dynamic threshold instead of a fixed count?

Valid natural language contains legitimate short repetitions (words like "the" or "and"), so short sequences need higher repeat allowances to avoid false positives. Longer repetitive patterns almost always indicate model degeneration. The inverse scaling formula mathematically encodes this linguistic reality, tightening scrutiny as sequence length increases.

How does Chandra recover from detected repeat loops?

When detect_repeat_token identifies a loop, the vLLM wrapper in chandra/model/vllm.py treats this as a retry-eligible condition. If retries remain in the budget, the pipeline increases the sampling temperature and regenerates the completion. This perturbation typically breaks the model out of its local minimum, allowing diverse token selection on subsequent attempts.

What is the performance impact of this detection?

The algorithm operates in O(n×m) time where n is the window_size (default 250) and m is the string length being checked, though early exit on threshold breach prevents worst-case scenarios. Since detection runs only once per generation attempt and uses simple string slicing rather than tokenization, the overhead remains negligible compared to the GPU inference time of the vLLM backend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}