How to Debug Self-Eval Failures for Visual Jumps or Audio Pops at Cut Boundaries in video-use

To debug self-eval failures in video-use, inspect the 30ms audio fade parameters in extract_segment(), verify HDR tone-mapping consistency across cuts, and validate overlay PTS timing in build_final_composite(), then manually run timeline_view.py on suspect boundaries to isolate frame discontinuities or audio clicks.

The video-use repository renders final videos from an EDL (Edit Decision List) using a three-phase pipeline that includes a self-evaluation loop. When the LLM-driven self-eval reports visual jumps or audio pops at cut boundaries, you need to systematically trace the issue through the render pipeline's extraction, concatenation, and overlay logic. This guide walks through the specific source code locations and debugging techniques to resolve these failures.

Understanding the Self-Eval Pipeline

The rendering pipeline in video-use processes cuts through three distinct phases before self-evaluation:

• Per-segment extraction – Each cut range is trimmed, color-graded, and receives a 30ms audio fade-in/out (afade) to prevent pops. This logic resides in helpers/render.py within the extract_segment() function (lines 61-90).
• Lossless concatenation – Trimmed clips are joined using FFmpeg's concat demuxer (ffmpeg -c copy) via the concat_segments() function (lines 66-82).
• Self-evaluation loop – The pipeline runs timeline_view on the rendered output at every cut boundary, generating PNGs for LLM inspection. If visual jumps or audio artefacts are detected, the system re-renders the offending segment up to three passes (as defined in SKILL.md rule 99) before surfacing the error to the user.

Common Causes of Visual Jumps at Cut Boundaries

Visual discontinuities typically stem from mismatched color spaces or incorrect timestamp calculations between adjacent segments.

HDR Tone-Mapping Mismatches

When extract_segment() detects an HDR source via is_hdr_source(), it applies a TONEMAP_CHAIN to convert to SDR. If the detection fails or the tone-map chain is omitted, adjacent clips may exhibit brightness or color shifts at the cut boundary.

Check the HDR detection output by printing out.stdout.strip() before the return statement in is_hdr_source():

def is_hdr_source(video: Path) -> bool:
    try:
        out = subprocess.run(
            ["ffprobe", "-v", "error", "-select_streams", "v:0",
             "-show_entries", "stream=color_transfer",
             "-of", "default=noprint_wrappers=1:nokey=1", str(video)],
            capture_output=True, text=True, check=True,
        )
        hdr = out.stdout.strip() in HDR_TRANSFERS
        print(f"HDR check for {video.name}: {out.stdout.strip()} → {hdr}")
        return hdr
    except subprocess.CalledProcessError:
        return False

Overlay PTS Timing Errors

In build_final_composite(), overlays use setpts filters to align with the timeline. A misconfigured PTS shift causes frame discontinuities where overlays meet primary content.

The filter construction uses:

filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

Verify that t (the overlay start time) is expressed in seconds, not frames. Using frame counts instead of seconds here creates a visual jump at the overlay boundary.

Common Causes of Audio Pops at Cut Boundaries

Audio clicks occur when the 30ms fade buffer is insufficient or when sample-rate conversion introduces artefacts during concatenation.

Insufficient Fade Durations

The extract_segment() function applies dual afade filters (lines 87-90):

fade_out_start = max(0.0, duration - 0.03)
af = f"afade=t=in:st=0:d=0.03,afade=t=out:st={fade_out_start:.3f}:d=0.03"

If duration is less than 60ms, the out-fade starts before the in-fade completes, generating a click. For segments shorter than 60ms, modify the fade calculation:

# Insert before the fade definition in extract_segment()

if duration < 0.06:          # less than 60ms total

    # shrink both fades to half the segment length

    fade_len = duration / 2
    af = f"afade=t=in:st=0:d={fade_len:.3f},afade=t=out:st={fade_len:.3f}:d={fade_len:.3f}"
else:
    fade_out_start = max(0.0, duration - 0.03)
    af = f"afade=t=in:st=0:d=0.03,afade=t=out:st={fade_out_start:.3f}:d=0.03"

Sample-Rate Conversion Artefacts

While the 30ms fade handles most continuity issues, sample-rate mismatches between source files can introduce pops during the lossless concatenation phase. Ensure all source files use consistent sample rates before they reach concat_segments().

Step-by-Step Debugging Workflow

Follow this systematic approach to isolate self-eval failures.

Enable Verbose Logging

Add print statements before FFmpeg calls or set quiet=False in the run() helper to view full commands. The helper already prints the first six arguments; removing the quiet flag reveals the complete filter chain.

Validate Audio Fade Parameters

Verify that duration exceeds 60ms before applying the standard 30ms fades. Check the af string construction in helpers/render.py around line 87 to confirm fade boundaries do not overlap.

Verify HDR Detection and Scaling

Print the ffprobe output for color transfer characteristics in is_hdr_source(). Ensure HDR sources trigger the TONEMAP_CHAIN in extract_segment() to prevent brightness jumps between HDR and SDR segments.

Check Overlay Synchronization

Inspect the setpts calculation in build_final_composite(). Confirm that overlay start times (t) align with the EDL entries and use seconds-based timestamps. Misaligned overlays create apparent jumps at cut boundaries.

Manual Timeline Inspection

Invoke the self-eval visualization tool directly on suspect cuts:

python helpers/timeline_view.py final.mp4 --time 12.34

This generates a PNG showing frames before and after the cut at 12.34 seconds, allowing manual inspection for discontinuities without running the full LLM pipeline.

Summary

• Visual jumps at cut boundaries typically indicate HDR tone-mapping mismatches or incorrect overlay PTS calculations in build_final_composite().
• Audio pops usually result from overlapping fade filters in segments shorter than 60ms or insufficient fade buffers in extract_segment().
• The self-eval loop caps re-renders at three passes per SKILL.md rule 99 before surfacing failures to the user.
• Use timeline_view.py manually to isolate specific cut boundaries without LLM inference overhead.
• Always verify duration values and fade start times in helpers/render.py when debugging audio artefacts.

Frequently Asked Questions

Why does the self-eval loop fail after three attempts?

The system implements a hard cap of three re-render passes as defined in SKILL.md rule 99 to prevent infinite loops. If visual jumps or audio pops persist after three iterations, the LLM surfaces the failure to the user rather than continuing to retry, indicating a fundamental issue with the source material or EDL timing that requires manual intervention.

How do I identify which specific cut boundary is failing?

Run timeline_view.py manually with the --time flag pointing to the suspected cut boundary. For example, python helpers/timeline_view.py final.mp4 --time 12.34 generates a PNG showing the exact frame transition at 12.34 seconds. Compare this visual output against the EDL entries to identify frame discontinuities or overlay misalignments.

What causes audio pops if the 30ms fade is already applied?

Audio clicks occur when the segment duration is less than 60ms, causing the fade-out to start before the fade-in completes. Additionally, sample-rate conversion artefacts or clipped audio streams (where the fade extends beyond the actual audio data) can produce pops. Check the af string construction in extract_segment() and ensure fade_out_start never exceeds duration - fade_length.

Can I disable the self-eval loop for faster rendering?

While the source code does not expose a direct flag to disable self-eval, you can bypass the evaluation by manually running the extraction and concatenation functions (extract_segment() and concat_segments()) without invoking the timeline_view analysis. However, this risks producing undetected visual jumps or audio pops in the final output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}