The 12 Hard Rules in video-use: 3 Rules That Cause Silent Failures

video-use enforces 12 non-negotiable hard rules for production video editing, and 3 of them—subtitle filter placement, overlay timestamp shifting, and subtitle timeline offsets—cause silent failures where the render succeeds but the visual output is wrong.

The browser-use/video-use repository relies on these 12 hard rules to prevent subtle, expensive defects in automated video pipelines. They are formally documented in SKILL.md and summarized in README.md, with core enforcement logic implemented in helpers/render.py and supporting modules.

Overview of the 12 Hard Rules in video-use

These rules govern ffmpeg filter graphs, audio boundaries, transcription strategy, and output hygiene. Here is the complete list with silent-failure risk for each:

1. Subtitles are applied LAST in the filter chain, after every overlay. Otherwise overlays hide captions. Silent failure: Yes.
2. Per-segment extract → lossless -c copy concat, not a single-pass filtergraph. Prevents double-encoding. Silent failure: No.
3. 30 ms audio fades at every segment boundary using afade=t=in:st=0:d=0.03,afade=t=out:st={dur-0.03}:d=0.03. Prevents audible pops. Silent failure: No.
4. Overlays use setpts=PTS-STARTPTS+T/TB to shift the overlay's frame 0 to its window start. Prevents mid-animation playback. Silent failure: Yes.
5. Master SRT uses output-timeline offsets: output_time = word.start - segment_start + segment_offset. Prevents caption misalignment after concat. Silent failure: Yes.
6. Never cut inside a word. Snap every cut edge to a word boundary from the Scribe transcript. Silent failure: No.
7. Pad every cut edge with a working window of 30–200 ms to absorb Scribe timestamp drift of 50–100 ms. Silent failure: No.
8. Word-level verbatim ASR only. Never SRT/phrase mode or normalized fillers. Silent failure: No.
9. Cache transcripts per source. Never re-transcribe unless the source file itself changed. Silent failure: No.
10. Parallel sub-agents for multiple animations. Spawn N at once via the Agent tool so total wall time equals the slowest task. Silent failure: No.
11. Strategy confirmation before execution. Never touch the cut until the user approves the plain-English plan. Silent failure: No.
12. All session outputs in <videos_dir>/edit/. Never write inside the video-use/ project directory. Silent failure: No.

The 3 Rules That Cause Silent Failures

Three rules can make the final video look finished while actually delivering broken visual output. As implemented in browser-use/video-use, these are the highest-risk constraints because ffmpeg exits cleanly when they are violated.

Rule 1 — Subtitles Applied Last in the Filter Chain

The ffmpeg filter graph executes sequentially. If the subtitle filter appears before overlay filters, later overlays paint over the burned-in text. Because the render completes without error, the captions are simply missing or partially hidden.

In helpers/render.py, the pipeline enforces correct ordering by appending subtitles after all overlays:

# helpers/render.py – final filter chain (simplified)

filter_chain = [
    # … per-segment extracts and overlays …

    f'overlay={overlay_path}:enable=\'between(t,{start},{end})\'',   # overlay first

    f'subtitles={subtitle_path}:force_style=...:stream_index=0'      # subtitles LAST

]
cmd = ['ffmpeg', '-i', input_path, '-filter_complex', ';'.join(filter_chain), '-c:v', 'libx264', out_path]

Placing subtitles= after every overlay= is mandatory to avoid hidden captions.

Rule 4 — Overlay Timestamps Shifted With setpts

Animated overlays must begin at their own frame 0 exactly when their display window starts. Without setpts=PTS-STARTPTS+T/TB, the overlay stream runs from its internal beginning rather than the segment start, causing the animation to start mid-sequence. The composite renders successfully, but the visual timing is wrong.

The helpers/render.py implementation shifts the overlay stream before compositing:

# helpers/render.py – overlay filter for a slot

overlay_filter = (
    f'[1:v]setpts=PTS-STARTPTS+{slot_start}/TB[ov];'   # shift start to slot_start seconds

    f'[0:v][ov]overlay=shortest=1[outv]'               # composite overlay

)

You can verify alignment before the final render by inspecting the timeline with helpers/timeline_view.py.

Rule 5 — Master SRT Rewired to Output Timeline

After lossless concatenation, subtitle timestamps from the original source no longer map to the output timeline. The pipeline must recalculate every subtitle entry with output_time = word.start - segment_start + segment_offset. If this step is skipped, subtitles remain present but appear at the wrong moment.

In helpers/render.py, the offset_subtitles function remaps each entry:

# helpers/render.py – generate master.srt with corrected offsets

def offset_subtitles(take_start, segment_offset, srt_path):
    for sub in parse_srt(srt_path):
        sub.start = sub.start - take_start + segment_offset
        sub.end   = sub.end   - take_start + segment_offset
        write_srt(sub, out_path)

Each subtitle is shifted to the concatenated timeline before burn-in so captions stay synchronized.

The Remaining 9 Hard Rules and Why They Fail Loudly

The other nine rules degrade quality in obvious ways or affect workflow rather than pixel output:

• Rule 2 — Lossless per-segment concat. helpers/render.py and helpers/grade.py extract each segment with -c copy before concatenation. A single-pass filtergraph instead would cause visible double-encoding artifacts.
• Rule 3 — 30 ms audio fades. Missing afade filters at boundaries produce audible clicks that are immediately noticeable.
• Rule 6 — Word-boundary snapping. Cuts aligned to Scribe transcript word boundaries prevent garbled speech artifacts.
• Rule 7 — Cut-edge padding. A 30–200 ms pad absorbs 50–100 ms of Scribe timestamp drift. Missing padding reduces smoothness but does not hide defects.
• Rule 8 — Word-level verbatim ASR. helpers/transcribe.py and helpers/transcribe_batch.py enforce ElevenLabs Scribe word-level mode. Phrase-level output erases sub-second gap data, creating obvious dead-air or overlap errors.
• Rule 9 — Transcript caching. The helpers cache transcripts per source hash. Re-transcribing wastes API calls but produces identical data, not silent corruption.
• Rule 10 — Parallel animation agents. Spawning agents via the Agent tool in parallel reduces wall time; sequential execution is slower but functionally equivalent.
• Rule 11 — Strategy confirmation. The UI blocks execution until the user approves the plan. Bypassing this is a workflow error caught in the interface.
• Rule 12 — Output directory isolation. All renders write to <videos_dir>/edit/ to keep the video-use/ directory untouched. Writing elsewhere is a file-system layout issue.

Summary

• video-use defines 12 hard rules in SKILL.md to ensure production-correct video output.
• Rules 1, 4, and 5 can cause silent failures: the ffmpeg process exits successfully, but subtitles are hidden behind overlays, animations start mid-sequence, or captions drift out of sync.
• These three rules are enforced in helpers/render.py through strict filter chain ordering, setpts timestamp shifting, and SRT offset recalculation.
• The remaining nine rules fail audibly, visibly, or procedurally, making them easier to detect during quality control.

Frequently Asked Questions

What is a silent failure in video-use?

A silent failure occurs when the ffmpeg rendering pipeline finishes with exit code zero and no error messages, yet the delivered video contains visual defects such as missing captions, misaligned subtitles, or incorrectly timed overlays.

Why does subtitle filter order cause a silent failure?

When the subtitles filter precedes overlay filters in the ffmpeg filter graph, later overlays paint over the burned-in text. Because every frame still renders successfully, there is no runtime error—only unreadable captions—which is why Rule 1 is critical.

What happens if overlay timestamps are not shifted with setpts?

Without setpts=PTS-STARTPTS+T/TB, an overlay stream begins at its own internal frame 0 rather than the segment start time. The render succeeds, but the animation plays from the wrong frame, producing a visual mismatch that looks like a content bug instead of a technical crash.

Where are the 12 hard rules documented?

All 12 rules are documented in the repository's SKILL.md and summarized in README.md. The core enforcement logic lives in helpers/render.py, while related constraints are implemented in helpers/grade.py, helpers/transcribe.py, helpers/transcribe_batch.py, and helpers/timeline_view.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}