How video-use Manages Multiple Animation Slots with Parallel Sub-Agents

video-use orchestrates multiple animation slots by spawning independent sub-agents for each overlay, then composites them via FFmpeg in a single pass after all parallel renders complete.

The browser-use/video-use repository enables LLM-driven video editing through a slot-based architecture where each animation overlay runs concurrently. When the system generates a video, it treats every requested animation—whether HyperFrames, Remotion, Manim, or PIL—as an independent slot in the Edit Decision List (EDL). This design isolates CPU-intensive rendering tasks from the final compositing step, allowing parallel execution without blocking the main pipeline.

The EDL Overlays Array Structure

Each animation slot is declared as a dictionary within the EDL's overlays array. According to the source code, these entries specify the file path, temporal placement, and duration for every overlay.

The schema requires three critical fields per slot:

• file – The path to the rendered asset (MP4 or PNG sequence) produced by the sub-agent
• start_in_output – The exact timestamp in seconds where the overlay appears in the final video
• duration – How long the overlay remains visible

# Example EDL snippet generated by the LLM planner

{
  "overlays": [
    {
      "file": "anim_01.mp4",
      "start_in_output": "12.34",
      "duration": "5.0"
    },
    {
      "file": "anim_02.mp4",
      "start_in_output": "30.00",
      "duration": "3.2"
    }
  ]
}

Parallel Sub-Agent Execution Model

When the skill executes, video-use launches one sub-agent per overlay slot. These sub-agents operate in separate processes, rendering animations independently while the coordinating process tracks completion status.

The implementation follows a fork-join pattern:

1. Dispatch – Iterate through edl["overlays"] and spawn a sub-agent for each slot's animation_spec
2. Render – Each sub-agent writes its output to the specified file path without blocking siblings
3. Synchronization – The main thread waits until all overlay files exist before invoking the compositing step

This parallelism ensures that total render time equals the duration of the slowest animation, not the sum of all animations.

The Compositing Pipeline in render.py

The build_final_composite function in helpers/render.py transforms the EDL overlay array into an FFmpeg filter graph. This single-pass approach eliminates intermediate files and maintains frame-accurate synchronization.

Input Handling and FFmpeg Setup

The function first constructs the input arguments by appending each overlay file as an additional -i flag following the base video:

# Excerpt from helpers/render.py (lines 15-19)

inputs = ["-i", str(base_path)]
for ov in overlays:
    ov_path = resolve_path(ov["file"], edit_dir)
    inputs += ["-i", str(ov_path)]

PTS Shifting for Timing Synchronization

To align each overlay with its designated start_in_output timestamp, the pipeline applies a setpts filter. This shifts the presentation timestamp of each overlay stream so that frame 0 corresponds to the exact moment it should appear in the composite:

# From helpers/render.py (lines 21-25)

filter_parts = []
for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

The PTS-STARTPTS+{t}/TB expression calculates the timebase-adjusted offset required to synchronize the overlay with the base timeline.

Overlay Chaining and Final Composition

The filter graph then chains these shifted streams using the overlay filter with temporal enablement. Each overlay is applied only during its specified duration window:

# From helpers/render.py (lines 27-35)

# Overlay chain logic applying between(t,start,end) constraints

overlay_filter = f"overlay=enable='between(t,{t:.3f},{end:.3f})'"

Subtitles are appended last in the pipeline, ensuring text layers remain visible above all animated content.

Performance Benefits of Parallel Rendering

The architecture separates animation generation from video stitching to maximize throughput. Because sub-agents run concurrently using separate processes, the system scales horizontally with available CPU cores. The heavy-weight operations—such as HyperFrames API calls or Manim scene rendering—execute in isolation, preventing memory pressure or I/O blocking in the main thread.

The helpers/pack_transcripts.py module supports this workflow by enabling the editor sub-agent to read packed transcripts and select cuts that later drive overlay timing decisions. This ensures that animation slots are planned before rendering begins, eliminating wasted compute on unused assets.

Summary

• EDL overlays array declares animation slots with file paths, timestamps, and durations
• Parallel sub-agents render each slot independently in separate processes before compositing
• build_final_composite in helpers/render.py constructs FFmpeg commands that PTS-shift and overlay all slots in a single pass
• FFmpeg filter-complex handles the final composition using setpts for timing and overlay for visibility windows
• Concurrent execution ensures total render time equals the longest animation duration, not the sum of all slots

Frequently Asked Questions

How does video-use ensure animations appear at the correct timestamp?

The system uses FFmpeg's setpts filter to shift each overlay's presentation timestamps. In helpers/render.py, the filter expression PTS-STARTPTS+{t}/TB adjusts the overlay stream so that frame 0 aligns with the start_in_output value specified in the EDL slot.

What happens if one sub-agent fails while others are rendering?

The analysis indicates that the coordinating code waits for all output files to appear before invoking FFmpeg. While the specific error handling isn't detailed, the architecture implies that missing files would cause the compositing step to fail, as the FFmpeg input list expects every declared overlay file to exist.

Can different animation types (Manim, Remotion, PIL) run in the same video?

Yes. The slot-based architecture is agnostic to animation technology. Each sub-agent handles its specific rendering requirements—whether calling external APIs like HyperFrames or executing Python scripts—while the EDL simply tracks the resulting file paths for final compositing.

Why does the compositing step wait for all sub-agents to finish?

The FFmpeg filter graph requires all inputs to be present when the command executes. Since the filter-complex must reference every overlay stream via indexed inputs ([1:v], [2:v], etc.), the pipeline cannot begin final composition until all parallel renders complete and write their output files to disk.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}