How to Add Animation Overlays in video-use Without Breaking the Render Pipeline

The correct way to add animation overlays in video-use is to generate standalone MP4 or WebM files in isolated animation slots under edit/animations/slot_<id>/, then reference them in the overlays array of your edl.json with precise start_in_output and duration values, allowing the build_final_composite function in helpers/render.py to handle PTS synchronization automatically.

The video-use repository (browser-use/video-use) processes video through a strict three-stage pipeline that concatenates base footage before compositing any overlays. Because the final compositing step relies on a single ffmpeg filter graph, animation overlays must conform to specific format and directory conventions to avoid misalignment or render failures.

Understanding the video-use Render Pipeline

The render pipeline defined in helpers/render.py executes in three distinct phases:

1. Per-segment extraction & grading – Each cut is extracted into individual MP4 segments via extract_segment.
2. Loss-less concatenation – Graded clips are concat-demuxed into a base video file.
3. Final compositing – The base video is overlaid with animation clips and finally subtitles via build_final_composite (lines 96-108).

According to the video-use source code, the build_final_composite function constructs an ffmpeg filter graph where each overlay video is PTS-shifted using setpts=PTS-STARTPTS+<t>/TB to align its first frame with the start_in_output timestamp defined in your EDL. Subtitles are always appended last (lines 38-44), ensuring they appear above all animations.

Animation Slot Isolation and Directory Structure

The pipeline requires strict slot isolation for external animation engines. As documented in SKILL.md and install.md, you must create an animation slot under edit/animations/slot_<id>/ rather than placing files at the repository root.

• HyperFrames, Remotion, and Manim are treated as optional engines installed lazily on first use.
• Each slot operates as an isolated workspace where the engine outputs a render.mp4 or render.webm file.
• The edl.json references these outputs using relative paths from the EDL location.

Supported Animation Engines

HyperFrames Overlays

HyperFrames generates transparent WebM files suitable for overlay graphics. The engine runs via npx without global installation:

mkdir -p edit/animations/slot_1
cd edit/animations/slot_1

# Scaffold and render (lazy install)

npx --yes hyperframes init . --example blank --non-interactive --skip-skills
npx --yes hyperframes render . -o render.webm --format webm

Remotion Overlays

Remotion uses React components to generate video. Scaffold a local project within a slot:

mkdir -p edit/animations/slot_2 && cd edit/animations/slot_2

# Scaffold locally

npx create-video@latest .

# Render exact duration (e.g., 3 seconds, no audio)

npx remotion render src/Video.jsx 0 3 --output render.mp4

Ensure your component renders for exactly the duration specified in your EDL entry.

Manim Overlays

For mathematical animations, Manim installs via pip on first use:

mkdir -p edit/animations/slot_3 && cd edit/animations/slot_3
pip install manim

# Create scene file then render

manim example.scene.py Intro -qm -o render.webm --format webm

Manim outputs should use WebM format with alpha channels for transparency.

PIL (Pillow) Static Overlays

For programmatically generated graphics or simple fades, use Pillow to generate frame sequences and encode with ffmpeg:

from PIL import Image, ImageDraw, ImageFont
import subprocess
from pathlib import Path

def make_overlay(out_path: Path, duration: float, fps: int = 24):
    frames = int(duration * fps)
    img = Image.new("RGBA", (1920, 1080), (0, 0, 0, 0))
    draw = ImageDraw.Draw(img)
    font = ImageFont.truetype("DejaVuSans-Bold.ttf", 80)
    draw.text((960, 540), "PIL overlay", font=font, fill=(255, 255, 255, 255), anchor="mm")
    
    tmp_dir = out_path.parent / "tmp_frames"
    tmp_dir.mkdir(parents=True, exist_ok=True)
    
    for i in range(frames):
        # Fade-in calculation for first 0.5s

        alpha = int(255 * min(1, (i / (fps * 0.5)))) if i < fps * 0.5 else 255
        frame = img.copy()
        frame.putalpha(alpha)
        frame.save(tmp_dir / f"frame_{i:04d}.png")
    
    subprocess.run([
        "ffmpeg", "-y", "-framerate", str(fps), "-i",
        str(tmp_dir / "frame_%04d.png"),
        "-c:v", "libvpx-vp9", "-pix_fmt", "yuva420p", "-lossless", "1",
        str(out_path)
    ], check=True)
    
    # Cleanup

    for p in tmp_dir.iterdir(): p.unlink()
    tmp_dir.rmdir()

if __name__ == "__main__":
    make_overlay(Path("render.webm"), duration=2.5)

Run this script inside an animation slot to produce a transparent WebM ready for compositing.

EDL Configuration for Overlays

Each overlay requires a JSON entry in the overlays list within edl.json:

{
  "sources": { "take1": "raw/take1.mp4" },
  "ranges": [
    { "source": "take1", "start": 0.0, "end": 6.0 }
  ],
  "overlays": [
    {
      "file": "animations/slot_1/render.webm",
      "start_in_output": 1.5,
      "duration": 4.0
    }
  ]
}

Critical parameters:

• file: Relative path to the animation slot output
• start_in_output: Exact timestamp in seconds where frame 0 of the overlay appears
• duration: Length of the overlay in seconds (must match the generated video duration)

As implemented in browser-use/video-use, the build_final_composite function automatically applies the setpts filter to synchronize timing. Do not pre-offset the video yourself.

Technical Requirements for Overlay Videos

To prevent pipeline breakage, adhere to these constraints:

• Resolution: Match the base video resolution (default 1080p). Mismatched sizes trigger automatic scaling but may degrade quality.
• Frame rate: Use 24 fps to match the default base video frame rate.
• Format: Use WebM with alpha channel (yuva420p) for transparent overlays; MP4 for opaque content.
• Audio: Overlays must be silent. The filter graph ignores audio tracks in overlay files.
• PTS handling: Let the pipeline manage timestamp shifting via the loop at line 21 of helpers/render.py.

Summary

• Slot isolation is mandatory: Create animation outputs under edit/animations/slot_<id>/ only.
• Lazy installation of HyperFrames, Remotion, and Manim occurs on first use; no global installs required.
• Standalone video files (MP4/WebM) must match base video resolution and frame rate.
• EDL entries require file, start_in_output, and duration parameters; the pipeline handles PTS synchronization automatically.
• Subtitles render last, ensuring they appear above all animation overlays.

Frequently Asked Questions

Why does my overlay appear at the wrong timestamp?

The video-use pipeline uses setpts=PTS-STARTPTS+<t>/TB to shift overlay timestamps automatically. If your overlay appears misaligned, verify that the start_in_output value in edl.json matches the intended output time, and ensure you have not pre-offset the video during generation. The build_final_composite function handles synchronization as long as the overlay video starts at 0 seconds relative to its own timeline.

Can I use transparent backgrounds with animation overlays?

Yes. Use WebM format with alpha channel support (yuva420p pixel format) for transparent overlays. HyperFrames generates these by default when using --format webm, and the PIL example above uses libvpx-vp9 with yuva420p to preserve transparency. Transparent overlays are composited correctly before the final subtitle layer is applied.

What happens if my overlay resolution doesn't match the base video?

The ffmpeg filter graph in helpers/render.py will automatically scale mismatched resolutions, but this may introduce quality degradation or letterboxing. For best results, render overlays at the exact output resolution (default 1920x1080) and 24 fps to avoid any scaling artifacts during the final compositing stage.

Do I need to install HyperFrames, Remotion, or Manim globally?

No. According to install.md, these engines are optional and installed lazily per slot. HyperFrames runs via npx --yes hyperframes, Remotion via npx create-video@latest, and Manim via pip install manim executed within the specific animation slot directory. This isolation prevents dependency conflicts with the core video-use pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}