How to Integrate External Animations (HyperFrames, Remotion, Manim) with video-use Edits

video-use treats animation overlays as independent sub-agents that render into isolated slots, then composites them onto the final timeline using PTS-shifting to ensure frame-perfect synchronization.

Integrating external animation engines into a video-use editing workflow allows you to add complex motion graphics, code-driven animations, or mathematical visualizations to your video projects. The browser-use/video-use repository implements a slot-based architecture where each animation engine runs as a parallel sub-agent, generating videos that are later stitched into the final output without re-encoding the base footage.

The Three-Stage Integration Pipeline

The integration follows a strict pipeline defined in helpers/render.py and SKILL.md. Each stage maintains separation between the animation generation and the final compositing, ensuring zero re-encoding of source material until the final filtergraph.

Stage 1: Creating an Animation Slot

Each animation lives in a dedicated directory under edit/animations/slot_<id>/. A sub-agent spawns the appropriate engine via the Agent tool, scaffolding the project files and rendering an output file (typically render.mp4 or render.webm) 【4†L99-L106】.

The slot acts as a sandboxed workspace. For example, when using HyperFrames, the directory contains an HTML/CSS/GSAP project, while Remotion slots hold React components, and Manim slots contain Python scene scripts.

Stage 2: Referencing Overlays in the EDL

Once rendered, the animation must be registered in edit/edl.json under the overlays array. Each entry requires three fields:

• file: Relative path to the rendered video (e.g., edit/animations/slot_1/render.mp4)
• start_in_output: Timestamp in seconds where the overlay begins
• duration: Length of the overlay in seconds

The render.py script consumes this array at line 【5†L421-L429】 to build the compositing graph.

Stage 3: Final Compositing with PTS Shifting

The build_final_composite() function in helpers/render.py handles the stitching 【5†L96-L106】. It first extracts and concatenates all cut segments losslessly (rule 2) 【5†L49-L53】, then processes each overlay:

1. Loads the overlay video
2. Applies setpts=PTS-STARTPTS+<t>/TB to shift the timeline so frame 0 aligns with start_in_output 【5†L21-L26】
3. Overlays the animation onto the base video
4. Adds subtitles last (rule 1) 【5†L38-L44】 to ensure text remains visible above animations

Engine-Specific Implementation Details

Each animation engine follows the same slot pattern but uses different tooling to generate the render.mp4 file.

HyperFrames (HTML/CSS/GSAP)

HyperFrames specializes in web-based animations using standard HTML, CSS, and GSAP timelines.

Setup command:

npx --yes hyperframes init . --example blank --non-interactive --skip-skills

Rendering:

npx --yes hyperframes render . -o render.mp4

# For alpha transparency, use:

npx --yes hyperframes render . -o render.webm --format webm

The output render.mp4 (or render.webm) is placed directly in the slot directory 【5†L99-L106】.

Remotion (React-Based Video)

Remotion allows you to write React components that render as video frames, ideal for data visualizations and programmatic motion graphics.

Setup: Initialize within the slot using npx create-video@latest or reference a pre-installed Remotion instance.

Rendering: Execute from within the slot directory to keep dependencies isolated:

remotion render <entry-file> out.mp4

Rename the output to render.mp4 or reference out.mp4 directly in the EDL 【5†L100-L107】.

Manim (Python Mathematical Animation)

Manim generates educational mathematical animations through Python scene scripts.

Workflow:

1. Create scene.py in the slot directory
2. Execute Manim to generate frames:

manim scene.py MyScene -ql

3. Convert the resulting PNG sequence to video:

ffmpeg -y -framerate 30 -i media/images/myscene/*.png -c:v libx264 -pix_fmt yuv420p render.mp4

The skills/manim-video/SKILL.md file provides the complete pipeline specification 【4†L3-L5】.

Practical Code Examples

Example 1: Scaffolding a HyperFrames Slot

Run these commands from your edit/animations directory:

# Create slot directory

mkdir -p slot_3 && cd slot_3

# Scaffold blank project

npx --yes hyperframes init . --example blank --non-interactive --skip-skills

# Edit index.html, style.css, and script.js to build animation

# Then render

npx --yes hyperframes render . -o render.mp4

Result: edit/animations/slot_3/render.mp4 ready for EDL reference.

Example 2: Registering an Overlay in edl.json

import json
from pathlib import Path

edl_path = Path("edit/edl.json")
edl = json.loads(edl_path.read_text())

# Append overlay entry

edl.setdefault("overlays", []).append({
    "file": "edit/animations/slot_3/render.mp4",
    "start_in_output": 12.3,  # Start at 12.3 seconds

    "duration": 5.0           # Display for 5 seconds

})

edl_path.write_text(json.dumps(edl, indent=2))

Example 3: Rendering the Final Composite

python helpers/render.py edit/edl.json -o final.mp4 --build-subtitles

This invokes the full pipeline:

• extract_all_segments() pulls cut segments 【5†L14-L22】
• concat_segments() joins them losslessly 【5†L66-L73】
• build_final_composite() applies overlays with PTS shifting 【5†L96-L106】
• build_master_srt() adds subtitles last 【5†L86-L94】

Example 4: Manim Slot Creation

mkdir -p edit/animations/slot_4 && cd edit/animations/slot_4

# Create scene.py with your Manim scene class

# Then render

manim scene.py MyScene -ql

# Convert PNG sequence to video if needed

ffmpeg -y -framerate 30 -i media/images/MyScene/*.png -c:v libx264 -pix_fmt yuv420p render.mp4

Why This Architecture Works

The video-use overlay system optimizes for production quality and build performance through four key design decisions:

• Parallelism – Each animation slot runs as an independent sub-agent (rule 10) 【4†L31-L33】, allowing wall-time to scale with the slowest engine rather than the sum of all engines.
• Zero Re-encoding – Per-segment extracts concatenate losslessly (rule 2) 【5†L49-L53】, and overlays composite only once in the final filtergraph, preventing generational quality loss.
• Precise Timing – The setpts filter shifts frame 0 of the overlay to the exact EDL timestamp (rule 4) 【5†L21-L26】, ensuring visual cues align perfectly with narration.
• Subtitle Safety – Subtitles burn last (rule 1) 【5†L38-L44】, guaranteeing they remain visible above any animation overlays.

Summary

• Slot-based workflow: Each animation engine occupies edit/animations/slot_<id>/ and produces a render.mp4 file.
• EDL registration: Add overlay entries to edl.json with file, start_in_output, and duration fields.
• PTS shifting: helpers/render.py uses setpts=PTS-STARTPTS+<t>/TB to synchronize overlay timelines without re-encoding base footage.
• Engine flexibility: HyperFrames (web), Remotion (React), and Manim (Python) all follow the same slot convention.
• Render order: Base video → Overlays → Subtitles, ensuring text remains legible.

Frequently Asked Questions

How do I ensure my animation aligns perfectly with specific audio cues in the video?

Use the start_in_output field in your edl.json overlay entry to specify the exact timestamp in seconds where frame 0 of your animation should appear. The build_final_composite() function in helpers/render.py applies a setpts filter to shift the overlay's presentation timestamp so its first frame aligns precisely with this value, independent of when the source video segments start 【5†L21-L26】.

Can I use video formats other than MP4 for overlays?

Yes. While render.mp4 is the default, render.webm is supported—particularly useful for HyperFrames when you need alpha transparency. The helpers/render.py pipeline detects the file extension specified in the EDL and processes it accordingly through the FFmpeg filtergraph. Ensure your target format supports the color space required for compositing.

Why are subtitles added after animations rather than before?

According to rule 1 in SKILL.md, subtitles must be burnt last to prevent them from being obscured by overlay content 【5†L38-L44】. The build_final_composite() function explicitly layers subtitles on top of the fully composited base video and animation stack, ensuring maximum readability.

Do I need to reinstall Node or Python dependencies for every animation slot?

No. The sub-agent architecture isolates dependencies within each slot. For Remotion and HyperFrames, the npx command handles temporary installations without polluting the global namespace. For Manim, you install the engine once per environment, but each slot contains only the scene script and output files. This keeps the edit repository lightweight while allowing parallel rendering of multiple animations 【4†L31-L33】.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}