How to Add Support for New Animation Engines in video-use: Beyond the Big Four

To add support for new animation engines in the browser-use/video-use repository, you implement a slot-based architecture that treats each animation as an independent overlay rendered externally, requiring updates to documentation, installation guides, and slot scaffolding while leveraging the existing ffmpeg compositing pipeline in helpers/render.py.

The browser-use/video-use repository currently orchestrates four animation engines—HyperFrames, Remotion, Manim, and PIL—through a decentralized slot system. To add support for new animation engines beyond these four, you must follow the repository's lazy-install, slot-first philosophy, which cleanly separates animation generation logic from final video compositing.

Understanding the Slot-Based Architecture

The core of video-use does not implement animation logic directly. Instead, it manages slot creation, lazy installation of dependencies, and final compositing of overlay videos. Each animation lives in edit/animations/slot_<id>/ as an independent unit that renders to render.mp4 or render.webm for alpha channels.

According to SKILL.md at line 19, the system "Generates animation overlays via HyperFrames, Remotion, Manim, or PIL". The helpers/render.py file consumes these overlays through an ffmpeg filter graph without engine-specific processing, reading paths directly from the EDL's overlays field as documented in the top comment (lines 1-8).

Step-by-Step Guide to Adding a New Engine

1. Declare the Engine in SKILL.md

Update SKILL.md at line 19 to include your new engine in the supported engines table. This allows users to select it when authoring a slot by specifying the engine name in the slot configuration.

| Core | MyEngine | Scene rendering, animation engine |

Add this entry near the existing core entries in the table.

2. Document Lazy Installation Requirements

As noted in install.md at line 160, "HyperFrames, Remotion, and Manim are optional animation engines". Add a bullet describing your new engine's runtime requirements, dependencies, and the exact installation command(s) needed for first-use setup. This maintains the repository's lazy-install model where engines are only installed when first needed.

3. Create Slot Scaffolding

When users create animation slots, the repository expects the directory edit/animations/slot_<id>/ to contain everything the engine needs, including source files, configuration, and a build script. Mimic the HyperFrames scaffolding pattern documented at SKILL.md line 210.

Your scaffolding helper should:

• Detect the selected engine from the slot's SKILL.md entry (e.g., engine: myengine)
• Run the engine's CLI to render a video into render.mp4 (or render.webm for alpha)

# In the edit/animations directory

mkdir -p slot_3 && cd slot_3

# Initialise a MyEngine project (replace with the real init command)

myengine init . --template blank

# Edit the source files, then render

myengine render . -o render.mp4

# Write the overlay entry back to the EDL (handled by the sub-agent)

4. Hook Into the Rendering Pipeline

The helpers/render.py compositing step requires no engine-specific code. It builds the final video by concatenating graded clips and overlaying animation videos in a single ffmpeg filter graph. Simply ensure your slot's output path is written to the EDL under the overlays field so render.py will pick it up automatically.

{
  "overlays": [
    {
      "file": "edit/animations/slot_3/render.mp4",
      "start_in_output": 12.3,
      "duration": 5.0
    }
  ]
}

When helpers/render.py runs, it automatically inserts this overlay into the ffmpeg filter chain.

5. Update Documentation and Validation

Update README.md at line 19 to list the new engine alongside the existing four in the feature list. Describe any special considerations such as required Node.js versions, Python dependencies, or GPU support requirements.

6. Optional: Provide a Wrapper Library

If your engine exposes a complex API (e.g., a Python library rather than a CLI), add a tiny wrapper module such as helpers/myengine.py. This wrapper converts the slot's JSON description into the engine's native format and invokes the library. The sub-agent (the LLM-driven process that runs the engine) imports this wrapper, but render.py does not need to call it directly.

Summary

• Update SKILL.md at line 19 to declare the new engine in the supported engines table.
• Document installation in install.md at line 160 following the lazy-install pattern.
• Create scaffolding that generates render.mp4 in edit/animations/slot_<id>/ following the pattern at SKILL.md line 210.
• Integrate with the pipeline by writing overlay paths to the EDL; helpers/render.py handles compositing automatically.
• Update README.md at line 19 to reflect the new capability in the feature list.
• Optional: Add a wrapper module in helpers/ for complex library-based engines.

Frequently Asked Questions

Do I need to modify helpers/render.py to add a new animation engine?

No. The helpers/render.py file uses a generic ffmpeg filter graph that consumes overlay paths from the EDL. As long as your engine outputs a standard video file (MP4 or WebM) and the path is correctly entered in the EDL's overlays field, the rendering pipeline will composite it automatically without engine-specific modifications.

What file format should the animation engine output?

The engine must output either render.mp4 for standard video or render.webm for videos requiring alpha transparency. These files must be placed in the slot directory (edit/animations/slot_<id>/). The compositing pipeline in helpers/render.py reads these files and layers them over the base video using ffmpeg.

How does video-use handle dependencies for new engines?

The repository follows a lazy installation model. According to install.md line 160, animation engines are optional and installed only when first needed. When adding support for new animation engines, you must document the installation commands in install.md so the sub-agent can execute them on first use of that engine type.

Can I use a Python library instead of a CLI tool for my animation engine?

Yes. If your engine requires a Python API rather than a command-line interface, create a wrapper module in helpers/ (e.g., helpers/myengine.py). This module translates the slot's JSON configuration into the library's required format. The sub-agent imports and calls this wrapper during the rendering phase, while helpers/render.py continues to work with the resulting video file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}