How to Extend video-use with Additional Animation Rendering Engines

You can extend video-use by creating a skill directory with a SKILL.md descriptor, a slot-scaffolding script, and registering the engine in the top-level SKILL.md, while the existing helpers/render.py pipeline handles the final overlay compositing automatically.

The video-use repository treats every animation as a slot that independent rendering engines populate with video files. Because the core pipeline is engine-agnostic, you can integrate any animation tool—from Node.js CLIs to Python packages—without modifying the central rendering logic. This guide walks through the exact file structure, conventions, and code required to extend video-use with new animation rendering engines.

Understanding the video-use Animation Architecture

The Slot-Based Workflow

video-use manages animations through a strict directory convention. Each animation occupies a unique slot inside <videos_dir>/edit/animations/slot_<id>/. According to the source code in SKILL.md (lines 205-212), the agent scaffolds these slots at runtime by executing engine-specific initialization commands.

The workflow follows three distinct phases:

1. Slot creation – The agent creates a sub-directory and populates it with starter files specific to the chosen engine (Node.js or Python-based).
2. Engine invocation – The slot's build script runs the rendering engine to produce a single video file, typically named render.mp4 or render.webm.
3. Overlay stitching – The helpers/render.py script receives the rendered file path via the --overlay flag, applies a setpts timestamp shift, and composites the animation into the final edit.

The Rendering Pipeline

The central render helper located at helpers/render.py implements the build_final_composite function (line 575) to handle all overlay processing. This implementation enforces two hard rules automatically:

• PTS shifting – Applies setpts=PTS-STARTPTS+T/TB to synchronize overlay timing (Rule 4, line 25).
• Subtitle layering – Adds subtitles last to ensure they remain visible over animations (Rule 1, line 22).

Because this pipeline only requires a valid video file path, it remains completely decoupled from the specific animation engine used to generate the content.

Step-by-Step Guide to Adding a New Animation Engine

Step 1: Create a Skill Sub-Directory

Create a new directory under skills/<your-engine>/ to house your engine's configuration. This mirrors the existing patterns found in skills/manim-video/ and skills/hyperframes/.

Your skill directory must contain a SKILL.md file describing:

• The engine's purpose and output format
• Required dependencies and installation commands
• Exact CLI commands for slot initialization and rendering

Step 2: Scaffold the Slot with an Initialization Script

Provide a Bash, Node, or Python script (conventionally named init.sh) that creates the slot folder and writes starter project files. The script receives the target slot directory as its first argument ($1).

For example, the HyperFrames integration uses npx --yes hyperframes init to scaffold browser-based projects. Your script should follow this pattern, preparing all files necessary for the user to edit and render.

Step 3: Implement the Render Command

Your engine must output exactly one video file (e.g., render.mp4 or render.webm) that the main render helper can consume. The file path is passed to helpers/render.py via the --overlay parameter during the stitching phase.

Ensure your render command respects the slot directory structure and deposits the final video at the path expected by the agent.

Step 4: Register the Engine in SKILL.md

Add a reference to your engine in the top-level SKILL.md under the Animations section. This registration allows the LLM agent to select your engine during conversation planning, alongside existing options like HyperFrames, Remotion, and Manim.

Step 5: Document Dependencies in install.md

Update install.md to mention any additional system dependencies (such as new CLI tools or binary requirements). Following the existing pattern for HyperFrames and Remotion, note that these dependencies will be installed lazily on first use to keep the base installation lightweight.

Complete Example: Adding an SVG-Animator Engine

Below is a complete implementation adding a fictional svg-animator CLI to the video-use pipeline.

First, create the skill directory and descriptor:

mkdir -p skills/svg-animator/scripts
cat > skills/svg-animator/SKILL.md <<'EOF'
---
name: svg-animator
description: Render SVG-based animations (e.g., animated infographics) into video overlays.
---

# SVG-Animator

* **Dependencies** – `npm i -g svg-animator-cli`
* **Slot scaffold** – `svg-animator init .` creates `src/` with `animation.svg`
* **Render command** – `svg-animator render src/animation.svg -o render.mp4`
EOF

Next, create the slot initialization script:

cat > skills/svg-animator/scripts/init.sh <<'EOF'
#!/usr/bin/env bash
set -e
slot_dir=$1
mkdir -p "$slot_dir/src"
cd "$slot_dir"
svg-animator init .
echo "Slot ready. Edit $slot_dir/src/animation.svg then run:"
echo "  svg-animator render src/animation.svg -o render.mp4"
EOF
chmod +x skills/svg-animator/scripts/init.sh

At runtime, the agent executes the workflow as follows:

# Create and scaffold the slot

slot_dir=$(mktemp -d /tmp/slot_svg_XXXX)
skills/svg-animator/scripts/init.sh "$slot_dir"

# User edits $slot_dir/src/animation.svg, then renders:

svg-animator render "$slot_dir/src/animation.svg" -o "$slot_dir/render.mp4"

# Hand the produced video to the main renderer

python helpers/render.py edl.json -o final.mp4 --overlay "$slot_dir/render.mp4"

The helpers/render.py script automatically applies the required setpts shift and subtitle layering, producing the final composite without engine-specific modifications.

Key Files and Extension Points

• SKILL.md (top-level) – Defines core animation skills and lists supported engines. Located at the repository root.
• install.md – Documents lazy installation patterns for engine dependencies.
• helpers/render.py – Central render helper that handles overlay stitching and enforces rendering rules (lines 22, 25, and 575).
• skills/manim-video/ – Reference implementation showing a Python-based math animation engine.
• skills/hyperframes/ – Reference for browser-based engines, demonstrating slot-scaffold commands.

Summary

• video-use uses a slot-based architecture where each animation engine populates a directory with a renderable video file.
• To add a new engine, create a skills/<engine>/ directory containing a SKILL.md descriptor and an initialization script.
• The engine must output a single video file (e.g., render.mp4) that helpers/render.py composites via the --overlay flag.
• Registration requires updating the top-level SKILL.md (for agent selection) and install.md (for dependency documentation).
• The rendering pipeline in helpers/render.py is engine-agnostic, automatically handling timestamp synchronization (setpts) and subtitle layering.

Frequently Asked Questions

Do I need to modify helpers/render.py to add a new engine?

No. The helpers/render.py pipeline is engine-agnostic and consumes any valid video file passed via the --overlay flag. You only need to modify this file if you require engine-specific optimizations to the compositing logic, such as custom FFmpeg filters beyond the standard setpts shift.

What video formats are supported for the rendered output?

The overlay system accepts standard video formats including render.mp4 and render.webm. As long as FFmpeg can decode the file, helpers/render.py can process it. The engine should deposit the final rendered file in the slot directory with a predictable filename so the agent can reference it correctly.

How does video-use handle timing synchronization for custom animations?

The helpers/render.py script automatically applies a PTS shift using setpts=PTS-STARTPTS+T/TB (as implemented at line 25) to align the overlay's timeline with the main video sequence. This ensures that animations render at their intended temporal positions regardless of when the engine generated the file.

Can I use a Python-based animation library instead of a Node.js CLI?

Yes. The extension architecture supports any rendering engine that produces a video file. Reference the skills/manim-video/ directory for an example of a Python-based implementation. You would create a Python script instead of a Bash script for slot initialization, but the registration process in SKILL.md and install.md remains identical.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}