How to Get Started with Video-Use Development: A Complete Setup Guide

Video-use development starts with cloning the browser-use/video-use repository, installing dependencies (ffmpeg and ElevenLabs API), and symlinking the skill into your LLM agent's directory to enable natural-language video editing through a seven-layer transcription-to-render pipeline.

Video-use is an open-source tool that enables you to edit videos through natural-language interaction with Claude Code or other LLM agents. The repository implements a lightweight pipeline that keeps the LLM’s context window small by using text transcripts and on-demand visuals rather than exhaustive frame dumps. According to the browser-use/video-use source code, the architecture follows five core principles: text-plus-on-demand visuals, audio-primary editing, explicit confirmation workflows, zero content-type assumptions, and hard production rules with artistic freedom.

Architecture Overview

The video-use development workflow is organized into seven distinct layers that transform raw footage into polished output. Each layer is implemented by specific helper modules in the repository.

Layer 1: Audio Transcription

The pipeline begins in helpers/transcribe.py, which calls ElevenLabs Scribe to generate word-level transcripts including speaker diarization and audio events. The output is stored in a compact Markdown file called takes_packed.md that serves as the primary reading surface for the LLM.

Layer 2: Visual Composite (On-Demand)

When the LLM needs visual context for ambiguous pauses or retake comparisons, helpers/timeline_view.py creates a PNG composite merging a filmstrip, waveform, and word-label overlay for the specified time range.

Layer 3: Packing

The helpers/pack_transcripts.py module collates all individual transcripts into a single, tiny (~12 KB) Markdown file (takes_packed.md) that the LLM consumes.

Layer 4: LLM Reasoning

The LLM receives the packed transcript and suggests an editing plan (cuts, color-grading, subtitles, animation overlays) based on the production rules codified in SKILL.md. This file contains 12 non-negotiable rules that guarantee clean output while leaving stylistic decisions to the user.

Layer 5: Rendering

Once approved, helpers/render.py stitches the edited segments while helpers/grade.py applies ffmpeg color-grade chains (warm, neutral, or custom). The system also handles subtitles and animation overlays via Manim, HyperFrames, Remotion, or PIL.

Layer 6: Self-Evaluation

After each cut, the pipeline runs timeline_view.py on the freshly rendered segment. The LLM checks for visual jumps, audio pops, or missing subtitles and can trigger a re-render (maximum three attempts).

Layer 7: Persistence

Session state—including transcripts, editing decisions, and the final EDL—is saved in project.md so you can resume work in later sessions.

Installation and Setup

Clone and Link the Repository

First, clone the repository and link it into your agent's skill directory:

git clone https://github.com/browser-use/video-use ~/Developer/video-use
ln -sfn ~/Developer/video-use ~/.claude/skills/video-use

# For Codex: ln -sfn ~/Developer/video-use ~/.codex/skills/video-use

Install Dependencies

Install Python dependencies and system requirements:

cd ~/Developer/video-use
uv sync              # or: pip install -e .

brew install ffmpeg  # required for video processing

brew install yt-dlp  # optional, for downloading online sources

Configure your ElevenLabs API key:

cp .env.example .env

# Edit .env to add: ELEVENLABS_API_KEY=your_key

All steps are documented in install.md.

Basic Usage Workflow

Starting a Session

Navigate to your raw footage directory and launch your LLM agent:

cd /path/to/your/raw/videos
claude  # or codex, hermes, etc.

Inside the LLM console, initiate editing with a natural language command:

> edit these into a launch video

The Edit Loop

The LLM executes a five-step workflow defined in the video-use architecture:

1. Inventory – Scans and catalogs source files.
2. Propose – Generates a cutting, color-grading, subtitle, and animation plan.
3. Confirm – Waits for your explicit approval (yes/no) before mutating video.
4. Render – Executes the plan and outputs to <videos_dir>/edit/final.mp4.
5. Evaluate – Runs self-evaluation on the rendered output.

Programmatic Python API

For custom workflows, import the helper modules directly to drive the pipeline programmatically:

from helpers.transcribe import transcribe_folder
from helpers.pack_transcripts import pack_transcripts
from helpers.render import render_edl

# 1. Transcribe everything in raw/

transcribe_folder("raw/")

# 2. Pack into a single markdown view

pack_transcripts("raw/", "takes_packed.md")

# 3. Define your edit decision list (EDL)

edl = {
    "cuts": [
        {"start": 2.5, "end": 5.3, "grade": "warm"},
        {"start": 7.1, "end": 12.0, "grade": "neutral"},
    ],
    "subtitles": True,
    "animations": ["manim"],  # optional overlay types

}

# 4. Render the final output

render_edl(edl, source_dir="raw/", out_dir="edit/")

The LLM-generated EDL follows the format described in SKILL.md, making your edits reproducible.

Summary

• Video-use development relies on a seven-layer pipeline that prioritizes audio transcripts over raw video frames to minimize LLM context usage.
• Core modules include helpers/transcribe.py for audio processing, helpers/pack_transcripts.py for context compression, and helpers/render.py for final output.
• Installation requires symlinking the repository to ~/.claude/skills/video-use, installing ffmpeg, and configuring an ElevenLabs API key.
• The workflow follows a strict ask-confirm-execute-evaluate-persist cycle codified in SKILL.md with 12 non-negotiable production rules.
• You can drive the pipeline programmatically using Python functions like transcribe_folder(), pack_transcripts(), and render_edl().

Frequently Asked Questions

What is video-use development?

Video-use development refers to building and using the video-use open-source tool that enables natural-language video editing through LLM agents. It involves setting up a seven-layer pipeline that transcribes audio, packs transcripts into lightweight Markdown, and renders final videos based on LLM-generated edit decision lists.

Do I need Claude Code to use video-use?

No. While the repository is optimized for Claude Code with a skill directory symlink at ~/.claude/skills/video-use, you can use other LLM agents like Codex or Hermes by adjusting the symlink path accordingly. The SKILL.md file provides the production rules that any compatible agent can follow.

What file formats does video-use support?

Video-use supports any format that ffmpeg can process, as the rendering layer relies on ffmpeg for video manipulation. The transcription layer uses ElevenLabs Scribe, which handles most common audio and video formats. Specific format limitations depend on your ffmpeg and ElevenLabs Scribe configurations.

How does video-use keep LLM context windows small?

Instead of dumping thousands of video frames into the context window, video-use development employs text-primary transcripts stored in takes_packed.md (~12 KB per project). Visual information is provided on-demand via helpers/timeline_view.py, which generates PNG composites only when the LLM needs to resolve ambiguities like retake comparisons or pause verification.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}