How to Integrate video-use into a Web Application: Complete Implementation Guide

You can integrate video-use into any web application by invoking its five-stage Python pipeline as subprocesses from your backend, monitoring the edit/ directory for the generated final.mp4, and streaming the result to clients.

video-use is an open-source Python framework—developed by browser-use—that enables language models to edit video through text-based transcript manipulation. To integrate video-use into a web application, you orchestrate its file-based pipeline stages from your backend server, allowing the LLM to propose edits via markdown while FFmpeg handles the heavy lifting of video rendering.

Understanding the video-use Pipeline Architecture

The video-use architecture consists of five discrete stages that communicate through the filesystem rather than APIs. According to the browser-use/video-use source code, each stage is implemented as a pure function that reads from and writes to an edit/ folder adjacent to your raw footage.

Stage	Function	Source File
Transcribe	Extracts 16 kHz mono audio and sends it to ElevenLabs Scribe, producing JSON transcripts with word-level timestamps	helpers/transcribe.py
Pack	Collates all per-take transcripts into a single takes_packed.md file that serves as the LLM's primary editing view	helpers/pack_transcripts.py
LLM Reasoning	The language model (Claude Code, Codex, or Hermes) reads the packed transcript and proposes an edit plan (EDL) stored in project.md, following rules defined in SKILL.md	SKILL.md
Render	Executes FFmpeg commands to cut, color-grade, add fades, burn subtitles, and stitch animation overlays (HyperFrames, Remotion, Manim, or Pillow), outputting edit/final.mp4	helpers/render.py
Self-Eval	Generates a PNG timeline composite via timeline_view and feeds it back to the LLM for quality verification, with automatic re-rendering (max 3 passes) if issues are detected	helpers/timeline_view.py

Because the interface is file-based, any backend language can trigger these scripts, poll for output files, and stream results to browsers.

Prerequisites and Environment Setup

Before integrating video-use into your web application, prepare the host environment with the following dependencies:

1. Install the video-use package using uv sync or pip install -e . from the repository root
2. Verify FFmpeg and yt-dlp are available on the system PATH
3. Configure the ElevenLabs API key in a .env file or as an environment variable (ELEVENLABS_API_KEY)
4. Ensure sufficient disk space for the edit/ folder, which stores intermediate audio files, JSON transcripts, and the final rendered video

The pipeline is stateless; the only persistent state lives in project.md and takes_packed.md, making it safe to run multiple concurrent sessions isolated in separate directories.

Step-by-Step Integration Process

Uploading Raw Footage

Your frontend should upload video files to a session-specific directory on the server. Each session requires a unique workspace folder containing a raw/ subdirectory for input files and an edit/ directory for pipeline outputs.

Running the Transcription Stage

Invoke helpers/transcribe.py for each uploaded video. This script extracts the audio track and caches the JSON transcript:

subprocess.run(
    ["python", "helpers/transcribe.py", str(video_path)],
    cwd=work_dir,
    check=True
)

Packing Transcripts for LLM Processing

After transcribing all takes, run helpers/pack_transcripts.py to generate takes_packed.md. This markdown file aggregates all word-level timestamps and speaker diarization data into a format the LLM can manipulate.

Executing the Render Pipeline

Once your application has confirmed the edit plan (stored in project.md), trigger helpers/render.py to generate the final video:

subprocess.run(
    ["python", "helpers/render.py"],
    cwd=work_dir,
    check=True
)

The script outputs edit/final.mp4, which your server can then stream to the client.

Production Implementation Examples

Flask (Python) Backend

The following Flask implementation demonstrates how to handle uploads, execute the pipeline, and serve the resulting video:

import os
import subprocess
import pathlib
import uuid
from flask import Flask, request, send_from_directory, jsonify

app = Flask(__name__)
BASE = pathlib.Path("/var/www/video-use")

def run_script(args, cwd):
    """Execute a helper script and stream stdout for logging."""
    proc = subprocess.Popen(
        ["python", *args],
        cwd=cwd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
    )
    for line in proc.stdout:
        print(line, end="")
    proc.wait()
    if proc.returncode != 0:
        raise RuntimeError(f"{args[0]} failed")

@app.route("/upload", methods=["POST"])
def upload():
    # Create isolated session directory

    session_id = uuid.uuid4().hex
    work_dir = BASE / session_id
    raw_dir = work_dir / "raw"
    raw_dir.mkdir(parents=True)
    
    # Save uploaded files

    for f in request.files.getlist("videos"):
        f.save(raw_dir / f.filename)
    
    # Stage 1: Transcribe

    for video_path in raw_dir.iterdir():
        run_script(["helpers/transcribe.py", str(video_path)], cwd=work_dir)
    
    # Stage 2: Pack transcripts

    run_script(["helpers/pack_transcripts.py"], cwd=work_dir)
    
    # Stage 3: Render (assuming LLM has generated project.md)

    run_script(["helpers/render.py"], cwd=work_dir)
    
    return jsonify(video_url=f"/download/{session_id}")

@app.route("/download/<session_id>")
def download(session_id):
    edit_dir = BASE / session_id / "edit"
    return send_from_directory(edit_dir, "final.mp4")

Node.js/Express Backend

For JavaScript backends, use child_process.spawn to invoke the same Python scripts:

const express = require('express');
const { spawn } = require('child_process');
const path = require('path');
const fs = require('fs');
const multer = require('multer');
const { v4: uuidv4 } = require('uuid');

const app = express();
const upload = multer({ dest: 'uploads/' });
const BASE = '/var/www/video-use';

function runScript(args, cwd) {
  return new Promise((resolve, reject) => {
    const proc = spawn('python', args, { cwd });
    proc.stdout.on('data', d => console.log(d.toString()));
    proc.stderr.on('data', d => console.error(d.toString()));
    proc.on('close', code => {
      if (code === 0) resolve();
      else reject(new Error(`${args[0]} failed`));
    });
  });
}

app.post('/upload', upload.array('videos'), async (req, res) => {
  const sessionId = uuidv4();
  const workDir = path.join(BASE, sessionId);
  const rawDir = path.join(workDir, 'raw');
  fs.mkdirSync(rawDir, { recursive: true });
  
  // Move uploads to session folder
  for (const file of req.files) {
    const dest = path.join(rawDir, file.originalname);
    fs.renameSync(file.path, dest);
  }
  
  // Execute pipeline
  for (const video of fs.readdirSync(rawDir)) {
    await runScript(['helpers/transcribe.py', path.join(rawDir, video)], workDir);
  }
  
  await runScript(['helpers/pack_transcripts.py'], workDir);
  await runScript(['helpers/render.py'], workDir);
  
  res.json({ videoUrl: `/download/${sessionId}` });
});

app.get('/download/:sessionId', (req, res) => {
  const editDir = path.join(BASE, req.params.sessionId, 'edit');
  res.sendFile('final.mp4', { root: editDir });
});

app.listen(3000);

Handling Concurrent Sessions and State Management

Because video-use stores state in files (project.md, takes_packed.md) rather than memory, you can safely run multiple editing sessions concurrently. Isolate each session in its own UUID-named directory under your base path. Your backend should clean up temporary raw/ files after processing to conserve disk space, though you may want to retain the edit/ directory for a short period to allow re-downloads.

Summary

• video-use exposes a file-based interface through five Python scripts in helpers/ that any backend can invoke via subprocess calls.
• The pipeline requires ElevenLabs API keys, FFmpeg, and Python dependencies installed on the host server.
• Your backend uploads files to a session-specific raw/ directory, executes transcribe.py, pack_transcripts.py, and render.py in sequence, then serves edit/final.mp4.
• Both Flask and Express implementations follow the same pattern: create workspace, run scripts, poll for output, stream result.
• The stateless architecture supports horizontal scaling and concurrent sessions without memory conflicts.

Frequently Asked Questions

Can I integrate video-use with a non-Python backend?

Yes. Because video-use exposes a command-line interface through its helper scripts, any backend language capable of spawning subprocesses—such as Node.js, Ruby, Go, or PHP—can invoke helpers/transcribe.py, helpers/pack_transcripts.py, and helpers/render.py. The integration requires only that Python and dependencies are installed on the server, not that your application itself uses Python.

How does video-use handle concurrent video editing sessions?

The pipeline is stateless and filesystem-based. Each session should use a unique working directory (typically a UUID-named folder) containing its own raw/ and edit/ subdirectories. Since project.md and takes_packed.md live within these isolated folders, multiple sessions can run simultaneously without interfering with each other, provided the server has sufficient CPU and disk I/O for FFmpeg rendering.

What video file formats does video-use support?

video-use relies on FFmpeg for all video processing, so it inherits FFmpeg's broad format support including MP4, MOV, AVI, and MKV containers. The transcription stage extracts mono 16 kHz audio, which FFmpeg can generate from any source format it supports. For best results, use H.264 or H.265 encoded MP4 files as input, as these minimize transcoding time during the render stage.

How do I customize the LLM editing behavior?

The editing rules and constraints are defined in SKILL.md at the repository root. This file contains the system prompts and production rules that guide the LLM when generating edit decisions (EDL) in project.md. To customize behavior—such as enforcing specific transition styles, color-grading preferences, or subtitle formatting—modify SKILL.md before invoking the LLM reasoning stage, or pass a custom path to your modified skill file via environment variables if supported by your agent configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}