Roadmap for Future Development of video-use: 8 Milestones for the Transcript-First Video Editor

The video-use roadmap prioritizes eight concrete milestones—ranging from animation source expansion to collaborative editing—that extend the existing transcript-first architecture while maintaining zero-presets and low-token design principles.

The video-use repository by browser-use implements a lightweight, transcript-first pipeline that enables language models to edit videos without loading raw frames. Understanding the roadmap for future development of video-use is essential for contributors and users who want to leverage its audio-first, self-evaluation architecture. The project maintains a strict separation between transcription, LLM reasoning, and rendering phases, with each milestone extending these layers while preserving the system's token efficiency.

Current Architecture and Design Principles

The video-use core architecture consists of three distinct layers that dictate the roadmap direction. First, audio transcription via ElevenLabs Scribe converts speech to timestamped text. Second, on-demand visual composites generated by timeline_view.py provide frame context without loading full video files. Third, an LLM-driven decision loop produces an edit decision list (EDL) that render.py executes before final output.

This design enforces text + on-demand visuals, audio-first cuts, and a self-evaluation loop that validates each cut before committing it. Consequently, the roadmap focuses on extending these pillars rather than replacing them, ensuring the system remains zero-presets and production-ready according to the "12 hard rules" defined in SKILL.md.

The 8-Milestone Development Roadmap

1. Expand Animation Sources

The first milestone adds native support for FFmpeg-based overlays, Lottie, and Web-GPU canvases. Rather than introducing new rendering engines, these formats will integrate into the existing helpers/render.py pipeline, splicing new overlay streams using the same injection method currently employed for HyperFrames, Remotion, and Manim. New modules helpers/animation_*.py will handle format-specific parsing while render.py maintains the final assembly logic.

2. UI / Web Front-End

A lightweight web interface will wrap the existing CLI tools, enabling browser-based footage uploads and cut confirmation. This implementation introduces a new app.py file providing a Flask or FastAPI wrapper that calls transcribe.py and timeline_view.py through HTTP endpoints. The UI preserves the "LLM-only" decision flow by displaying the same self-evaluation previews that the command-line tool generates.

3. Cloud-Native Deployments

To support SaaS adoption, the roadmap includes a one-click Docker image and serverless "Edit-as-a-Service" endpoints. This involves containerizing the current uv sync environment and exposing CLI entry-points through containerized APIs. The Dockerfile and docker-compose.yml will replicate the local self-evaluation loop in cloud environments without architectural changes.

4. Pluggable Transcriber Back-Ends

Future releases will abstract transcription behind a Transcriber interface, allowing Whisper, AssemblyAI, or custom ASR models to replace ElevenLabs Scribe. The helpers/transcribe.py refactor will introduce a TranscriberBase class while keeping the pack_transcripts function signature stable. This ensures downstream LLM reasoning and rendering logic remains unchanged regardless of the transcription provider.

5. Multi-Language and Localization

Non-English footage support requires extending the transcript schema to include language tags and locale-aware subtitle presets. helpers/transcribe.py will detect language codes, and helpers/timeline_view.py will render localized text overlays. The existing word-level timestamp format provides the foundation for this extension without breaking the current timeline visualization logic.

6. Collaborative Editing

Multiple agents and human reviewers will coordinate through a shared project.md lock-file and a "review-queue" system. A new helpers/collab.py module will manage concurrency, while helpers/grade.py checks the review queue before allowing the LLM to proceed past step four of the pipeline. This enables parallel cut suggestions without corrupting the EDL state.

7. Advanced Self-Evaluation Metrics

The self-evaluation loop will incorporate visual similarity (SSIM), audio continuity (spectral flux), and semantic consistency checks. These validators will hook into helpers/grade.py, extending the existing validation that currently runs timeline_view at every cut. The metrics provide quantitative guarantees for the "12 hard rules" while maintaining the asynchronous evaluation pattern.

8. Documentation and Community SDK

The final milestone releases an extensible Python SDK on PyPI, complete with type hints and example notebooks. Users will import video_use directly rather than relying on CLI invocations. Updates to pyproject.toml will define the package metadata, while the existing README.md structure provides the tutorial foundation.

Code Implementation Examples

The following snippets demonstrate how current workflows operate and how roadmap features will integrate.

Existing Workflow: Transcribe, Edit, and Render

from video_use.helpers import transcribe, render

# Step 1: Transcribe all source files using ElevenLabs Scribe

transcribe.transcribe_folder("raw_videos")

# Step 2: LLM produces EDL (executed automatically via Claude Code or similar)

# Step 3: Render final edit with overlays and color grades

render.render_final("raw_videos")

Future: Pluggable Whisper Transcriber

from video_use.helpers.transcribe import WhisperTranscriber, pack_transcripts

# Initialize alternative transcriber

transcriber = WhisperTranscriber(model="large-v2")
transcribe_folder = transcriber.transcribe_folder

# Execute same pipeline with different back-end

transcribe_folder("raw_videos")
pack_transcripts("raw_videos")

Future: Lottie Animation Overlays

from video_use.helpers.animation_lottie import LottieOverlay
from video_use.helpers import render

# Create overlay from JSON animation file

overlay = LottieOverlay("assets/intro.json", start=0, duration=5)
render.add_overlay(overlay)
render.render_final("raw_videos")

Key Files in the video-use Repository

• helpers/transcribe.py: Wrapper around ElevenLabs Scribe; primary entry point for audio-to-text conversion. Will host the TranscriberBase abstraction in milestone 4.

• helpers/timeline_view.py: Generates on-demand visual composites (filmstrip + waveform) used for the self-evaluation loop. Handles frame sampling without full video loading.

• helpers/render.py: Core rendering pipeline that assembles cuts, applies color grades, and merges animation overlays. Extended by milestone 1 for Lottie and Web-GPU support.

• helpers/grade.py: Performs self-evaluation checks after each render step. Will host SSIM and spectral flux validators in milestone 7, and check the collaborative review queue in milestone 6.

• SKILL.md: Defines the "12 hard rules" and production guidelines that constrain the LLM's decision-making process.

• README.md: High-level overview of the transcript-first pipeline and design philosophy.

Summary

• The video-use roadmap extends the existing three-layer architecture (transcription → LLM reasoning → rendering) without breaking changes.
• Animation expansion leverages render.py to support Lottie and FFmpeg overlays.
• Cloud and UI milestones containerize the current CLI workflow via app.py and Docker configurations.
• Pluggable transcribers require a TranscriberBase interface in transcribe.py to support Whisper and AssemblyAI.
• Collaborative features introduce collab.py and a project.md lock-file for multi-agent editing.
• All milestones preserve the zero-presets, low-token, and self-evaluation principles established in the current codebase.

Frequently Asked Questions

What is the video-use transcript-first pipeline?

The transcript-first pipeline is an architecture where language models edit videos by reasoning over text transcripts and on-demand visual composites rather than raw frame data. According to the browser-use/video-use source code, this approach uses helpers/transcribe.py for audio conversion, helpers/timeline_view.py for lightweight visual context, and an LLM loop to generate an edit decision list before helpers/render.py produces the final output.

How does the roadmap maintain zero-presets design?

Each milestone extends specific pipeline stages without introducing default configurations. For example, animation sources (milestone 1) inject into the existing render.py stream without preset templates, while pluggable transcribers (milestone 4) abstract the audio input stage without changing the LLM reasoning logic. The SKILL.md file enforces the "12 hard rules" that prevent hard-coded presets across all new features.

Will video-use support real-time collaborative editing?

Milestone 6 introduces collaborative editing through a shared project.md lock-file and a review queue managed by helpers/collab.py. While not real-time in the traditional sense of simultaneous frame editing, this system allows multiple agents or human reviewers to suggest and approve cuts asynchronously before the LLM proceeds to the next pipeline step.

Can I use custom transcription models with video-use?

Yes. The roadmap explicitly includes a TranscriberBase interface in helpers/transcribe.py that allows you to implement custom ASR models such as Whisper or AssemblyAI. You instantiate your custom transcriber, call transcribe_folder, and the rest of the pipeline—including pack_transcripts and render_final—functions identically to the default ElevenLabs Scribe implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}