Segment Extraction and Concatenation in video-use: FFmpeg-Based Pipeline Explained

video-use processes source videos by cutting them into individual segments with optional per-segment colour-grading and audio fades, then stitches them back together using FFmpeg's concat demuxer in copy mode to avoid re-encoding.

The browser-use/video-use project implements a deterministic video rendering pipeline that treats edit-decision-lists (EDLs) as the single source of truth. According to the video-use source code, all extraction and concatenation logic lives in helpers/render.py and follows a strict three-phase workflow: per-segment extraction, batch processing, and lossless concatenation.

How Per-Segment Extraction Works

The foundation of video-use's segment extraction strategy is the extract_segment() function. This function receives a source video path, start time, duration, and filter string, then constructs an FFmpeg command that applies visual and audio processing in a single pass.

Building the FFmpeg Filter Chain

For each segment, extract_segment() assembles a command following this structure:

ffmpeg -ss <start> -t <duration> -i <source> \
  -vf "<colour_grade_filter>" \
  -af "afade=t=in:st=0:d=0.03,afade=t=out:st=<duration-0.03>:d=0.03" \
  -c:v libx264 -crf 18 -pix_fmt yuv420p \
  <output_clip>

The audio fade implementation hard-codes 30-millisecond fade-in and fade-out envelopes using afade filters. This eliminates click artifacts at segment boundaries without perceptibly altering the audio content.

Auto-Grade vs. Static Preset Handling

The filter string passed to extract_segment() originates from one of two sources:

• auto_grade_for_clip() – Computes scene-specific colour-grading parameters based on segment content luminance and chroma distribution. Triggered when grade="auto".
• Named preset – A static filters string from video-use's preset library (e.g., "warm_vintage", "high_contrast").

This design enables adaptive visual treatment where each segment receives grades calibrated to its own content characteristics.

Batch Extraction of All Segments

The extract_all_segments() function orchestrates the complete extraction phase. It iterates over EDL ranges, determines the appropriate grade strategy, and delegates to extract_segment() for each clip.

# Simplified workflow from helpers/render.py

segment_paths = extract_all_segments(
    src_path="/path/to/source.mp4",
    ranges=[(0.0, 5.5), (5.5, 12.0), (12.0, 18.5)],  # (start, duration) tuples

    grade="auto",                                   # or preset string

    edit_dir=Path("./workspace/edit_001"),
    preview=False,
    draft=False,
)

Key behaviors of extract_all_segments():

• Creates a dedicated clips subdirectory under edit_dir for temporary segment files.
• Maintains segment order to preserve EDL sequencing.
• Returns an ordered list of absolute paths ready for concatenation.
• Applies consistent encoding settings (H.264, CRF 18, 4:2:0 chroma subsampling) across all clips to ensure codec compatibility during final assembly.

Concatenation Strategy: Lossless Assembly

The concat_segments() function handles final video assembly using FFmpeg's concat demuxer in copy mode (-c copy). This architectural choice is central to video-use's performance characteristics.

The Concat List File

Before execution, concat_segments() writes a temporary text file following FFmpeg's concat syntax:

file '/absolute/path/to/workspace/edit_001/clips/segment_000.mp4'
file '/absolute/path/to/workspace/edit_001/clips/segment_001.mp4'
file '/absolute/path/to/workspace/edit_001/clips/segment_002.mp4'

Each line specifies an absolute path wrapped in single quotes, with the file keyword prefix required by the demuxer.

The Final FFmpeg Command

The resulting concatenation command executes as:

ffmpeg -f concat -safe 0 -i <concat_list_path> \
  -c copy \
  -movflags +faststart \
  <output_path>

Critical flags explained:

• -f concat – Forces use of the concat demuxer rather than the concat filter.
• -safe 0 – Allows absolute paths and special characters in filenames (required for cross-platform compatibility).
• -c copy – Streams are copied without re-encoding, preserving original codecs and achieving near-instant assembly regardless of video length.
• -movflags +faststart – Moov atom repositioning for progressive download compatibility.

Why Copy Mode Matters

The video-use source code explicitly avoids the concat filter ([v:0][v:1][v:2]concat=n=3[v]), which would force a full decode-encode cycle. Instead, the concat demuxer operates at the container level, requiring that all input segments share identical:

• Video codec (H.264 profile/level)
• Audio codec (AAC with matching sample rate)
• Resolution and pixel format
• Timebase

extract_all_segments() guarantees this consistency by re-encoding every segment with uniform parameters during extraction, making the subsequent copy-mode concatenation valid.

Pipeline Architecture Benefits

This segment extraction and concatenation strategy delivers three operational advantages:

Benefit	Mechanism	Source Implementation
Granular visual control	Per-segment filter application via extract_segment()	helpers/render.py:extract_segment()
Fast final renders	Copy-mode concatenation avoids re-encoding	helpers/render.py:concat_segments()
Deterministic output	Linear pipeline with explicit intermediate files	Batch extraction → ordered list → concat demuxer

The intermediate file approach also simplifies debugging: developers can inspect individual segment_NNN.mp4 files to verify colour-grading and audio fade behavior before investigating concatenation issues.

Integration with the Rendering Workflow

The complete segment extraction and concatenation sequence executes as a coordinated pair:

from helpers.render import extract_all_segments, concat_segments

# Phase 1: Extract all EDL-defined segments with processing

segment_paths = extract_all_segments(
    src_path=edl.source_path,
    ranges=edl.ranges,
    grade=edl.grade,
    edit_dir=edit_dir,
    preview=is_preview,
    draft=is_draft,
)

# Phase 2: Assemble final output without re-encoding

final_output = edit_dir / "final.mp4"
concat_segments(segment_paths, final_output, edit_dir)

The timeline_view.py module generates the edl.ranges data structure that drives this pipeline, converting user interface interactions into precise (start, duration) tuples.

Summary

• extract_segment() in helpers/render.py cuts individual clips with FFmpeg, applying colour-grades and 30ms audio fades using -vf and -af filter chains.
• extract_all_segments() batches the extraction across an entire EDL, selecting between auto-graded or preset filters for each range.
• concat_segments() writes a concat list file and invokes FFmpeg with -f concat -safe 0 -c copy for lossless, non-destructive assembly.
• The strategy trades temporary disk space (individual segment files) for rendering speed and per-segment processing flexibility.

Frequently Asked Questions

Why does video-use extract segments to separate files instead of using FFmpeg's concat filter?

Separate file extraction enables per-segment colour-grading and audio processing that would be impossible with the concat filter alone. The concat filter operates during encoding and cannot apply different filters to each input segment. By extracting processed clips first, video-use achieves granular visual control while still using copy-mode concatenation for final assembly speed.

What happens if segments have different resolutions or codecs?

The pipeline prevents this condition. extract_all_segments() re-encodes every segment with identical parameters (H.264, consistent resolution, 4:2:0 chroma subsampling) during extraction. This guarantees codec compatibility required for the copy-mode concatenation in concat_segments().

How does the auto-grade feature determine colour settings for each segment?

When grade="auto", the code calls auto_grade_for_clip() (internal helper in helpers/render.py) to analyze segment luminance and chroma statistics. It computes adaptive brightness, contrast, and saturation adjustments specific to that clip's content, then injects the resulting filter string into the FFmpeg command for that segment only.

Can the audio fade duration be customized from the default 30ms?

The current implementation in helpers/render.py hard-codes 30-millisecond fades in the afade filter expressions within extract_segment(). Modification would require editing the source code where the filter string is constructed: afade=t=in:st=0:d=0.03 defines the fade-in, and the fade-out calculation uses duration - 0.03 as the start point.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}