How the Auto-Grade Feature Analyzes Video Segments and Applies Per-Clip Color Corrections

The auto-grade feature samples frame-level statistics using FFmpeg's signalstats filter, derives modest adjustments for contrast, gamma, and saturation clamped to ±8%, and injects a tailored eq= filter string into each clip's extraction pipeline.

The browser-use/video-use repository automates color grading through its intelligent auto-grade mode. This feature eliminates manual correction by analyzing luminance and saturation data from each source segment and computing subtle, data-driven adjustments. The resulting corrections are applied individually to every extracted clip before final concatenation, ensuring consistent exposure across the rendered timeline.

Frame-Level Statistics Collection

The analysis begins in helpers/grade.py with the _sample_frame_stats function (lines 78-99). This utility probes a short temporal slice of the source video to gather quantitative metrics.

The function constructs an FFmpeg command that seeks to a specific start time, limits analysis to a specified duration, and decodes n_samples frames (default 10) while applying the signalstats filter:

def _sample_frame_stats(
    video: Path,
    start: float,
    duration: float,
    n_samples: int = 10,
) -> dict[str, float]:
    ...
    cmd = [
        "ffmpeg", "-y", "-hide_banner", "-nostats",
        "-ss", f"{start:.3f}",
        "-i", str(video),
        "-t", f"{duration:.3f}",
        "-vf", f"fps={fps:.2f},signalstats,metadata=print:file={metadata_path}",
        "-f", "null", "-",
    ]
    subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The signalstats filter writes per-frame metadata—specifically YAVG, YMIN, YMAX, and SATAVG—to a temporary text file. The routine parses these values, normalizes them to a 0-1 range independent of source bit depth, and returns three key metrics:

• y_mean: Average luma (brightness)
• y_std: Estimated luma standard deviation (used to derive range)
• sat_mean: Average saturation

Computing Correction Values

The auto_grade_for_clip function (lines 84-122 and 124-149 in helpers/grade.py) transforms the sampled statistics into concrete FFmpeg eq= filter parameters. The algorithm targets a "clean, not graded" look by bounding all adjustments to ±8%.

Contrast Adjustment

The system targets a luma range of approximately 0.72. When the sampled y_range (derived as y_std * 4.0) falls below 0.65, contrast is gently boosted up to +8%. Clips with adequate range receive a minimal +3% adjustment:

contrast_adj = 1.0
if y_range < 0.65:
    t = max(0.0, min(1.0, (y_range - 0.50) / 0.15))
    contrast_adj = 1.08 - 0.05 * t
else:
    contrast_adj = 1.03

Gamma (Brightness) Adjustment

Targeting a mean luma of approximately 0.48, the function lifts dark clips (y_mean < 0.42) and slightly pulls back very bright clips (y_mean > 0.60):

gamma_adj = 1.0
if y_mean < 0.42:
    t = max(0.0, min(1.0, (y_mean - 0.30) / 0.12))
    gamma_adj = 1.10 - 0.08 * t
elif y_mean > 0.60:
    gamma_adj = 0.97

Saturation Adjustment

The baseline applies a slight desaturation (0.98). Flat clips (sat_mean < 0.18) receive a +4% boost, while overly punchy clips (sat_mean > 0.38) are reduced by 4%:

sat_adj = 0.98
if sat_mean < 0.18:
    sat_adj = 1.04
elif sat_mean > 0.38:
    sat_adj = 0.96

Clamping and Filter Construction

All values are hard-clamped to prevent extreme adjustments:

• Contrast: 0.94 to 1.08
• Gamma: 0.94 to 1.10
• Saturation: 0.94 to 1.06

The function constructs the eq= string by concatenating only non-neutral values (those differing from 1.0 by more than 0.005):

eq_parts = []
if abs(contrast_adj - 1.0) > 0.005:
    eq_parts.append(f"contrast={contrast_adj:.3f}")
if abs(gamma_adj - 1.0) > 0.005:
    eq_parts.append(f"gamma={gamma_adj:.3f}")
if abs(sat_adj - 1.0) > 0.005:
    eq_parts.append(f"saturation={sat_adj:.3f}")

filter_string = "" if not eq_parts else "eq=" + ":".join(eq_parts)

Per-Clip Injection in the Rendering Pipeline

The rendering logic in helpers/render.py orchestrates the per-clip application of these corrections. When an edit-decision list (EDL) specifies "grade": "auto", the resolve_grade_filter helper (lines 66-76) maps this to a sentinel value "__AUTO__".

Inside extract_all_segments, the pipeline iterates through each range in the EDL. When is_auto is true, it invokes auto_grade_for_clip for every segment, passing the specific start and duration of that clip:

def extract_all_segments(
    edl: dict,
    edit_dir: Path,
    preview: bool,
    draft: bool = False,
) -> list[Path]:
    ...
    resolved = resolve_grade_filter(edl.get("grade"))
    is_auto = resolved == "__AUTO__"
    ...
    for i, r in enumerate(ranges):
        ...
        if is_auto:
            seg_filter, _stats = auto_grade_for_clip(src_path,
                                                    start=start,
                                                    duration=duration,
                                                    verbose=False)
        else:
            seg_filter = resolved
        ...
        extract_segment(src_path, start, duration, seg_filter,
                        out_path, preview=preview, draft=draft)

The generated eq= string is passed to extract_segment, which inserts it into the FFmpeg filter graph (lines 79-86 and 180-185). The graph typically processes optional HDR tone-mapping, scaling, the per-clip color-grade filter, and audio fades. This ensures each extracted MP4 segment receives its own tailored correction before lossless concatenation.

Practical Usage Examples

Analyzing a Clip via CLI

You can preview the auto-grade analysis for any video file without rendering:

python helpers/grade.py --analyze path/to/clip.mp4

Typical output shows the derived statistics and resulting filter:

auto-grade stats:
  y_mean=0.38  y_range=0.54  sat_mean=0.22
  → contrast=1.072  gamma=1.080  sat=1.040
  → filter: eq=contrast=1.072:gamma=1.080:saturation=1.040
filter: eq=contrast=1.072:gamma=1.080:saturation=1.040

Rendering an EDL with Auto-Grade

Create an EDL with the grade field set to "auto":

{
  "grade": "auto",
  "ranges": [
    {"source": "src1", "start": 0, "end": 12},
    {"source": "src2", "start": 5, "end": 20}
  ],
  "sources": {
    "src1": "videos/a.mp4",
    "src2": "videos/b.mp4"
  }
}

Run the render pipeline:

python helpers/render.py edl.json -o final.mp4

The console output confirms per-clip analysis:

extracting 2 segment(s) → clips_graded/
  (auto-grade per segment: analyzing each range)
  [00] src1  0.00-12.00  (12.00s)  
        grade: eq=contrast=1.045:gamma=1.020:saturation=0.990
  [01] src2  5.00-20.00  (15.00s)  
        grade: (none)

Summary

• Statistical Sampling: The _sample_frame_stats function in helpers/grade.py uses FFmpeg's signalstats filter to measure y_mean, y_std, and sat_mean from a 10-frame sample.
• Bounded Corrections: The auto_grade_for_clip function calculates contrast, gamma, and saturation adjustments clamped to ±8% to maintain a natural look.
• Per-Clip Application: The rendering pipeline in helpers/render.py resolves the "auto" grade setting to "__AUTO__", triggering individual analysis and eq= filter injection for every segment during extraction.
• Integration: Corrections are applied within the FFmpeg filter graph before concatenation, ensuring each clip in the final output retains its tailored color correction.

Frequently Asked Questions

How does the auto-grade feature determine the correction values for each clip?

The feature analyzes a 10-frame sample from the specific time range of each clip using FFmpeg's signalstats filter. It extracts average luma (y_mean), luma range (y_std), and average saturation (sat_mean). Based on these metrics, it calculates contrast, gamma, and saturation multipliers: contrast is boosted when the luma range is narrow, gamma is adjusted when the mean is too dark or bright, and saturation is nudged based on flatness or punchiness. All adjustments are clamped to ±8% to avoid an artificial appearance.

What is the difference between "auto" and a manual grade in the EDL?

When the EDL's grade field is set to "auto", the resolve_grade_filter function in helpers/render.py converts this to an internal "__AUTO__" sentinel. This triggers the per-clip analysis loop, calling auto_grade_for_clip for every segment. If a manual grade is specified (e.g., a specific eq= filter string), that string is passed directly to all segments without statistical analysis, applying a uniform correction across the entire timeline.

Why are the corrections limited to ±8%?

The ±8% clamping (implemented via max/min constraints in helpers/grade.py lines 124-149) ensures the output remains "clean, not graded." This prevents aggressive corrections that would create an artificial or heavily processed look, maintaining the natural characteristics of the source footage while correcting only exposure and color imbalances.

Where does the per-clip filter actually get applied in the FFmpeg pipeline?

The generated eq= filter string is injected into the filter graph within the extract_segment function in helpers/render.py (lines 180-185). The graph processes the video through optional HDR tone-mapping and scaling, then applies the color-grade filter, followed by audio fades. Each segment is rendered to a separate MP4 file with its specific correction before all segments are concatenated losslessly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}