How Hyperframes Handles HDR Video Capture and Encoding (HDR10, PQ)

Hyperframes implements end-to-end HDR10 video production by combining Puppeteer-controlled canvas capture in the BT.2020 color space with FFmpeg HEVC encoding using SMPTE 2084 (PQ) transfer characteristics and 10-bit color depth.

Hyperframes, the open-source video generation engine from HeyGen, enables high-dynamic-range (HDR) video capture and encoding through a specialized HDR-IO pipeline. The system captures raw radiance data as 16-bit PNG frames from a headless browser and transcodes them into standards-compliant HDR10 MP4 files using hardware-accelerated HEVC encoding.

The HDR-IO Pipeline Architecture

The HDR workflow in Hyperframes operates as a four-stage pipeline managed through the @hyperframes/producer and @hyperframes/engine packages:

1. Composition Configuration: The HDRIOBlock mutates the rendering context to enable HDR mode
2. Frame Capture: Puppeteer renders content to a BT.2020 canvas and exports 16-bit PNGs
3. Video Encoding: FFmpeg processes raw frames into HEVC with HDR10 metadata
4. Playback: The native player streams the HDR-enabled MP4 via standard <video> elements

Configuring HDR Mode with HDRIOBlock

To initiate HDR capture, compositions include the hdr-io block defined in packages/producer/src/blocks/hdr-io.ts. This block instantiates an HDRIOBlock class that mutates the RenderContext via ctx.setHDRMode(), accepting either pq (SMPTE 2084) or hlg (Hybrid Log-Gamma) as the EOTF (Electro-Optical Transfer Function).

The block also injects optional SMPTE-2086 master-display metadata into the context. In packages/engine/src/context.ts, the RenderContext interface exposes hdrMode and masterDisplay properties that downstream processes reference during capture and encoding.

{
  "blocks": [
    {
      "type": "hdr-io",
      "mode": "pq",
      "masterDisplay": {
        "redX": 34000,
        "redY": 16000,
        "greenX": 13250,
        "greenY": 34500,
        "blueX": 7500,
        "blueY": 3000,
        "whiteX": 15635,
        "whiteY": 16450,
        "maxLuminance": 1000,
        "minLuminance": 0
      }
    }
  ]
}

Capturing 16-Bit PNG Frames from Headless Chrome

The capture mechanism in packages/engine/src/capture.ts creates a Puppeteer-controlled page with a high-bit-depth canvas context. Rather than using standard 8-bit RGB, the pipeline initializes the canvas with colorSpace: 'display-p3' (representing BT.2020 primaries) and requests 16-bit per channel output to preserve HDR luminance values without tone-mapping.

The captureFrames function screenshots the canvas element into raw PNG buffers. These intermediates retain the full dynamic range of the source content, avoiding the crushing of highlights or loss of shadow detail that occurs in SDR conversion. Files are written as 16-bit PNGs to disk before entering the encoding queue.

// Canvas initialization for HDR capture
const canvas = await page.evaluateHandle(() => {
  const c = document.createElement('canvas');
  c.width = 1920;
  c.height = 1080;
  const ctx = c.getContext('2d', {
    colorSpace: 'display-p3',
    pixelFormat: 'float16'  // Ensures 16-bit depth retention
  });
  return c;
});

Encoding HDR10 with FFmpeg

The ffmpegEncode function in packages/engine/src/ffmpeg.ts processes the 16-bit PNG sequence into a standards-compliant HDR10 MP4. The implementation spawns FFmpeg with libx265 and injects precise color metadata required for HDR10 playback compatibility.

Key encoding parameters include:

• -pix_fmt yuv420p10le: Forces 10-bit 4:2:0 chroma subsampling
• -color_primaries bt2020: Declares BT.2020 color gamut
• -color_trc smpte2084: Sets PQ (SMPTE 2084) transfer curve
• -colorspace bt2020nc: Specifies BT.2020 non-constant luminance matrix
• -x265-params: Configures full-range quantization and 10-bit internal precision
• -metadata:s:v:0 master-display=...: Embeds SMPTE-2086 display primaries and luminance data

ffmpeg -framerate 30 -i pipe:0 \
  -c:v libx265 \
  -pix_fmt yuv420p10le \
  -color_primaries bt2020 \
  -color_trc smpte2084 \
  -colorspace bt2020nc \
  -x265-params "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:range=full" \
  -metadata:s:v:0 "master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,0)" \
  output.mp4

Native HDR Playback in the Player

Playback requires no special handling beyond serving the generated MP4. The @hyperframes/player package in packages/player/src/video.ts renders the video through a standard HTML5 <video> element. Modern browsers automatically detect the HDR10 metadata flags and switch the display into HDR mode when available hardware supports the BT.2020 gamut and PQ curve.

Summary

• HDRIOBlock configures the render context with PQ or HLG modes and optional SMPTE-2086 metadata
• Puppeteer captures 16-bit PNG frames in the BT.2020 color space without tone-mapping
• FFmpeg encodes to HEVC using 10-bit precision and embedded HDR10 mastering display metadata
• Native browser playback automatically triggers HDR mode on compatible displays when video tags carry the correct color characteristics

Frequently Asked Questions

What HDR formats does Hyperframes currently support?

Hyperframes currently supports HDR10 using the PQ (Perceptual Quantizer, SMPTE 2084) transfer function. You configure this via the mode: 'pq' property in the HDRIOBlock. Support for HLG (Hybrid Log-Gamma) is architecturally supported through the same block interface but requires explicit future implementation in the encoding pipeline.

Why does the pipeline use 16-bit PNG intermediates instead of video-native formats?

The 16-bit PNG intermediates preserve raw radiance data without tone-mapping between the browser capture and final encode. This prevents clipping of highlight details above the SDR threshold (100 nits) and ensures the BT.2020 color gamut remains intact throughout the compositing process. Using lossless PNGs also allows multiple rendering passes without generational quality loss.

How can I verify that my output video contains correct HDR10 metadata?

Run ffprobe on the generated MP4 and inspect the video stream metadata. Confirm that color_primaries equals bt2020, color_trc equals smpte2084, and colorspace equals bt2020nc. Additionally, verify the presence of the master-display metadata tag, which carries the SMPTE-2086 mastering display color volume information required for accurate HDR tone-mapping on consumer displays.

Can I override the default HDR color space or master display parameters?

Yes. The ExportConfig interface supports optional overrides including forceColorPrimaries, forceColorTransfer, and overrideMasterDisplay. When provided, the ffmpegEncode function in packages/engine/src/ffmpeg.ts prefers these user-supplied values over the defaults derived from the RenderContext, allowing custom HDR grading workflows or compatibility adjustments for specific delivery targets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}