Hyperframes Audio Mixing Pipeline: How Composition Audio Tracks Are Processed for Final Video

The Hyperframes audio mixing pipeline processes composition audio tracks through a seven-stage FFmpeg-driven workflow in packages/engine/src/services/audioMixer.ts that parses HTML elements, resolves remote sources, extracts PCM audio, and mixes tracks into a single AAC output.

The audio mixing pipeline in the heygen-com/hyperframes repository transforms discrete <audio> and <video data-has-audio="true"> elements from composition HTML into a single, time-synchronized audio track ready for final video encoding. Implemented in the engine package, this deterministic system handles everything from remote asset downloading to FFmpeg filter-complex generation, ensuring precise audio alignment with composition timelines according to the Hyperframes source code.

Overview of the Audio Processing Architecture

The core logic resides in packages/engine/src/services/audioMixer.ts, which exports processCompositionAudio as the primary orchestration function. This service coordinates with packages/engine/src/utils/ffprobe.js for metadata extraction, packages/engine/src/utils/urlDownloader.js for remote assets, and packages/engine/src/utils/runFfmpeg.js for command execution. Type definitions in packages/engine/src/services/audioMixer.types.ts define the AudioElement, AudioTrack, and MixResult interfaces that structure data throughout the pipeline.

Stage-by-Stage Pipeline Breakdown

Parsing Audio Elements from Composition HTML

The pipeline begins with parseAudioElements(html), which scans composition HTML between lines 27-81 in audioMixer.ts. This function identifies <audio> tags and <video> elements marked with data-has-audio="true", constructing an array of AudioElement objects containing source URLs, start/end times, and volume levels.

Resolving Source Paths and Downloading Remote Assets

For each element, processCompositionAudio (lines 28-45) invokes resolveProjectRelativeSrc to normalize filesystem paths. When sources are HTTP URLs, the pipeline calls downloadToTemp from urlDownloader.js to cache files locally before processing, ensuring the mixer works with accessible filesystem assets regardless of origin.

Probing Durations and Extracting Raw Audio

When an element lacks explicit duration metadata (end-start ≤ 0), extractAudioMetadata—an ffprobe wrapper—reads the actual file length (lines 51-57). The pipeline then branches based on source type:

• Video sources: extractAudioFromVideo (lines 84-119) uses FFmpeg to extract PCM WAV audio from the specified time range.
• Audio sources: prepareAudioTrack (lines 122-171) trims, offsets, and converts files to PCM format using FFmpeg.

Building the FFmpeg Filter-Complex for Mixing

The mixAudioTracks function (lines 178-260) constructs a sophisticated FFmpeg filter-complex:

1. Per-track processing: Each AudioTrack receives atrim (trimming), volume (individual gain), and adelay (timeline positioning) filters.
2. Combination: Streams labeled [aN] feed into the amix filter with dropout=0 and normalize=0.
3. Master gain: Final output applies volume=${masterOutputGain} before encoding to AAC.

Handling Silent Compositions and Cleanup

If no audio tracks exist, generateSilence creates a silent PCM placeholder matching the composition's total duration. After mixing, temporary work files are removed, and processCompositionAudio (lines 108-122) returns a MixResult object containing the output path, processing time, track count, and any accumulated warnings.

Implementation Example: Processing a Composition

import { readFileSync } from "fs";
import { parseAudioElements, processCompositionAudio } from "hyperframes/packages/engine/src/services/audioMixer.js";

const html = readFileSync("example/composition.html", "utf8");

// Extract audio elements from HTML
const audioElements = parseAudioElements(html);

// Define composition length in seconds
const totalDuration = 30;

// Execute full pipeline
const mixResult = await processCompositionAudio(
  audioElements,
  process.cwd(),          // baseDir
  "./tmp/audio-work",     // workDir
  "./out/mixed.aac",      // outputPath
  totalDuration,
);

console.log(mixResult.success, mixResult.outputPath);

Advanced: Manual Track Mixing

For scenarios requiring direct track manipulation without HTML parsing, mixAudioTracks accepts pre-configured AudioTrack arrays:

import { mixAudioTracks } from "hyperframes/packages/engine/src/services/audioMixer.js";

const tracks = [
  {
    id: "bg",
    srcPath: "./tmp/audio-work/bg.wav",
    start: 0,
    end: 30,
    mediaStart: 0,
    duration: 30,
    volume: 0.6,
  },
  {
    id: "voice",
    srcPath: "./tmp/audio-work/voice.wav",
    start: 5,
    end: 25,
    mediaStart: 0,
    duration: 20,
    volume: 1.0,
  },
];

const result = await mixAudioTracks(
  tracks,
  "./out/mixed.aac",
  30,  // totalDuration
);

Key Files and Dependencies

• packages/engine/src/services/audioMixer.ts: Core orchestration containing parseAudioElements, processCompositionAudio, and mixAudioTracks.
• packages/engine/src/services/audioMixer.types.ts: Interface definitions for AudioElement, AudioTrack, and MixResult.
• packages/engine/src/utils/ffprobe.js: Metadata extraction wrapper for determining audio durations.
• packages/engine/src/utils/runFfmpeg.js: Command execution utility with timeout and abort signal support.
• packages/engine/src/utils/urlDownloader.js: HTTP asset retrieval for remote audio/video sources.

Summary

• The Hyperframes audio mixing pipeline processes composition audio through seven deterministic stages in audioMixer.ts.
• FFmpeg drives both extraction (PCM conversion) and mixing (filter-complex with amix).
• The pipeline handles remote URLs via temporary downloads and video files via audio extraction.
• Volume control operates at both individual track and master output levels.
• Silent compositions automatically generate placeholder audio to prevent rendering errors.

Frequently Asked Questions

What audio formats does Hyperframes output?

The pipeline produces AAC files as final output, while using PCM WAV as the intermediate format for processing. All FFmpeg conversion handles the transcoding automatically during the prepareAudioTrack or extractAudioFromVideo stages.

How does Hyperframes handle video files that contain audio?

Video elements marked with data-has-audio="true" trigger extractAudioFromVideo (lines 84-119), which uses FFmpeg to extract the audio stream to PCM WAV format before mixing. This allows video and standalone audio elements to coexist on the same timeline.

What happens if my composition has no audio tracks?

The generateSilence function creates a silent audio track matching the composition's total duration. This ensures the final video renders with a valid audio stream even when no <audio> or video audio elements are present.

Can I adjust individual track volumes in the mixing pipeline?

Yes. Each AudioTrack object includes a volume property (0.0 to 1.0+) that the mixAudioTracks function applies via FFmpeg's volume filter before the amix combination. A master output gain is applied to the final mixed result.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}