# Hyperframes Audio Mixing Pipeline: How Composition Audio Tracks Are Processed for Final Video

> Explore the Hyperframes audio mixing pipeline, a seven-stage FFmpeg workflow that processes composition audio tracks from HTML elements to a final AAC output. Learn how audio is mixed for video.

- Repository: [HeyGen/hyperframes](https://github.com/heygen-com/hyperframes)
- Tags: internals
- Published: 2026-05-17

---

**The Hyperframes audio mixing pipeline processes composition audio tracks through a seven-stage FFmpeg-driven workflow in [`packages/engine/src/services/audioMixer.ts`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/services/audioMixer.ts) that parses HTML elements, resolves remote sources, extracts PCM audio, and mixes tracks into a single AAC output.**

The audio mixing pipeline in the [heygen-com/hyperframes](https://github.com/heygen-com/hyperframes) repository transforms discrete `<audio>` and `<video data-has-audio="true">` elements from composition HTML into a single, time-synchronized audio track ready for final video encoding. Implemented in the engine package, this deterministic system handles everything from remote asset downloading to FFmpeg filter-complex generation, ensuring precise audio alignment with composition timelines according to the Hyperframes source code.

## Overview of the Audio Processing Architecture

The core logic resides in [`packages/engine/src/services/audioMixer.ts`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/services/audioMixer.ts), which exports `processCompositionAudio` as the primary orchestration function. This service coordinates with [`packages/engine/src/utils/ffprobe.js`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/utils/ffprobe.js) for metadata extraction, [`packages/engine/src/utils/urlDownloader.js`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/utils/urlDownloader.js) for remote assets, and [`packages/engine/src/utils/runFfmpeg.js`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/utils/runFfmpeg.js) for command execution. Type definitions in [`packages/engine/src/services/audioMixer.types.ts`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/services/audioMixer.types.ts) define the `AudioElement`, `AudioTrack`, and `MixResult` interfaces that structure data throughout the pipeline.

## Stage-by-Stage Pipeline Breakdown

### Parsing Audio Elements from Composition HTML

The pipeline begins with `parseAudioElements(html)`, which scans composition HTML between lines 27-81 in [`audioMixer.ts`](https://github.com/heygen-com/hyperframes/blob/main/audioMixer.ts). This function identifies `<audio>` tags and `<video>` elements marked with `data-has-audio="true"`, constructing an array of `AudioElement` objects containing source URLs, start/end times, and volume levels.

### Resolving Source Paths and Downloading Remote Assets

For each element, `processCompositionAudio` (lines 28-45) invokes `resolveProjectRelativeSrc` to normalize filesystem paths. When sources are HTTP URLs, the pipeline calls `downloadToTemp` from [`urlDownloader.js`](https://github.com/heygen-com/hyperframes/blob/main/urlDownloader.js) to cache files locally before processing, ensuring the mixer works with accessible filesystem assets regardless of origin.

### Probing Durations and Extracting Raw Audio

When an element lacks explicit duration metadata (end-start ≤ 0), `extractAudioMetadata`—an ffprobe wrapper—reads the actual file length (lines 51-57). The pipeline then branches based on source type:

- **Video sources**: `extractAudioFromVideo` (lines 84-119) uses FFmpeg to extract PCM WAV audio from the specified time range.
- **Audio sources**: `prepareAudioTrack` (lines 122-171) trims, offsets, and converts files to PCM format using FFmpeg.

### Building the FFmpeg Filter-Complex for Mixing

The `mixAudioTracks` function (lines 178-260) constructs a sophisticated FFmpeg filter-complex:

1. **Per-track processing**: Each `AudioTrack` receives `atrim` (trimming), `volume` (individual gain), and `adelay` (timeline positioning) filters.
2. **Combination**: Streams labeled `[aN]` feed into the `amix` filter with dropout=0 and normalize=0.
3. **Master gain**: Final output applies `volume=${masterOutputGain}` before encoding to AAC.

### Handling Silent Compositions and Cleanup

If no audio tracks exist, `generateSilence` creates a silent PCM placeholder matching the composition's total duration. After mixing, temporary work files are removed, and `processCompositionAudio` (lines 108-122) returns a `MixResult` object containing the output path, processing time, track count, and any accumulated warnings.

## Implementation Example: Processing a Composition

```typescript
import { readFileSync } from "fs";
import { parseAudioElements, processCompositionAudio } from "hyperframes/packages/engine/src/services/audioMixer.js";

const html = readFileSync("example/composition.html", "utf8");

// Extract audio elements from HTML
const audioElements = parseAudioElements(html);

// Define composition length in seconds
const totalDuration = 30;

// Execute full pipeline
const mixResult = await processCompositionAudio(
  audioElements,
  process.cwd(),          // baseDir
  "./tmp/audio-work",     // workDir
  "./out/mixed.aac",      // outputPath
  totalDuration,
);

console.log(mixResult.success, mixResult.outputPath);

```

## Advanced: Manual Track Mixing

For scenarios requiring direct track manipulation without HTML parsing, `mixAudioTracks` accepts pre-configured `AudioTrack` arrays:

```typescript
import { mixAudioTracks } from "hyperframes/packages/engine/src/services/audioMixer.js";

const tracks = [
  {
    id: "bg",
    srcPath: "./tmp/audio-work/bg.wav",
    start: 0,
    end: 30,
    mediaStart: 0,
    duration: 30,
    volume: 0.6,
  },
  {
    id: "voice",
    srcPath: "./tmp/audio-work/voice.wav",
    start: 5,
    end: 25,
    mediaStart: 0,
    duration: 20,
    volume: 1.0,
  },
];

const result = await mixAudioTracks(
  tracks,
  "./out/mixed.aac",
  30,  // totalDuration
);

```

## Key Files and Dependencies

- **[`packages/engine/src/services/audioMixer.ts`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/services/audioMixer.ts)**: Core orchestration containing `parseAudioElements`, `processCompositionAudio`, and `mixAudioTracks`.
- **[`packages/engine/src/services/audioMixer.types.ts`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/services/audioMixer.types.ts)**: Interface definitions for `AudioElement`, `AudioTrack`, and `MixResult`.
- **[`packages/engine/src/utils/ffprobe.js`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/utils/ffprobe.js)**: Metadata extraction wrapper for determining audio durations.
- **[`packages/engine/src/utils/runFfmpeg.js`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/utils/runFfmpeg.js)**: Command execution utility with timeout and abort signal support.
- **[`packages/engine/src/utils/urlDownloader.js`](https://github.com/heygen-com/hyperframes/blob/main/packages/engine/src/utils/urlDownloader.js)**: HTTP asset retrieval for remote audio/video sources.

## Summary

- The **Hyperframes audio mixing pipeline** processes composition audio through seven deterministic stages in [`audioMixer.ts`](https://github.com/heygen-com/hyperframes/blob/main/audioMixer.ts).
- **FFmpeg** drives both extraction (PCM conversion) and mixing (filter-complex with `amix`).
- The pipeline handles **remote URLs** via temporary downloads and **video files** via audio extraction.
- **Volume control** operates at both individual track and master output levels.
- **Silent compositions** automatically generate placeholder audio to prevent rendering errors.

## Frequently Asked Questions

### What audio formats does Hyperframes output?

The pipeline produces **AAC** files as final output, while using **PCM WAV** as the intermediate format for processing. All FFmpeg conversion handles the transcoding automatically during the `prepareAudioTrack` or `extractAudioFromVideo` stages.

### How does Hyperframes handle video files that contain audio?

Video elements marked with `data-has-audio="true"` trigger `extractAudioFromVideo` (lines 84-119), which uses FFmpeg to extract the audio stream to PCM WAV format before mixing. This allows video and standalone audio elements to coexist on the same timeline.

### What happens if my composition has no audio tracks?

The `generateSilence` function creates a silent audio track matching the composition's total duration. This ensures the final video renders with a valid audio stream even when no `<audio>` or video audio elements are present.

### Can I adjust individual track volumes in the mixing pipeline?

Yes. Each `AudioTrack` object includes a `volume` property (0.0 to 1.0+) that the `mixAudioTracks` function applies via FFmpeg's `volume` filter before the `amix` combination. A master output gain is applied to the final mixed result.