How Meetily’s FFmpeg Audio Mixer Implements Adaptive Buffering and Ducking

Meetily’s FFmpeg-style audio mixer uses device-specific timeout windows (20–50 ms for wired, 80–200 ms for Bluetooth) and RMS-based ducking to auto-lower system audio when speech is detected, ensuring clear voice capture and synchronized output.

Meetily’s audio pipeline, located in the frontend/src-tauri/src/audio/ directory of the Zackriya-Solutions/meetily repository, implements an FFmpeg-style adaptive mixer that handles heterogeneous input devices with varying latency characteristics. The ffmpeg_mixer.rs file contains the core logic for per-source buffering and automatic ducking, while device_detection.rs classifies hardware to optimize buffer timeouts. This architecture ensures reliable synchronization between microphone and system audio streams without manual configuration.

Adaptive Buffering Strategy

The mixer maintains separate SourceBuffer instances for microphone and system audio, adapting its buffering strategy based on detected device types to minimize latency while preventing underruns.

Device Type Detection and Classification

Device classification begins with InputDeviceKind::detect in frontend/src-tauri/src/audio/device_detection.rs (line 27). This function analyzes device names, native OS hints, and buffer-size heuristics to classify sources as Wired, Bluetooth, or Unknown, allowing the mixer to apply appropriate timing strategies for each hardware category.

Per-Source Timeout Management

Each SourceBuffer retrieves device-specific timeout ranges via InputDeviceKind::buffer_timeout at line 71 of device_detection.rs. Wired devices receive a tight 20–50 ms window, while Bluetooth devices receive a generous 80–200 ms window to accommodate jitter. When constructing a buffer in SourceBuffer::new (lines 89–90 of ffmpeg_mixer.rs), the maximum timeout from the range is selected as the initial conservative buffer threshold.

Gap Detection and Silence Handling

When new chunks arrive, SourceBuffer::push (lines 138–160) checks the elapsed time since the previous chunk. If the gap exceeds twice the expected chunk duration, the mixer records a gap and logs a debug warning for Bluetooth devices. The SourceBuffer::has_data method (lines 168–176) returns true only when the oldest chunk has been present longer than the adaptive timeout, ensuring Bluetooth jitter does not trigger premature mixing.

If a buffer exhausts its data while the mixer pulls a 50 ms window, SourceBuffer::pop_samples (lines 199–212) fills missing samples with zeros and records the amount of inserted silence, maintaining continuous output without crashing the pipeline.

RMS-Based Audio Ducking

The AudioMixer struct in ffmpeg_mixer.rs implements professional-grade ducking by monitoring microphone levels and dynamically attenuating system audio when speech is present.

struct AudioMixer {
    mic_ducking: f32,
    system_ducking: f32,
    adaptive_ducking: bool,
}

Speech Detection Thresholds

During the mixing process in AudioMixer::mix, the system calculates the root-mean-square (RMS) of microphone samples using calculate_rms (lines 48–56). When the computed mic_rms exceeds SPEECH_THRESHOLD (0.01), the mixer classifies the input as active speech.

Dynamic Gain Adjustment

If speech is detected, the mixer applies system_ducking (approximately 60 % volume, or 0.6 gain) to the system audio track. When the microphone is silent, system audio plays at full volume (1.0 gain). The final mixed sample combines the microphone sample (scaled by mic_ducking) and the system sample (scaled by the adaptive system_gain), clamped to [-1.0, 1.0] to prevent clipping.

let mic_rms = calculate_rms(mic);
let is_speech = mic_rms > SPEECH_THRESHOLD;      // 0.01
let system_gain = if is_speech {
    self.system_ducking   // duck system audio to ~60%
} else {
    1.0                  // full system audio
};

Practical Implementation Example

The following Rust code demonstrates how to instantiate the adaptive mixer and process audio streams:

use meetily::audio::ffmpeg_mixer::FFmpegAudioMixer;
use meetily::audio::device_detection::InputDeviceKind;

// 1. Create the adaptive mixer with 50 ms windows
let mut mixer = FFmpegAudioMixer::new(
    "Built‑in Mic".to_string(),
    InputDeviceKind::Wired,
    "BlackHole 2ch".to_string(),
    InputDeviceKind::Wired,
    48_000,
);

// 2. Push raw audio chunks as they arrive (e.g., from cpal streams)
mixer.push_mic(mic_chunk);
mixer.push_system(system_chunk);

// 3. Pull a mixed window only when both buffers are ready
if let Some(mixed) = mixer.pop_mixed() {
    // mixed is a Vec<f32> of 50 ms audio ready for recording or Whisper
    process_mixed_audio(mixed);
}

// 4. Inspect buffer health for debugging
let (mic_stats, sys_stats) = mixer.get_stats();
log::info!("Mic buffer latency: {:.0} ms", mic_stats.buffer_latency_ms);
log::info!("System buffer latency: {:.0} ms", sys_stats.buffer_latency_ms);

This workflow constructs the mixer with adaptive ducking enabled, feeds per-source samples through the internal SourceBuffer logic, and returns synchronized 50 ms windows only after both sources satisfy their device-specific timeouts.

Summary

• Device-specific buffering: The mixer detects hardware types via InputDeviceKind::detect and applies tailored timeout ranges (20–50 ms for wired, 80–200 ms for Bluetooth) to balance latency and stability.
• Gap tolerance: SourceBuffer::push tracks inter-chunk gaps, while SourceBuffer::has_data enforces adaptive timeouts to prevent Bluetooth underruns.
• RMS-based ducking: The mixer calculates microphone RMS levels in real-time, automatically reducing system audio to ~60 % volume when speech exceeds the 0.01 threshold.
• Continuous output: SourceBuffer::pop_samples inserts silence zeros when buffers underflow, ensuring the mixing pipeline never blocks or crashes.

Frequently Asked Questions

How does Meetily’s FFmpeg audio mixer handle Bluetooth audio latency differently from wired devices?

According to the source code in frontend/src-tauri/src/audio/device_detection.rs, the mixer classifies devices using InputDeviceKind::detect and assigns timeout ranges via buffer_timeout. Wired devices receive 20–50 ms windows, while Bluetooth devices receive 80–200 ms windows. This extra headroom in SourceBuffer::has_data prevents underruns caused by Bluetooth jitter while keeping wired latency minimal.

What threshold triggers the ducking effect in the audio mixer?

The ducking activates when the microphone’s RMS level exceeds SPEECH_THRESHOLD, defined as 0.01 in the mixing logic. When calculate_rms (lines 48–56 of ffmpeg_mixer.rs) returns a value above this threshold, AudioMixer::mix applies system_ducking (approximately 0.6 gain) to reduce system audio volume and prioritize voice clarity.

How does the mixer recover from buffer underruns or missing audio chunks?

If SourceBuffer::pop_samples exhausts available data while pulling a 50 ms mixing window, it automatically fills the missing segment with zero-value samples (silence) and records the insertion amount in the buffer statistics (lines 199–212 of ffmpeg_mixer.rs). This ensures continuous output without blocking the consumer thread.

Where is the adaptive buffering logic implemented in the Meetily repository?

The core adaptive buffering and ducking logic resides in frontend/src-tauri/src/audio/ffmpeg_mixer.rs, which defines SourceBuffer, AudioMixer, and FFmpegAudioMixer. Device classification logic is located in frontend/src-tauri/src/audio/device_detection.rs, specifically the InputDeviceKind enum and its detect method (line 27).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}