Meetily Audio Pipeline Architecture for Simultaneous Microphone and System Audio Capture

Meetily captures independent PCM streams from microphone and system audio, synchronizes them through a VAD-driven ring buffer pipeline, and mixes them with RMS-based scaling to produce transcription-ready speech segments and archival WAV recordings.

The architecture of the audio pipeline for simultaneous microphone and system audio capture in Meetily enables real-time meeting transcription by treating input and output devices as distinct capture sources that are synchronized, professionally mixed, and filtered before reaching the local Whisper engine. Written in Rust within the Tauri backend at frontend/src-tauri/src/audio/, this pipeline handles heterogeneous sample rates, applies broadcast-quality audio processing, and maintains lock-step synchronization between user voice and system sounds.

Core Audio Capture Layer

The pipeline begins with dual independent capture streams that pull raw PCM data from separate hardware abstraction layers.

Microphone Capture is handled by the AudioCapture struct defined in pipeline.rs:194-215. This implementation applies a preprocessing chain including high-pass filtering, optional RNNoise suppression, and EBU-R128 loudness normalization specifically to voice input. The capture runs at the device's native sample rate but feeds into a persistent resampler when the hardware does not match the pipeline's 48 kHz target.

System Audio Capture on macOS uses the SystemAudioStreamManager located in system_audio_stream.rs:11-76. This component opens a CoreAudio loopback (or falls back to ScreenCaptureKit) to capture desktop audio in 1024-sample chunks. Unlike microphone input, system audio bypasses enhancement filters because the source is already considered clean digital audio. Both streams ultimately push data into the same AudioCapture::process_audio_data method found in pipeline.rs:99-166.

Ring Buffer Synchronization

Synchronization relies on the AudioMixerRingBuffer defined in pipeline.rs:16-45, which implements a lock-step buffering strategy to align asymmetric input streams.

The ring buffer maintains separate FIFO queues for microphone and system samples. It only permits mixing when both queues contain at least one window of data (approximately 50 ms), enforced by the can_mix predicate. When mixing is authorized, extract_window pulls synchronized slices from each queue, zero-padding the shorter stream if one source lags behind. This design prevents phase misalignment and ensures that user speech and system sounds remain temporally correlated in the final output.

Professional Audio Mixing

Once synchronized, samples pass through the ProfessionalAudioMixer in pipeline.rs:45-90, which performs RMS-based mixing without aggressive ducking.

The mixer applies a simple but effective scaling algorithm: system audio is attenuated to approximately 70% of its original level while microphone audio remains at full scale. The signals are summed and subjected to soft clipping to prevent inter-sample peaks from exceeding the ±1.0 floating-point boundary. This approach preserves the dynamic range of both sources while preventing the "pumping" artifacts common in automatic gain control schemes.

Voice Activity Detection Integration

The mixed audio stream feeds into ContinuousVadProcessor within the AudioPipeline::run loop at pipeline.rs:78-88. This VAD-driven architecture segments the continuous audio stream into speech-positive chunks.

The processor identifies segments containing at least 800 samples at 16 kHz (approximately 50 ms) of contiguous speech. Only these segments are wrapped in AudioChunk structures and forwarded via transcription_sender to the Whisper engine. Non-speech audio—including system sounds during silent periods—is filtered from the transcription path while still being preserved in the archival recording.

Sample Rate Normalization

Devices operating at sample rates other than 48 kHz undergo persistent resampling inside AudioCapture::process_audio_data.

Rather than processing audio chunk-by-chunk (which caused RMS energy blow-up in earlier implementations), the pipeline uses a persistent Rubato SincFixedIn resampler. Input samples accumulate in resampler_input_buffer until a fixed 512-sample chunk is available. The resampler processes these blocks while preserving energy continuity across chunk boundaries, ensuring that resampled microphone audio maintains consistent loudness characteristics required for accurate VAD triggering.

Pipeline Lifecycle and Shutdown

Lifecycle management is orchestrated by AudioPipelineManager, which handles channel creation, device metadata configuration, and async task spawning.

The manager supports a "flush-and-stop" shutdown sequence that injects special flush chunks with chunk_id = u64::MAX into the buffer. This guarantees that all residual audio is processed through the VAD and recording paths before the pipeline terminates, preventing truncation of trailing speech segments.

Implementation Example

Creating and starting the pipeline from a Tauri command context:

let mic_device = Arc::new(AudioDevice::new("Built‑in Mic".to_string(), DeviceType::Input));
let sys_device = Arc::new(AudioDevice::new("BlackHole 2ch".to_string(), DeviceType::Output));

let mut manager = AudioPipelineManager::new();
let (trans_sender, trans_receiver) = mpsc::unbounded_channel(); // Whisper side
let recording_sender = Some(mpsc::unbounded_sender()?); // WAV recorder

manager.start(
    state.clone(),
    trans_sender,
    /*target_chunk_duration_ms=*/0,
    /*sample_rate=*/48_000,
    recording_sender,
    mic_device.name.clone(),
    DeviceType::Input.kind(),
    sys_device.name.clone(),
    DeviceType::Output.kind(),
)?;

The mixing loop inside AudioPipeline::run demonstrates the synchronized extraction:

while self.ring_buffer.can_mix() {
    let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
    let mixed = self.mixer.mix_window(&mic_win, &sys_win);
    // VAD → Whisper
    self.vad_processor.process_audio(&mixed)?;
    // Record mixed WAV
    if let Some(rec) = &self.recording_sender_for_mixed {
        let _ = rec.send(AudioChunk { data: mixed, sample_rate: 48_000, chunk_id: self.chunk_id });
    }
}

Summary

• Dual-stream capture isolates microphone and system audio into separate AudioCapture instances, allowing device-specific preprocessing.
• Lock-step ring buffering via AudioMixerRingBuffer synchronizes 50 ms windows across heterogeneous sources, preventing temporal drift.
• RMS-based mixing scales system audio to 70% and applies soft clipping to prevent distortion while preserving dynamic range.
• VAD gating sends only speech segments (≥50 ms) to Whisper, reducing computational load while maintaining full mixed audio in the WAV archive.
• Persistent resampling using Rubato's SincFixedIn maintains energy continuity when converting device-native rates to the 48 kHz pipeline standard.

Frequently Asked Questions

How does Meetily prevent audio drift between microphone and system sources?

Meetily prevents drift through the AudioMixerRingBuffer in pipeline.rs:16-45, which implements a blocking synchronization mechanism. The pipeline only mixes audio when both the microphone and system queues contain sufficient samples for a complete 50 ms window. If one source lags, the buffer holds the leading stream until synchronization is restored, ensuring both channels remain sample-aligned throughout the recording session.

What mixing algorithm does Meetily use to combine audio streams?

The pipeline uses the ProfessionalAudioMixer located in pipeline.rs:45-90, which performs RMS-based amplitude scaling rather than complex dynamic range compression. System audio is attenuated to approximately 70% of its original level, summed with full-scale microphone audio, and limited through soft clipping. This technique avoids aggressive ducking artifacts while preventing inter-sample clipping that would occur from simple addition.

Why does the pipeline resample all audio to 48 kHz?

The 48 kHz target sample rate serves as the lowest common denominator that accommodates both high-quality microphone capture and system audio loopback while providing sufficient headroom for the VAD algorithm. The persistent resampler implementation in process_audio_data (lines 99-166) ensures that devices operating at 44.1 kHz or other native rates are converted without the energy discontinuities that plagued earlier per-chunk resampling approaches.

How does the VAD filter affect the final meeting recording?

The ContinuousVadProcessor only filters the transcription path, sending speech segments to the Whisper engine while discarding silence for efficiency. However, the recording_sender_for_mixed channel receives the complete unfiltered mix including all system sounds and non-speech audio. This dual-path architecture ensures that the archived WAV file contains the full meeting context while the transcription engine processes only relevant speech segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}