How Meetily's Audio Pipeline Handles Microphone and System Audio Mixing

Meetily captures microphone input and system audio simultaneously through separate device streams, synchronizes them via a ring buffer, and blends them into a single professional-grade mix using soft-clipping algorithms before routing to Whisper transcription and WAV file storage.

Meetily is an open-source meeting assistant built with Tauri and Rust that requires pristine audio capture for accurate AI transcription. The application's audio pipeline implements a sophisticated multi-stage subsystem that processes microphone and system audio as independent streams, applies broadcast-quality enhancement and mixing, and delivers synchronized output for both real-time speech recognition and archival recording. This implementation resides primarily in frontend/src-tauri/src/audio/pipeline.rs within the Zackriya-Solutions/meetily repository.

Stage 1: Device-Specific Capture and Audio Enhancement

Each audio source operates through its own AudioCapture instance that handles device-specific ingestion and preprocessing. The microphone stream undergoes a professional enhancement chain before entering the mixing pipeline:

• High-pass filtering to remove low-frequency rumble
• RNNoise noise suppression for neural speech enhancement
• EBU-R128 loudness normalization targeting broadcast-standard -23 LUFS

The system audio stream (capturing application sounds, notifications, and media) bypasses enhancement and enters the pipeline at raw volume. Both streams are converted to mono format and resampled to the pipeline's native 48 kHz sample rate using Rubato resamplers when device-native rates differ. The processed data is packaged as AudioChunk structs and forwarded to the shared pipeline state via AudioCapture::process_audio_data.

// Simplified excerpt from AudioCapture implementation
impl AudioCapture {
    fn process_audio_data(&mut self, raw_samples: &[f32]) {
        // Apply enhancement chain for microphone only
        let processed = if self.device_kind == DeviceKind::Microphone {
            self.apply_highpass(raw_samples);
            self.apply_rnnoise(processed);
            self.normalize_loudness(processed)
        } else {
            raw_samples.to_vec()
        };
        
        // Convert to mono and resample if necessary
        let mono = stereo_to_mono(&processed);
        let resampled = self.resampler.process(mono);
        
        // Forward to pipeline
        self.state.send_audio_chunk(AudioChunk {
            samples: resampled,
            timestamp: Instant::now(),
            device_id: self.device_id,
        });
    }
}

Stage 2: Ring-Buffer Synchronization for Dual-Stream Alignment

The AudioPipeline receives chunks from both devices through an unbounded Tokio channel and manages temporal alignment via an AudioMixerRingBuffer. This component stores microphone and system samples in two independent VecDeque buffers until sufficient data accumulates for a fixed-size mixing window (default 50 ms).

The ring buffer implements critical synchronization logic:

• Latency absorption: Maintains a max_buffer_size of 400 ms (8× the mixing window) to handle bursty delivery patterns common in macOS Core Audio drivers
• Silence padding: When one stream temporarily stalls, extract_window pads the missing window with zeros rather than repeating samples, eliminating click artifacts
• Memory protection: Caps total buffer size to prevent unbounded growth during extended recording sessions

Once both buffers contain sufficient samples, can_mix returns true and extract_window yields synchronized windows ready for combination.

// Ring buffer extraction logic
if let Some((mic_win, sys_win)) = self.ring_buffer.extract_window() {
    // Both windows guaranteed equal length and temporal alignment
    let mixed = self.mixer.mix_window(&mic_win, &sys_win);
    // Forward to downstream processors...
}

Stage 3: Professional Mixing with Soft-Clipping Algorithm

The ProfessionalAudioMixer combines synchronized windows through mix_window, implementing broadcast-grade mixing techniques to prevent distortion. The mixer applies proportional gain staging: system audio receives a 70% volume reduction (attenuating to 30% of original level) while microphone audio maintains full normalized level from the EBU-R128 stage.

To prevent the digital clipping that occurs when combined samples exceed the ±1.0 floating-point range, Meetily implements soft-clipping protection:

impl ProfessionalAudioMixer {
    fn mix_window(&mut self, mic_window: &[f32], sys_window: &[f32]) -> Vec<f32> {
        let max_len = mic_window.len().max(sys_window.len());
        let mut mixed = Vec::with_capacity(max_len);
        
        for i in 0..max_len {
            let mic = mic_window.get(i).copied().unwrap_or(0.0);
            let sys = sys_window.get(i).copied().unwrap_or(0.0) * 0.3; // 70% reduction
            
            let sum = mic + sys;
            
            // Soft clipping: proportional scaling if exceeding unit range
            let mixed_sample = if sum.abs() > 1.0 {
                sum / sum.abs() // Scales back to ±1.0 proportionally
            } else {
                sum
            };
            
            mixed.push(mixed_sample);
        }
        mixed
    }
}

This approach preserves audio fidelity by smoothly compressing transients rather than introducing the harsh "radio-break" distortion of hard clipping.

Handling Sample Rate Mismatches and Platform Latency

When input devices operate at different sample rates (e.g., 44.1 kHz consumer microphone vs. 48 kHz system audio), AudioCapture instantiates a persistent Rubato resampler that buffers variable-size input chunks until accumulating a 512-sample block for processing. This block-wise approach preserves RMS energy across chunk boundaries while maintaining the pipeline's strict 48 kHz internal standard.

The ring buffer's 400 ms capacity specifically addresses platform-specific jitter characteristics, particularly the bursty callback patterns observed in macOS Core Audio implementations where system audio may arrive in irregular bursts rather than steady intervals.

Downstream Routing: Transcription vs. Recording

After mixing, the pipeline performs no additional gain staging—the microphone signal already maintains broadcast-level loudness and the system audio has been attenuated. The mixed stream then splits for dual consumption:

1. VAD Processing: The mixed audio passes through voice-activity detection (vad.rs), which extracts speech-only segments and forwards them to Whisper for transcription
2. Recording Archive: The complete mixed stream (both mic and system audio) routes to a WAV file writer, ensuring saved meetings contain the full conversational context including background audio and participant system sounds

This separation prevents echo and double-recording while ensuring the transcription engine receives clean speech data without non-verbal system sounds.

// Orchestration in AudioPipeline::run
loop {
    if let Some((mic, sys)) = self.ring_buffer.extract_window() {
        let mixed = self.mixer.mix_window(&mic, &sys);
        
        // Parallel downstream routing
        self.vad.process(&mixed);           // Speech extraction for Whisper
        self.recording_sender.send(mixed);  // Full mix to WAV file
    }
}

Summary

• Dual-stream capture: Separate AudioCapture instances handle microphone and system audio with device-specific preprocessing chains
• Broadcast-quality enhancement: Microphone audio receives RNNoise suppression and EBU-R128 normalization to -23 LUFS before mixing
• Temporal alignment: AudioMixerRingBuffer synchronizes streams using 50 ms windows with 400 ms jitter absorption and zero-padding for missing data
• Professional mixing: ProfessionalAudioMixer applies 70% system audio attenuation and soft-clipping to prevent distortion when summing streams
• Flexible resampling: Rubato-based resamplers handle arbitrary input sample rates by buffering to 512-sample blocks while preserving energy
• Dual routing: Mixed audio feeds both VAD/Whisper transcription (speech only) and WAV recording (full mix) without additional gain stages

Frequently Asked Questions

How does Meetily prevent hard clipping when mixing microphone and system audio?

The ProfessionalAudioMixer::mix_window implementation detects when the sum of microphone and scaled system samples exceeds the ±1.0 floating-point range. Instead of truncating values (hard clipping), it proportionally scales the entire sample back to the unit range using sum / sum.abs(), preserving the waveform's relative dynamics while preventing digital distortion.

What synchronization mechanism ensures microphone and system audio align correctly?

Meetily uses an AudioMixerRingBuffer that stores each stream in independent VecDeque buffers. The extract_window method only returns paired windows when both buffers contain sufficient samples for the 50 ms mixing window. If one stream lags, the buffer pads with silence (zeros) rather than repeating samples, maintaining temporal alignment without introducing phase artifacts.

How does Meetily handle different sample rates between input devices?

When AudioCapture detects a sample rate mismatch with the pipeline's native 48 kHz, it creates a persistent Rubato resampler for that device. This resampler accumulates variable-size input chunks until reaching a 512-sample processing block, converts the rate while preserving RMS energy, and forwards the resampled data to the mixing pipeline.

Why does Meetily attenuate system audio before mixing?

The mixer reduces system audio by approximately 70% (to 30% of original level) to maintain proper vocal prominence in the final mix. Since microphone audio undergoes EBU-R128 loudness normalization to broadcast standards (-23 LUFS) while system audio enters raw, this attenuation prevents background application sounds and notifications from overpowering the primary speaker while still remaining audible in the recording.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}