How Meetily's Audio Pipeline Handles Simultaneous Microphone and System Audio Capture with Professional Mixing

Meetily's audio pipeline captures microphone input and system audio as separate streams, synchronizes them through a dual-ring buffer, and mixes them using soft-clipping algorithms to prevent distortion while maintaining broadcast-standard loudness levels.

Meetily is an open-source meeting transcription application built with Tauri and Rust that records both user microphone input and system audio simultaneously. The core audio processing logic resides in frontend/src-tauri/src/audio/pipeline.rs, where a multi-stage pipeline ensures professional-grade audio mixing before sending data to Whisper for transcription or saving mixed WAV files to disk.

Device-Specific Capture and Preprocessing

The pipeline begins with independent AudioCapture instances for each audio source. All streams are converted to mono and resampled to the pipeline's native 48 kHz sample rate before entering the mixing stage.

Microphone Enhancement Chain

Microphone input undergoes a series of professional audio processors implemented in AudioCapture::process_audio_data. The signal passes through a high-pass filter to remove rumble, then RNNoise for real-time noise suppression, and finally EBU-R128 loudness normalization to achieve broadcast-standard -23 LUFS. This ensures the microphone signal maintains consistent perceived loudness regardless of hardware variations.

System Audio Handling

System audio (capturing application sounds, notifications, and music) is processed with minimal modification to preserve fidelity. The stream is converted to mono and resampled to 48 kHz when necessary, then forwarded as AudioChunk structs to the shared pipeline via Tokio unbounded channels.

Ring-Buffer Synchronization for Latency Management

The AudioPipeline receives raw chunks from both devices through an unbounded Tokio channel. To handle the inherent latency differences between microphone and system audio capture—particularly on macOS Core Audio—the pipeline employs an AudioMixerRingBuffer that stores samples in two independent VecDeque buffers.

The AudioMixerRingBuffer Implementation

The ring buffer accumulates samples until both streams contain sufficient data for a fixed-size mixing window of approximately 50 ms. The buffer logic, defined in AudioMixerRingBuffer::add_samples, can_mix, and extract_window, implements several safeguards:

• Maximum buffer size is capped at 400 ms (8× the mixing window) to absorb bursty system-audio delivery without runaway memory growth
• Silence padding replaces missing data with zeros rather than repeating samples, eliminating audible artifacts when one stream temporarily drops
• Synchronization ensures both streams align temporally before mixing begins

Professional Mixing with Soft-Clipping

Once the ring buffer reports a full window is ready, the ProfessionalAudioMixer combines the microphone and system windows. The mixing routine implemented in ProfessionalAudioMixer::mix_window applies professional broadcast techniques:

fn mix_window(&mut self, mic_window: &[f32], sys_window: &[f32]) -> Vec<f32> {
    let max_len = mic_window.len().max(sys_window.len());
    let mut mixed = Vec::with_capacity(max_len);
    for i in 0..max_len {
        let mic = mic_window.get(i).copied().unwrap_or(0.0);
        let sys = sys_window.get(i).copied().unwrap_or(0.0);
        // System is slightly reduced; mic stays at full level
        let sum = mic + sys * 0.7; // 70% system audio level
        // Soft clipping – proportionally scale if > 1.0
        let mixed_sample = if sum.abs() > 1.0 { sum / sum.abs() } else { sum };
        mixed.push(mixed_sample);
    }
    mixed
}

The mixer scales system audio to 70% of its original volume to prevent it from overpowering voice content. It then applies soft-clipping: when the sum of samples exceeds the ±1.0 unit range, the result is proportionally scaled back rather than hard-truncated. This prevents the "radio-break" distortion associated with digital clipping while preserving audio fidelity during loud passages.

Downstream Routing to VAD and Recording

After mixing, the pipeline applies no additional gain—the microphone signal already carries broadcast-level normalization and the system audio has been attenuated. The mixed chunk is then routed through two parallel paths:

1. Voice Activity Detection (VAD): The vad.rs processor extracts speech-only segments for Whisper transcription, filtering out silence and non-speech system sounds
2. Recording Sender: The full mixed audio stream writes to disk as a WAV file containing both microphone and system audio

This separation prevents double-recording and echo while ensuring the transcription engine receives clean speech data.

Handling Sample Rate Mismatches and Platform Differences

When input devices report different sample rates, AudioCapture initializes a persistent Rubato resampler that buffers variable-size chunks until accumulating 512 samples for processing. This preserves RMS energy across chunks and maintains synchronization even when Windows WASAPI, macOS CoreAudio, or Linux ALSA devices report different hardware rates.

The pipeline initialization from the Tauri command layer demonstrates how these components connect:

// In frontend/src-tauri/src/audio/pipeline.rs context
#[tauri::command]
async fn start_recording(
    app: tauri::AppHandle,
    mic_device_name: Option<String>,
    system_device_name: Option<String>,
) -> Result<(), String> {
    let state = RecordingState::new();
    let (trans_tx, trans_rx) = mpsc::unbounded_channel();
    let recording_sender = Some(mpsc::unbounded_channel().0);

    let mut manager = AudioPipelineManager::new();
    manager.start(
        Arc::new(state),
        trans_tx,
        20, // target_chunk_duration_ms
        48_000, // sample_rate
        recording_sender,
        mic_device_name.unwrap_or_default(),
        detect_mic_kind(),
        system_device_name.unwrap_or_default(),
        detect_system_kind(),
    )?;
    Ok(())
}

Summary

Meetily's audio pipeline implements a three-stage architecture for professional dual-source recording:

• Capture stage applies broadcast-standard normalization and noise suppression to microphone input while preserving system audio fidelity
• Synchronization stage uses a dual-ring buffer with 50 ms windows and 400 ms maximum latency absorption to align asynchronous streams
• Mixing stage applies soft-clipping and proportional scaling to combine sources at 70% system volume without distortion

The pipeline maintains 48 kHz throughout, handles sample-rate mismatches via Rubato resampling, and separates transcription streams (speech-only) from recording streams (full mix).

Frequently Asked Questions

How does Meetily prevent audio clipping when mixing microphone and system audio?

The ProfessionalAudioMixer::mix_window function implements soft-clipping by checking if the combined sample exceeds ±1.0. When overflow occurs, the mixer scales the result proportionally (sum / sum.abs()) rather than truncating it, preserving waveform fidelity and preventing the harsh distortion associated with digital clipping.

What sample rate does Meetily use for audio processing?

The pipeline standardizes all audio to 48 kHz mono format. Input devices running at different sample rates (44.1 kHz, 96 kHz, etc.) are transparently resampled using the Rubato library, which processes audio in 512-sample blocks to maintain energy consistency across variable buffer sizes.

How does the pipeline handle differences in microphone and system audio latency?

The AudioMixerRingBuffer accommodates latency variance by buffering up to 400 ms of audio (8× the 50 ms mixing window). When one stream arrives later than the other, the extractor pads the delayed window with silence (zeros) rather than repeating samples, ensuring synchronization without introducing audible artifacts or phase issues.

Why does Meetily reduce system audio volume by 30% during mixing?

System audio is attenuated to 70% of its original level to establish proper hierarchy in the final mix. This prevents application sounds, notifications, and music from masking the primary voice content while maintaining presence. Since the microphone stream is normalized to -23 LUFS broadcast standard, the 30% reduction creates a balanced blend where voice remains intelligible over background system audio.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}