How Meetily Handles Concurrent Microphone and System Audio Capture with Intelligent Mixing

Meetily captures microphone and system audio simultaneously through independent device streams, synchronizes them in a ring-buffer, and blends them using a soft-clipping algorithm to produce a unified 48 kHz mono mix for recording and Whisper transcription.

Meetily, an open-source meeting assistant by Zackriya-Solutions, implements a sophisticated Rust-based audio pipeline that manages concurrent microphone and system audio capture with professional-grade mixing. The architecture ensures broadcast-quality voice recording alongside system sounds (notifications, music, or application audio) while delivering clean speech segments to the transcription engine. This article examines the three-stage pipeline defined in frontend/src-tauri/src/audio/pipeline.rs, covering device capture, synchronization, and intelligent mixing.

Architecture Overview of the Audio Pipeline

Meetily’s audio subsystem divides concurrent capture into three distinct stages: device-specific preprocessing, temporal synchronization via ring buffer, and professional mixing with downstream routing.

Device Capture and Preprocessing

Each audio source is managed by an AudioCapture instance that handles platform-specific enumeration (WASAPI on Windows, CoreAudio on macOS, ALSA on Linux). The microphone stream undergoes a mandatory enhancement chain implemented in AudioCapture::process_audio_data: a high-pass filter removes low-frequency rumble, RNNoise suppresses background noise, and EBU-R128 loudness normalization targets -23 LUFS broadcast level. System audio bypasses enhancement but both streams are converted to mono and resampled to the pipeline’s native 48 kHz using a persistent Rubato resampler that buffers chunks until 512-sample blocks are available.

// In AudioCapture::process_audio_data (pipeline.rs)
fn process_audio_data(&mut self, data: &[f32], state: &RecordingState) {
    let processed = self.enhancement_chain.process(data);
    let resampled = self.resampler.process(&processed); // Rubato resampler
    state.send_audio_chunk(AudioChunk {
        data: resampled,
        is_mic: self.is_mic,
    });
}

Ring-Buffer Synchronization

The AudioPipeline receives raw chunks from both devices through an unbounded Tokio channel and stores them in an AudioMixerRingBuffer. This structure maintains two independent VecDeque buffers—one for microphone samples and one for system audio—waiting until both queues contain sufficient data for a fixed 50 ms mixing window. The buffer implements defensive caps: it pads missing streams with silence (zeros, not sample repetition) to prevent artifacts, and limits total buffer size to 400 ms (eight mixing windows) to absorb jitter from bursty system-audio delivery on macOS Core Audio without unbounded memory growth.

When the buffer reports readiness via can_mix(), the extract_window() method returns synchronized slices of exactly 50 ms from both streams, ensuring temporal alignment before mixing occurs.

Professional Mixing and Routing

Once synchronized windows are available, the ProfessionalAudioMixer::mix_window function combines the streams. The implementation applies a 70% attenuation to system audio (reducing it to approximately 30% of original volume) while keeping the microphone at full normalized level. To prevent the "radio-break" distortion of hard clipping, the mixer employs soft-scaling: if the sum of mic and system samples exceeds the ±1.0 floating-point range, the result is proportionally scaled back to fit within the unit range.

// Excerpt from ProfessionalAudioMixer::mix_window (pipeline.rs)
fn mix_window(&mut self, mic_window: &[f32], sys_window: &[f32]) -> Vec<f32> {
    let mut mixed = Vec::with_capacity(mic_window.len());
    for i in 0..mic_window.len() {
        let mic = mic_window[i];
        let sys = sys_window[i] * 0.3; // 70% reduction applied
        let sum = mic + sys;
        // Soft clipping: scale if outside ±1.0
        let sample = if sum.abs() > 1.0 { sum / sum.abs() } else { sum };
        mixed.push(sample);
    }
    mixed
}

After mixing, the pipeline performs no additional gain staging—the microphone signal already carries broadcast-level loudness. The mixed chunk is then dispatched along two paths: it flows to the Voice Activity Detection (VAD) processor in vad.rs for speech segmentation and Whisper transcription, and simultaneously to a WAV recording sender that writes the final mixed file to disk.

Handling Sample Rate Mismatches and Latency

Audio devices rarely share identical sample rates or clock domains. Meetily addresses this through stream-specific resampling and buffering:

• Sample-rate conversion: The AudioCapture struct detects mismatches between device native rates (e.g., 44.1 kHz) and the pipeline’s 48 kHz requirement. It instantiates a Rubato resampler per device that preserves RMS energy across variable-size chunks, ensuring loudness consistency even when operating systems deliver audio in non-uniform buffers.

• Jitter absorption: The AudioMixerRingBuffer caps its maximum size at 400 ms, specifically configured to handle sporadic delivery spikes from macOS Core Audio. This prevents buffer underruns when system audio arrives in bursts while maintaining low latency (50 ms windows) for real-time transcription.

Preventing Clipping and Audio Artifacts

Meetily implements three safeguards to maintain fidelity during intelligent audio mixing:

1. Soft-clipping protection: Unlike hard clipping which truncates values exceeding ±1.0, the mix_window function proportionally scales overflow samples, preserving the waveform shape and avoiding harmonic distortion.

2. Silence padding: When one stream temporarily lacks data (e.g., microphone mute), extract_window fills the gap with zeros rather than holding the last sample, eliminating audible clicks or repeats in the final recording.

3. Separation of concerns: Raw microphone and system streams are never written directly to disk. Only the professionally mixed output reaches the WAV recorder, while the transcription path receives VAD-filtered speech segments. This prevents double-recording and echo effects that would degrade Whisper accuracy.

Implementation: Starting the Pipeline

The pipeline lifecycle is orchestrated from recording_manager.rs via Tauri commands. The following simplified excerpt demonstrates how the system initializes dual capture:

// In recording_manager.rs – Tauri command entry point
#[tauri::command]
async fn start_recording(
    app: tauri::AppHandle,
    mic_device_name: Option<String>,
    system_device_name: Option<String>,
) -> Result<(), String> {
    let state = Arc::new(RecordingState::new());
    let (trans_tx, _trans_rx) = mpsc::unbounded_channel(); // Whisper channel
    let (rec_tx, _rec_rx) = mpsc::unbounded_channel();   // WAV writer channel

    let mut manager = AudioPipelineManager::new();
    manager.start(
        state,
        trans_tx,
        20,                      // Target chunk duration (ms)
        48_000,                  // Sample rate
        Some(rec_tx),            // Recording sender
        mic_device_name.unwrap_or_default(),
        detect_mic_kind(),
        system_device_name.unwrap_or_default(),
        detect_system_kind(),
    )?;
    Ok(())
}

Once running, the AudioPipeline::run loop continuously polls the ring buffer, invokes the mixer, and routes output to both VAD and recording channels.

Summary

• Three-stage architecture: Meetily separates capture (AudioCapture), synchronization (AudioMixerRingBuffer), and mixing (ProfessionalAudioMixer) into distinct components within frontend/src-tauri/src/audio/pipeline.rs.

• Intelligent synchronization: A 50 ms ring-buffer window with 400 ms max capacity aligns microphone and system audio temporally while handling OS-specific jitter.

• Professional mixing: System audio is attenuated by 70% and combined with normalized microphone input using soft-clipping to prevent distortion.

• Dual routing: The mixed stream feeds both the WAV recorder (full audio) and VAD processor (speech-only segments for Whisper), ensuring high-quality archives alongside accurate transcription.

Frequently Asked Questions

What sample rate does Meetily use for internal audio processing?

Meetily standardizes all audio to 48 kHz mono before mixing. The AudioCapture struct automatically resamples device-native streams (commonly 44.1 kHz or 48 kHz) using a Rubato resampler that processes audio in 512-sample blocks to maintain phase coherence and loudness consistency.

How does Meetily prevent clipping when mixing loud microphone and system audio?

The ProfessionalAudioMixer::mix_window function implements soft-clipping protection. When the sum of microphone and attenuated system audio exceeds the ±1.0 floating-point range, the mixer proportionally scales the sample back to the unit range rather than truncating it, preventing the harsh distortion associated with digital hard-clipping.

Can Meetily handle devices with different sample rates simultaneously?

Yes. Each AudioCapture instance manages its own resampler state. When the pipeline detects a sample-rate mismatch between the device and the 48 kHz pipeline requirement, it creates a persistent Rubato resampler for that specific device that buffers variable-size chunks until full 512-sample blocks can be processed, ensuring synchronized delivery to the ring buffer.

Why does Meetily attenuate system audio by 70% during mixing?

The 70% reduction (rendering system audio at approximately 30% of original volume) ensures voice intelligibility remains primary in the final mix. Since the microphone stream undergoes EBU-R128 loudness normalization targeting -23 LUFS while system audio receives no such normalization, this attenuation prevents application sounds, notifications, or music from overpowering the speaker’s voice in the recorded WAV file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}