How Meetily Handles Simultaneous Microphone and System Audio Capture with VAD Filtering

Meetily captures microphone and system audio in parallel, synchronizes them via a ring buffer, mixes both streams into a single track, and applies continuous Voice Activity Detection (VAD) to feed only speech segments to Whisper for transcription.

Meetily is an open-source meeting assistant built by Zackriya-Solutions that records both the user's microphone and system audio simultaneously. Understanding how Meetily handles simultaneous microphone and system audio capture with VAD filtering reveals a sophisticated Rust-based audio pipeline that prioritizes synchronization, clean mixing, and efficient transcription by stripping silence before it reaches the Whisper engine.

Independent Audio Capture Architecture

Meetily spawns two separate AudioCapture instances to handle input from different audio endpoints.

• DeviceType::Microphone: Captures the user's physical microphone via cpal streams in frontend/src-tauri/src/audio/capture/microphone.rs
• DeviceType::System: Captures system playback audio (e.g., browser audio, media players) via platform-specific APIs in frontend/src-tauri/src/audio/capture/system.rs

Each capture runs independently and pushes raw PCM chunks to a shared RecordingState. This design ensures that latency spikes from one device do not block the other, providing robust real-time performance.

Resampling and Pre-Processing Pipeline

When audio arrives at non-standard sample rates, Meetily normalizes everything to a 48 kHz target before mixing.

The resampling logic resides in AudioCapture::process_audio_data (lines 99-124 of frontend/src-tauri/src/audio/pipeline.rs). It uses a persistent Rubato resampler to avoid allocation overhead during streaming.

Microphone audio receives additional treatment:

• High-pass filter to eliminate low-frequency rumble
• RNNoise noise suppression (optional)
• EBU-R128 loudness normalization to ensure consistent levels

System audio bypasses these filters because it is already considered clean, reducing unnecessary CPU load.

Ring Buffer Synchronization

The AudioMixerRingBuffer (lines 16-45 and 49-85 of frontend/src-tauri/src/audio/pipeline.rs) solves the synchronization challenge between two streams that may deliver samples at slightly different rates.

It maintains two VecDeque<f32> buffers—one for microphone samples and one for system samples. The buffer accumulates data until a fixed-size window of approximately 50 ms is available for both streams. This time-alignment step prevents drift and ensures that a sound played on the system at time T is mixed with the user's reaction captured at the same moment.

Professional Audio Mixing

Once the ring buffer has a complete window, ProfessionalAudioMixer::mix_window (lines 54-88 of frontend/src-tauri/src/audio/pipeline.rs) combines the two streams.

The mixing algorithm applies:

• System audio at ~70% volume to prevent it from overpowering the microphone
• Soft-clipping limiter to keep the final waveform within ±1.0 without distortion
• No aggressive ducking to preserve speech clarity—the microphone stays at full level

This approach produces a natural-sounding recording where both sides of the conversation (user and system) are intelligible without jarring volume shifts.

Continuous VAD Filtering

The mixed audio flows into a ContinuousVadProcessor defined in frontend/src-tauri/src/audio/vad.rs. The VAD operates internally at 16 kHz (downsampled from the 48 kHz mixed stream) to balance accuracy and performance.

Unlike fixed-interval buffering, the VAD emits speech segments only when voice activity is detected. This eliminates silent frames from the transcription queue, significantly reducing data sent to Whisper and lowering latency.

The VAD-to-Whisper dispatch logic (lines 34-58 of frontend/src-tauri/src/audio/pipeline.rs) filters out segments shorter than 800 samples (~50 ms at 16 kHz). This threshold prevents useless Whisper invocations on brief noise spikes or breath sounds.

Transcription Dispatch and Recording

Each valid VAD segment is wrapped in an AudioChunk struct with a 16 kHz sample rate and dispatched to the Whisper engine via transcription_sender. The segment includes precise timestamps to maintain synchronization with the meeting timeline.

Simultaneously, the raw mixed audio (without VAD filtering) can be forwarded to recording_sender_for_mixed to produce a complete WAV file containing both microphone and system audio. This dual-path approach ensures that the transcription is efficient while the archival recording remains complete.

Code Implementation Examples

Frontend Recording Invocation

// Start a meeting from the TypeScript frontend
await invoke('start_recording', {
  mic_device_name: "Built-in Microphone",
  system_device_name: "BlackHole 2ch",   // macOS virtual audio device
  meeting_name: "Team Sync"
});

Pipeline Manager Initialization

// Rust backend: create and start the dual-capture pipeline
let mut manager = AudioPipelineManager::new();
manager.start(
    recording_state.clone(),
    transcription_sender,
    /* target_chunk_duration_ms */ 0,
    /* sample_rate */ 48000,
    Some(recording_sender),               // receive mixed audio for .wav
    mic_device_name,
    mic_device_kind,
    system_device_name,
    system_device_kind,
)?;

VAD-Driven Processing Loop

// Inside AudioPipeline::run – continuous VAD processing
while let Some(chunk) = self.receiver.recv().await {
    self.ring_buffer.add_samples(chunk.device_type.clone(), chunk.data);
    while self.ring_buffer.can_mix() {
        let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
        let mixed = self.mixer.mix_window(&mic_win, &sys_win);
        
        // VAD splits speech and sends to Whisper
        for seg in self.vad_processor.process_audio(&mixed)? {
            if seg.samples.len() >= 800 {
                self.transcription_sender.send(AudioChunk {
                    data: seg.samples,
                    sample_rate: 16000,
                    timestamp: seg.start_timestamp_ms / 1000.0,
                    chunk_id: self.chunk_id_counter,
                    device_type: DeviceType::Microphone,
                })?;
            }
        }
    }
}

Key Files in the Audio Pipeline

File	Purpose
frontend/src-tauri/src/audio/pipeline.rs	Central orchestration: ring buffer, mixing, VAD handling, and dispatch logic
frontend/src-tauri/src/audio/capture/microphone.rs	Microphone capture with optional preprocessing (RNNoise, EBU-R128)
frontend/src-tauri/src/audio/capture/system.rs	System audio capture via platform-specific APIs
frontend/src-tauri/src/audio/vad.rs	ContinuousVadProcessor implementation for real-time speech segmentation
frontend/src-tauri/src/audio/recording_manager.rs	Orchestrates AudioPipelineManager lifecycle and mixed-audio recording
frontend/src-tauri/src/audio/devices/discovery.rs	Device enumeration for microphone and system audio selection

Summary

• Dual capture: AudioCapture instances for DeviceType::Microphone and DeviceType::System run in parallel without blocking each other
• Sample rate normalization: Rubato resampler converts all input to 48 kHz; microphone audio gets high-pass filtering, optional RNNoise, and EBU-R128 normalization
• Synchronization: AudioMixerRingBuffer aligns both streams using ~50 ms fixed windows before mixing
• Clean mixing: ProfessionalAudioMixer::mix_window combines streams at 70% system volume with soft-clipping to prevent distortion
• Efficient transcription: ContinuousVadProcessor strips silence, emitting only speech segments ≥800 samples (16 kHz) to Whisper
• Complete recording: Mixed audio flows to both the VAD transcription path and an optional WAV recording path

Frequently Asked Questions

Why does Meetily use a ring buffer instead of simple timestamp matching?

The AudioMixerRingBuffer ensures that both microphone and system audio chunks are aligned before mixing, even when the two streams deliver data at different rates or experience jitter. By waiting for a complete ~50 ms window from both sources, the system prevents temporal drift and guarantees that system audio and user speech remain synchronized in the final output.

What sample rates does the VAD use internally?

The VAD processes mixed audio at 16 kHz internally, downsampled from the pipeline's 48 kHz target. This 16 kHz rate provides sufficient resolution for speech detection while keeping computational overhead low for real-time processing.

How does Meetily prevent system audio from drowning out the microphone?

The ProfessionalAudioMixer::mix_window function scales system audio to approximately 70% volume while keeping the microphone at full level. It also applies a soft-clipping limiter to ensure the combined waveform never exceeds ±1.0, preserving speech clarity without aggressive ducking algorithms.

Can the VAD filter be bypassed for specific use cases?

The VAD is integrated into the AudioPipeline::run loop and is essential for the transcription path. However, the mixed audio stream is also forwarded to recording_sender_for_mixed before VAD processing, allowing the optional WAV recording to contain the complete audio including silence and non-speech segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}