# How Meetily Handles Simultaneous Microphone and System Audio Capture with VAD Filtering

> Discover how Meetily captures mic and system audio simultaneously. Learn about VAD filtering, ring buffers, and seamless audio stream synchronization for accurate Whisper transcription.

- Repository: [Zackriya Solutions/meetily](https://github.com/Zackriya-Solutions/meetily)
- Tags: how-to-guide
- Published: 2026-07-10

---

**Meetily captures microphone and system audio in parallel, synchronizes them via a ring buffer, mixes both streams into a single track, and applies continuous Voice Activity Detection (VAD) to feed only speech segments to Whisper for transcription.**

Meetily is an open-source meeting assistant built by Zackriya-Solutions that records both the user's microphone and system audio simultaneously. Understanding how Meetily handles simultaneous microphone and system audio capture with VAD filtering reveals a sophisticated Rust-based audio pipeline that prioritizes synchronization, clean mixing, and efficient transcription by stripping silence before it reaches the Whisper engine.

## Independent Audio Capture Architecture

Meetily spawns two separate `AudioCapture` instances to handle input from different audio endpoints.

- **`DeviceType::Microphone`**: Captures the user's physical microphone via `cpal` streams in [`frontend/src-tauri/src/audio/capture/microphone.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/capture/microphone.rs)
- **`DeviceType::System`**: Captures system playback audio (e.g., browser audio, media players) via platform-specific APIs in [`frontend/src-tauri/src/audio/capture/system.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/capture/system.rs)

Each capture runs independently and pushes raw PCM chunks to a shared `RecordingState`. This design ensures that latency spikes from one device do not block the other, providing robust real-time performance.

## Resampling and Pre-Processing Pipeline

When audio arrives at non-standard sample rates, Meetily normalizes everything to a **48 kHz** target before mixing.

The resampling logic resides in `AudioCapture::process_audio_data` (lines 99-124 of [`frontend/src-tauri/src/audio/pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/pipeline.rs)). It uses a persistent **Rubato** resampler to avoid allocation overhead during streaming.

Microphone audio receives additional treatment:
- **High-pass filter** to eliminate low-frequency rumble
- **RNNoise** noise suppression (optional)
- **EBU-R128 loudness normalization** to ensure consistent levels

System audio bypasses these filters because it is already considered clean, reducing unnecessary CPU load.

## Ring Buffer Synchronization

The `AudioMixerRingBuffer` (lines 16-45 and 49-85 of [`frontend/src-tauri/src/audio/pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/pipeline.rs)) solves the synchronization challenge between two streams that may deliver samples at slightly different rates.

It maintains two `VecDeque<f32>` buffers—one for microphone samples and one for system samples. The buffer accumulates data until a fixed-size window of approximately **50 ms** is available for *both* streams. This time-alignment step prevents drift and ensures that a sound played on the system at time T is mixed with the user's reaction captured at the same moment.

## Professional Audio Mixing

Once the ring buffer has a complete window, `ProfessionalAudioMixer::mix_window` (lines 54-88 of [`frontend/src-tauri/src/audio/pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/pipeline.rs)) combines the two streams.

The mixing algorithm applies:
- **System audio at ~70% volume** to prevent it from overpowering the microphone
- **Soft-clipping limiter** to keep the final waveform within ±1.0 without distortion
- **No aggressive ducking** to preserve speech clarity—the microphone stays at full level

This approach produces a natural-sounding recording where both sides of the conversation (user and system) are intelligible without jarring volume shifts.

## Continuous VAD Filtering

The mixed audio flows into a `ContinuousVadProcessor` defined in [`frontend/src-tauri/src/audio/vad.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/vad.rs). The VAD operates internally at **16 kHz** (downsampled from the 48 kHz mixed stream) to balance accuracy and performance.

Unlike fixed-interval buffering, the VAD emits **speech segments** only when voice activity is detected. This eliminates silent frames from the transcription queue, significantly reducing data sent to Whisper and lowering latency.

The VAD-to-Whisper dispatch logic (lines 34-58 of [`frontend/src-tauri/src/audio/pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/pipeline.rs)) filters out segments shorter than **800 samples** (~50 ms at 16 kHz). This threshold prevents useless Whisper invocations on brief noise spikes or breath sounds.

## Transcription Dispatch and Recording

Each valid VAD segment is wrapped in an `AudioChunk` struct with a 16 kHz sample rate and dispatched to the Whisper engine via `transcription_sender`. The segment includes precise timestamps to maintain synchronization with the meeting timeline.

Simultaneously, the raw mixed audio (without VAD filtering) can be forwarded to `recording_sender_for_mixed` to produce a complete WAV file containing both microphone and system audio. This dual-path approach ensures that the transcription is efficient while the archival recording remains complete.

## Code Implementation Examples

### Frontend Recording Invocation

```typescript
// Start a meeting from the TypeScript frontend
await invoke('start_recording', {
  mic_device_name: "Built-in Microphone",
  system_device_name: "BlackHole 2ch",   // macOS virtual audio device
  meeting_name: "Team Sync"
});

```

### Pipeline Manager Initialization

```rust
// Rust backend: create and start the dual-capture pipeline
let mut manager = AudioPipelineManager::new();
manager.start(
    recording_state.clone(),
    transcription_sender,
    /* target_chunk_duration_ms */ 0,
    /* sample_rate */ 48000,
    Some(recording_sender),               // receive mixed audio for .wav
    mic_device_name,
    mic_device_kind,
    system_device_name,
    system_device_kind,
)?;

```

### VAD-Driven Processing Loop

```rust
// Inside AudioPipeline::run – continuous VAD processing
while let Some(chunk) = self.receiver.recv().await {
    self.ring_buffer.add_samples(chunk.device_type.clone(), chunk.data);
    while self.ring_buffer.can_mix() {
        let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
        let mixed = self.mixer.mix_window(&mic_win, &sys_win);
        
        // VAD splits speech and sends to Whisper
        for seg in self.vad_processor.process_audio(&mixed)? {
            if seg.samples.len() >= 800 {
                self.transcription_sender.send(AudioChunk {
                    data: seg.samples,
                    sample_rate: 16000,
                    timestamp: seg.start_timestamp_ms / 1000.0,
                    chunk_id: self.chunk_id_counter,
                    device_type: DeviceType::Microphone,
                })?;
            }
        }
    }
}

```

## Key Files in the Audio Pipeline

| File | Purpose |
|------|---------|
| [`frontend/src-tauri/src/audio/pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/pipeline.rs) | Central orchestration: ring buffer, mixing, VAD handling, and dispatch logic |
| [`frontend/src-tauri/src/audio/capture/microphone.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/capture/microphone.rs) | Microphone capture with optional preprocessing (RNNoise, EBU-R128) |
| [`frontend/src-tauri/src/audio/capture/system.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/capture/system.rs) | System audio capture via platform-specific APIs |
| [`frontend/src-tauri/src/audio/vad.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/vad.rs) | `ContinuousVadProcessor` implementation for real-time speech segmentation |
| [`frontend/src-tauri/src/audio/recording_manager.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/recording_manager.rs) | Orchestrates `AudioPipelineManager` lifecycle and mixed-audio recording |
| [`frontend/src-tauri/src/audio/devices/discovery.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/devices/discovery.rs) | Device enumeration for microphone and system audio selection |

## Summary

- **Dual capture**: `AudioCapture` instances for `DeviceType::Microphone` and `DeviceType::System` run in parallel without blocking each other
- **Sample rate normalization**: Rubato resampler converts all input to 48 kHz; microphone audio gets high-pass filtering, optional RNNoise, and EBU-R128 normalization
- **Synchronization**: `AudioMixerRingBuffer` aligns both streams using ~50 ms fixed windows before mixing
- **Clean mixing**: `ProfessionalAudioMixer::mix_window` combines streams at 70% system volume with soft-clipping to prevent distortion
- **Efficient transcription**: `ContinuousVadProcessor` strips silence, emitting only speech segments ≥800 samples (16 kHz) to Whisper
- **Complete recording**: Mixed audio flows to both the VAD transcription path and an optional WAV recording path

## Frequently Asked Questions

### Why does Meetily use a ring buffer instead of simple timestamp matching?

The `AudioMixerRingBuffer` ensures that both microphone and system audio chunks are aligned before mixing, even when the two streams deliver data at different rates or experience jitter. By waiting for a complete ~50 ms window from both sources, the system prevents temporal drift and guarantees that system audio and user speech remain synchronized in the final output.

### What sample rates does the VAD use internally?

The VAD processes mixed audio at **16 kHz** internally, downsampled from the pipeline's **48 kHz** target. This 16 kHz rate provides sufficient resolution for speech detection while keeping computational overhead low for real-time processing.

### How does Meetily prevent system audio from drowning out the microphone?

The `ProfessionalAudioMixer::mix_window` function scales system audio to approximately **70% volume** while keeping the microphone at full level. It also applies a soft-clipping limiter to ensure the combined waveform never exceeds ±1.0, preserving speech clarity without aggressive ducking algorithms.

### Can the VAD filter be bypassed for specific use cases?

The VAD is integrated into the `AudioPipeline::run` loop and is essential for the transcription path. However, the mixed audio stream is also forwarded to `recording_sender_for_mixed` before VAD processing, allowing the optional WAV recording to contain the complete audio including silence and non-speech segments.