How Voice Activity Detection (VAD) Filters Audio Before Whisper Transcription in Meetily

Meetily's ContinuousVadProcessor strips silence and non-speech audio from mixed microphone and system streams before sending only valid speech segments to Whisper, reducing computational overhead and preventing transcription hallucinations.

Meetily’s real-time transcription pipeline relies on Voice Activity Detection (VAD) to preprocess audio before it reaches OpenAI’s Whisper model. By intercepting mixed audio streams and removing silence, background noise, and short audio fragments, the VAD ensures that Whisper processes only genuine speech content. This preprocessing step, implemented in Rust within the Tauri-based frontend, dramatically improves transcription accuracy while minimizing CPU and GPU usage.

How the VAD Pipeline Works

The VAD filtering process follows a strict five-stage pipeline defined in frontend/src-tauri/src/audio/pipeline.rs and frontend/src-tauri/src/audio/vad.rs.

Step 1: Mixing Raw Audio Streams

The process begins in AudioPipeline::run, which continuously pulls raw audio chunks from an internal ring buffer. The pipeline mixes microphone and system audio using ProfessionalAudioMixer, producing a clean mono buffer called mixed_with_gain. This occurs at the pipeline's native 48 kHz sample rate before any voice detection begins.

Source: pipeline.rs#L1084-L1087

Step 2: Detecting Voice Activity

The mixed buffer feeds directly into ContinuousVadProcessor::process_audio. The processor runs Silero-VAD on 30 ms chunks, automatically resampling the input to 16 kHz if necessary to meet Silero's requirements. This real-time analysis calculates speech probability for each micro-window.

Source: pipeline.rs#L1085-L1088

Step 3: Filtering Speech Segments

For every detected utterance, the VAD returns a SpeechSegment containing audio samples, timestamps, and a confidence score. The processor applies a minimum speech time filter of 250 ms—segments shorter than this threshold are discarded to satisfy Whisper's 100 ms minimum requirement and prevent 40 ms noise fragments from being transcribed.

Source: vad.rs#L17-L27 and vad.rs#L54-L57

Step 4: Transmitting Clean Audio to Whisper

Each valid SpeechSegment gets wrapped in an AudioChunk structure with a fixed sample rate of 16 kHz. The pipeline sends this chunk over transcription_sender to the Whisper service. Silence, background noise, and sub-threshold fragments never leave the pipeline, ensuring Whisper receives only bona-fide speech.

Source: pipeline.rs#L1089-L1104

Step 5: Flushing Final Segments

When recording stops, AudioPipeline::flush_remaining_audio calls vad_processor.flush() to force-end any in-progress speech detection. This ensures that trailing words which might not have triggered the end-of-speech threshold are still captured and sent for transcription.

Source: pipeline.rs#L1093-L1099

VAD Configuration and Thresholds

The ContinuousVadProcessor initializes with sensible defaults in ContinuousVadProcessor::new that balance responsiveness against noise filtering:

// frontend/src-tauri/src/audio/vad.rs
let mut config = VadConfig::default();
config.sample_rate = 16_000;                 // Silero requires 16 kHz
config.positive_speech_threshold = 0.50;    // Accept speech above 50%
config.negative_speech_threshold = 0.35;    // Drop below 35%
config.redemption_time = Duration::from_millis(2000); // Allow 2s pauses
config.min_speech_time = Duration::from_millis(250);  // Filter sub-250ms noise

These settings ensure that natural speech patterns—including brief pauses within sentences—remain concatenated into single segments, while isolated noise bursts get filtered out before reaching the transcription engine.

Implementation Examples

Using the VAD Processor Directly

You can instantiate the VAD processor standalone for custom audio processing:

use meetily::frontend::src_tauri::audio::vad::{ContinuousVadProcessor, SpeechSegment};

// Initialize with 48kHz input and 2-second redemption time
let mut vad = ContinuousVadProcessor::new(48_000, 2000)?;
let speech_chunks: Vec<SpeechSegment> = vad.process_audio(&mixed_audio)?;
let final_chunks = vad.flush()?; // Capture any trailing speech

Integrating VAD in the Audio Pipeline

The AudioPipeline::run method demonstrates the integration pattern:

// Inside AudioPipeline::run (simplified)
self.ring_buffer.add_samples(chunk.device_type.clone(), chunk.data);
while self.ring_buffer.can_mix() {
    let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
    let mixed = self.mixer.mix_window(&mic_win, &sys_win);

    // VAD filters the mixed audio
    for segment in self.vad_processor.process_audio(&mixed)? {
        // Only speech reaches Whisper
        let whisper_chunk = AudioChunk {
            data: segment.samples,
            sample_rate: 16_000,
            timestamp: segment.start_timestamp_ms / 1000.0,
            chunk_id: self.chunk_id_counter,
            device_type: DeviceType::Microphone,
        };
        self.transcription_sender.send(whisper_chunk)?;
    }
}

Key Source Files

File	Role in VAD Filtering
frontend/src-tauri/src/audio/vad.rs	Implements ContinuousVadProcessor, configures Silero VAD, handles resampling, segmentation, and flushing.
frontend/src-tauri/src/audio/pipeline.rs	Orchestrates ring-buffer mixing, runs VAD on each mixed window, and forwards speech segments to Whisper.
frontend/src-tauri/src/audio/audio_capture.rs	Captures raw audio, resamples to the pipeline’s 48 kHz, and pushes chunks into the pipeline (pre-VAD).
frontend/src-tauri/src/audio/recording_state.rs	Holds the channel used by the pipeline to send VAD-filtered chunks to the transcription service.

Summary

• Voice Activity Detection acts as a gatekeeper between mixed audio capture and Whisper transcription, processing audio in 30 ms chunks at 16 kHz.
• The 250 ms minimum speech time filters out noise fragments and breathing sounds that could cause hallucinations.
• Silero-VAD thresholds (0.50 positive, 0.35 negative) with a configurable redemption time allow natural speech pauses while cutting silence.
• Only validated SpeechSegment objects wrapped in AudioChunk structures reach the transcription service, reducing computational load and latency.
• The flush mechanism ensures no trailing speech is lost when recording stops.

Frequently Asked Questions

What is the minimum speech duration required by Meetily's VAD?

Meetily's VAD requires a minimum speech time of 250 ms per the min_speech_time configuration in vad.rs. This threshold prevents the transcription of sub-100 ms noise fragments and brief audio artifacts that could trigger Whisper hallucinations.

Why does the VAD resample audio to 16 kHz?

The Silero VAD model requires 16 kHz PCM audio input. Since Meetily's pipeline operates at 48 kHz for quality mixing, ContinuousVadProcessor automatically resamples the mixed buffer before voice detection, then maintains the 16 kHz rate for the Whisper-compatible AudioChunk output.

How does VAD filtering improve Whisper transcription performance?

By preprocessing audio through VAD, Meetily reduces CPU/GPU usage (silence is never transcribed), cuts latency (Whisper begins processing speech immediately rather than waiting for fixed time windows), and prevents hallucinations (background noise and keyboard clicks are filtered out before reaching the model).

What happens to audio segments when recording stops?

When recording terminates, AudioPipeline::flush_remaining_audio invokes vad_processor.flush(), which forces the VAD to immediately emit any speech currently in its internal buffer. This ensures that final words or trailing syllables are captured and sent to Whisper rather than being discarded as incomplete segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}