How Voice Activity Detection (VAD) in Meetily Filters Audio for Whisper Transcription

Meetily's Voice Activity Detection (VAD) continuously analyzes mixed audio streams to extract only human speech segments, discarding silence and noise before sending the filtered audio to OpenAI's Whisper model for transcription.

In the open-source Meetily repository (Zackriya-Solutions/meetily), the audio processing pipeline employs a strategic Voice Activity Detection (VAD) layer that acts as a gatekeeper between raw audio capture and speech recognition. By processing audio in real-time through Silero VAD, the system ensures that Whisper receives only clean, validated speech segments, eliminating computational waste on silent or noisy passages.

How VAD Filters Audio in the Meetily Pipeline

The VAD integration operates as a continuous filter within the Rust-based audio pipeline, processing mixed microphone and system audio before it reaches the transcription service.

Step 1: Audio Mixing and Capture

The process begins in frontend/src-tauri/src/audio/pipeline.rs where the AudioPipeline::run method manages the audio flow. Raw audio chunks flow from the ring buffer through the ProfessionalAudioMixer, which combines microphone and system audio into a clean mono buffer called mixed_with_gain.

According to the source code at lines 1084-1087, this mixed buffer represents the combined audio stream that requires filtering before transcription.

Step 2: Real-Time Speech Detection

Once mixed, the audio immediately enters the ContinuousVadProcessor::process_audio method defined in frontend/src-tauri/src/audio/vad.rs. The processor divides the incoming stream into 30 ms chunks and applies Silero VAD analysis, automatically resampling the input to 16 kHz when necessary to meet the model's requirements.

As implemented in lines 17-27 and 54-57 of vad.rs, the VAD evaluates each chunk against configured confidence thresholds to determine speech presence.

Step 3: Segment Filtering and Validation

For every detected speech region, the VAD generates a SpeechSegment struct containing the audio samples, timestamps, and confidence scores. However, not all segments qualify for transcription.

The system enforces a minimum speech duration of 250 ms (configurable via min_speech_time), discarding shorter fragments that fall below Whisper's effective 100 ms processing threshold. This validation occurs in vad.rs lines 90-95, ensuring only substantial speech segments proceed through the pipeline.

Step 4: Transmission to Whisper

Valid SpeechSegment instances are wrapped into AudioChunk structures with a fixed 16 kHz sample rate and transmitted via transcription_sender. As shown in pipeline.rs lines 1089-1104, silence, background noise, and sub-threshold fragments never leave the pipeline, meaning Whisper processes only genuine speech content.

When recording stops, AudioPipeline::flush_remaining_audio calls vad_processor.flush() to force-end any in-progress speech and emit the final segments (lines 1093-1099).

VAD Configuration and Thresholds

The ContinuousVadProcessor::new constructor in vad.rs initializes the VAD with precise acoustic parameters optimized for meeting transcription:

// frontend/src-tauri/src/audio/vad.rs
let mut config = VadConfig::default();
config.sample_rate = 16_000;                 // Silero requires 16 kHz
config.positive_speech_threshold = 0.50;    // Accept speech above 50% confidence
config.negative_speech_threshold = 0.35;    // Drop below 35% confidence
config.redemption_time = Duration::from_millis(redemption_time_ms); // e.g., 2000 ms
config.min_speech_time = Duration::from_millis(250);               // Prevent 40 ms fragments

These settings ensure that continuous speech—including natural pauses up to the redemption time—remains intact as single segments, while isolated noises or brief utterances are filtered out.

Implementation Details and Code Examples

The VAD Processor Structure

You can utilize the VAD component directly for standalone speech extraction:

use meetily::frontend::src_tauri::audio::vad::{ContinuousVadProcessor, SpeechSegment};

let mut vad = ContinuousVadProcessor::new(48_000, 2000)?; // 48 kHz input, 2 s redemption
let speech_chunks: Vec<SpeechSegment> = vad.process_audio(&mixed_audio)?;
let final_chunks = vad.flush()?; // Force-end any dangling speech

Pipeline Integration

Within the production pipeline, VAD processing occurs inside the main audio loop:

// Inside AudioPipeline::run (simplified)
self.ring_buffer.add_samples(chunk.device_type.clone(), chunk.data);
while self.ring_buffer.can_mix() {
    let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
    let mixed = self.mixer.mix_window(&mic_win, &sys_win);

    // VAD filters the mixed audio
    for segment in self.vad_processor.process_audio(&mixed)? {
        // Only speech reaches Whisper
        let whisper_chunk = AudioChunk {
            data: segment.samples,
            sample_rate: 16_000,
            timestamp: segment.start_timestamp_ms / 1000.0,
            chunk_id: self.chunk_id_counter,
            device_type: DeviceType::Microphone,
        };
        self.transcription_sender.send(whisper_chunk)?;
    }
}

Performance Benefits of VAD Filtering

Integrating Voice Activity Detection directly into Meetily's audio pipeline delivers three critical advantages:

• Computational Efficiency – Whisper receives only genuine speech segments, eliminating GPU/CPU cycles wasted on transcribing silence or background noise.
• Transcription Accuracy – By filtering out noise and very short audio fragments, the system prevents Whisper from generating hallucinated words or spurious text from non-speech audio.
• Latency Reduction – Real-time segmentation allows Whisper to begin processing each speech chunk immediately upon detection, rather than waiting for fixed-size time windows to fill.

Summary

• Meetily's VAD operates in frontend/src-tauri/src/audio/vad.rs using the ContinuousVadProcessor struct to wrap Silero VAD functionality.
• The AudioPipeline::run method in pipeline.rs coordinates mixing and VAD processing before sending audio to Whisper.
• Audio is resampled to 16 kHz and analyzed in 30 ms chunks, with segments shorter than 250 ms discarded.
• Configuration thresholds (0.50 positive, 0.35 negative) and redemption time (typically 2000 ms) balance responsiveness against noise filtering.
• Only validated SpeechSegment instances wrapped in AudioChunk structures reach the transcription service, optimizing both performance and accuracy.

Frequently Asked Questions

What is the minimum speech duration Meetily's VAD will process?

Meetily configures the VAD with a min_speech_time of 250 milliseconds, as defined in the ContinuousVadProcessor initialization. Segments shorter than this threshold are discarded to prevent Whisper from processing ineffective 40 ms fragments and to avoid the 100 ms minimum limitation of the transcription model.

Why does Meetily resample audio to 16 kHz before VAD processing?

The Silero VAD model requires 16 kHz sample rate input for accurate speech detection. According to the implementation in vad.rs, the ContinuousVadProcessor automatically resamples incoming audio from the pipeline's native 48 kHz (or other capture rates) to 16 kHz before analysis, ensuring compatibility while maintaining the original audio quality for downstream processing.

How does Meetily handle audio when a meeting ends?

When recording stops, the AudioPipeline::flush_remaining_audio method calls vad_processor.flush() to force-end any speech segments currently in progress. As documented in pipeline.rs lines 1093-1099, this ensures that trailing speech near the end of a meeting is captured and transmitted to Whisper rather than being lost in the buffer.

What happens to non-speech audio like typing or background music?

Non-speech audio is filtered at the VAD layer and never reaches Whisper. Audio chunks that fall below the negative speech threshold of 0.35 confidence are classified as non-speech and discarded. Continuous sounds like typing or music typically fail to maintain the positive speech threshold of 0.50 for the required duration, causing the VAD to exclude them from the transcription stream.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}