How Voice Activity Detection (VAD) Filters Audio Before Sending to Whisper in Meetily

Meetily strips silence and non-speech audio using a ContinuousVadProcessor before forwarding only valid speech segments to Whisper, dramatically reducing CPU usage and preventing transcription hallucinations.

Meetily’s real-time transcription pipeline relies on Voice Activity Detection (VAD) to act as a gatekeeper between raw audio capture and the Whisper model. According to the Zackriya-Solutions/meetily source code, the system processes mixed microphone and system audio through Silero-VAD, emitting only confirmed speech chunks that meet strict duration and confidence thresholds.

How VAD Fits into Meetily’s Audio Pipeline

The VAD component sits between the audio mixing stage and the transcription service. In frontend/src-tauri/src/audio/pipeline.rs, the AudioPipeline::run method orchestrates this flow:

1. Audio Capture: Raw chunks from microphone and system sources enter a ring buffer
2. Mixing: ProfessionalAudioMixer combines streams into a single mono buffer at 48 kHz
3. VAD Filtering: ContinuousVadProcessor analyzes the mixed audio in real-time
4. Segment Emission: Only validated SpeechSegment objects pass through transcription_sender to Whisper

This architecture ensures that large stretches of silence, background noise, and very short audio fragments never reach the transcription model.

Step-by-Step VAD Processing Flow

Step 1: Mixed Audio Generation

The pipeline extracts synchronized windows from the ring buffer and mixes them using ProfessionalAudioMixer. At pipeline.rs lines 1084‑1087, the system produces a clean mixed_with_gain buffer ready for voice analysis.

// pipeline.rs (simplified)
let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
let mixed = self.mixer.mix_window(&mic_win, &sys_win);

Step 2: Silero-VAD Processing

The ContinuousVadProcessor::process_audio method receives the mixed buffer and processes it through Silero-VAD. The processor automatically resamples input to 16 kHz (required by the Silero model) and analyzes audio in 30 ms chunks to detect speech probability.

// vad.rs - ContinuousVadProcessor implementation
for segment in self.vad_processor.process_audio(&mixed)? {
    // Process only confirmed speech segments
}

Step 3: Speech Segment Validation

The VAD emits SpeechSegment objects containing audio samples, timestamps, and confidence scores. The system filters out segments shorter than the configured minimum speech time (default 250 ms), satisfying Whisper’s 100 ms minimum requirement while preventing noise fragments from proceeding.

Configuration in vad.rs lines 17‑27 and 54‑57 establishes these thresholds:

// src/audio/vad.rs
let mut config = VadConfig::default();
config.sample_rate = 16_000;
config.positive_speech_threshold = 0.50;    // Accept speech above 50% confidence
config.negative_speech_threshold = 0.35;    // Release below 35% confidence
config.min_speech_time = Duration::from_millis(250);

Step 4: Whisper-Ready Audio Chunks

Valid segments convert to AudioChunk structs at pipeline.rs lines 1089‑1104. Each chunk specifies 16 kHz sample rate and precise timestamps, then travels through transcription_sender directly to the Whisper inference engine.

let whisper_chunk = AudioChunk {
    data: segment.samples,
    sample_rate: 16_000,
    timestamp: segment.start_timestamp_ms / 1000.0,
    chunk_id: self.chunk_id_counter,
    device_type: DeviceType::Microphone,
};
self.transcription_sender.send(whisper_chunk)?;

Step 5: Final Flush on Recording Stop

When capture ends, AudioPipeline::flush_remaining_audio calls vad_processor.flush() (lines 1093‑1099) to force-end any in-progress speech and emit the final segment, ensuring no trailing words are lost.

Configuration Parameters and Thresholds

The ContinuousVadProcessor::new constructor accepts the input sample rate and redemption time (the grace period for speech pauses). Default configuration values optimize for meeting transcriptions:

• Sample Rate: 16 kHz (Silero requirement)
• Positive Threshold: 0.50 (50% confidence to start speech)
• Negative Threshold: 0.35 (35% confidence to end speech)
• Redemption Time: Configurable (e.g., 2000 ms for natural pauses)
• Minimum Speech Time: 250 ms (filters out coughs, clicks, and noise)

These settings maintain continuous speech segments including natural pauses, while isolating genuine noise.

Code Implementation Examples

Stand-Alone VAD Usage

You can instantiate the processor directly for custom audio processing:

use meetily::frontend::src_tauri::audio::vad::{ContinuousVadProcessor, SpeechSegment};

// Initialize with 48kHz input and 2-second redemption time
let mut vad = ContinuousVadProcessor::new(48_000, 2000)?;
let speech_chunks: Vec<SpeechSegment> = vad.process_audio(&mixed_audio)?;
let final_chunks = vad.flush()?; // Capture any dangling speech

Pipeline Integration

Within Meetily’s production pipeline, VAD integrates seamlessly with the mixing stage:

// Inside AudioPipeline::run (frontend/src-tauri/src/audio/pipeline.rs)
self.ring_buffer.add_samples(chunk.device_type.clone(), chunk.data);
while self.ring_buffer.can_mix() {
    let (mic_win, sys_win) = self.ring_buffer.extract_window().unwrap();
    let mixed = self.mixer.mix_window(&mic_win, &sys_win);
    
    // VAD filters mixed audio before Whisper sees it
    for segment in self.vad_processor.process_audio(&mixed)? {
        let whisper_chunk = AudioChunk {
            data: segment.samples,
            sample_rate: 16_000,
            timestamp: segment.start_timestamp_ms / 1000.0,
            chunk_id: self.chunk_id_counter,
            device_type: DeviceType::Microphone,
        };
        self.transcription_sender.send(whisper_chunk)?;
    }
}

Key Source Files

The VAD implementation spans four critical files in the Rust codebase:

• frontend/src-tauri/src/audio/vad.rs – Implements ContinuousVadProcessor with Silero-VAD integration, resampling logic, and speech segmentation algorithms
• frontend/src-tauri/src/audio/pipeline.rs – Orchestrates the full audio flow, mixing inputs, and routing VAD-filtered chunks to transcription
• frontend/src-tauri/src/audio/audio_capture.rs – Handles raw audio capture and resampling to 48 kHz before VAD processing
• frontend/src-tauri/src/audio/recording_state.rs – Manages the transcription_sender channel that carries filtered audio to Whisper

Summary

• ContinuousVadProcessor acts as a real-time gatekeeper, analyzing 30 ms chunks of mixed audio using Silero-VAD
• Audio resampling to 16 kHz occurs automatically within the VAD component to meet model requirements
• Minimum speech thresholds (250 ms) and confidence levels (0.50/0.35) filter out noise, clicks, and silence
• Direct integration with AudioPipeline::run ensures only validated SpeechSegment objects reach Whisper via transcription_sender
• Computational efficiency improves significantly because Whisper processes only genuine speech, eliminating transcription of silent periods

Frequently Asked Questions

What is the minimum speech duration required for VAD to forward audio to Whisper?

Meetily’s VAD discards any speech segments shorter than 250 ms by default. This threshold, configured in ContinuousVadProcessor, exceeds Whisper’s internal 100 ms minimum and effectively filters out accidental noises, keyboard clicks, and brief coughs that could otherwise generate spurious transcriptions.

Why does the VAD resample audio to 16 kHz?

The Silero-VAD model requires 16 kHz input to function correctly. While Meetily’s pipeline captures and mixes audio at 48 kHz for quality, the ContinuousVadProcessor automatically resamples incoming buffers before analysis. This resampling ensures accurate speech probability calculations while maintaining the original timestamps for synchronization.

How does VAD filtering prevent Whisper hallucinations?

By enforcing a 50% positive confidence threshold and stripping silence below the 35% negative threshold, the VAD ensures Whisper receives only high-quality audio containing actual speech. This prevents the model from attempting to transcribe pure noise or background conversations, which often causes "hallucinations"—phantom words generated when the model tries to interpret random audio patterns.

Can the VAD sensitivity be adjusted for different environments?

Yes. Developers can modify the VadConfig parameters when constructing ContinuousVadProcessor. The positive_speech_threshold and negative_speech_threshold values control how aggressively the system detects speech, while redemption_time adjusts the grace period for natural pauses in conversation. These settings apply per-instance, allowing different sensitivity profiles for quiet offices versus noisy meeting rooms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}