Meetily Audio Flow: From Device Capture to Whisper Transcription

Meetily processes audio through a fully asynchronous Rust pipeline that captures microphone and system streams separately, normalizes and mixes them via a ring buffer, applies voice activity detection to isolate speech segments, and streams only voiced audio to a local Whisper model for real-time transcription.

Meetily (Zackriya-Solutions/meetily) implements a sophisticated audio architecture designed for meeting transcription. Understanding the complete audio flow from capture to transcription in Meetily reveals how the application balances real-time performance with transcription accuracy using local AI models.

Audio Capture and Pre-Processing

Meetily initiates dual capture streams to record both sides of a conversation. The AudioCapture struct in frontend/src-tauri/src/audio/audio_capture.rs handles raw input from microphones and system audio devices independently.

Device Input Handling

Each audio source creates an AudioCapture instance configured for its specific device type. The process_audio_data method performs several critical transformations:

• Channel conversion: Multi-channel audio is converted to mono using audio_to_mono
• Resampling: Audio is resampled to 48 kHz when necessary using internal buffers
• Format normalization: All streams are standardized to floating-point samples for downstream processing

Real-Time Enhancement Pipeline

For microphone inputs, Meetily applies professional-grade audio enhancement before forwarding to the mixer:

// frontend/src-tauri/src/audio/audio_capture.rs
pub fn process_audio_data(&self, data: &[f32]) {
    // Convert to mono
    let mut mono = if self.channels > 1 {
        audio_to_mono(data, self.channels)
    } else {
        data.to_vec()
    };

    // Resample to 48 kHz if needed
    if self.needs_resampling {
        mono = self.resample_with_buffer(&mono);
    }

    // Apply high-pass → RNNoise → loudness normalisation (mic only)
    if matches!(self.device_type, DeviceType::Microphone) {
        if let Some(ref mut f) = *self.high_pass_filter.lock().unwrap() {
            mono = f.process(&mono);
        }
        if RNNOISE_APPLY_ENABLED {
            if let Some(ref mut ns) = *self.noise_suppressor.lock().unwrap() {
                mono = ns.process(&mono);
            }
        }
        if let Some(ref mut n) = *self.normalizer.lock().unwrap() {
            mono = n.normalize_loudness(&mono);
        }
    }

    // Send to pipeline
    let chunk = AudioChunk { data: mono, sample_rate: 48000, … };
    self.state.send_audio_chunk(chunk).ok();
}

The enhancement chain includes high-pass filtering, RNNoise-based suppression, and EBU R128 loudness normalization, ensuring consistent levels regardless of input device characteristics.

Ring Buffer Synchronization and Mixing

Once captured, audio streams enter the AudioPipeline in frontend/src-tauri/src/audio/pipeline.rs, which synchronizes asynchronous inputs using a ring buffer architecture.

Temporal Alignment via Ring Buffer

The AudioMixerRingBuffer aligns microphone and system streams into fixed 50 ms windows (approximately 2400 samples at 48 kHz). The buffer handles clock drift through intelligent padding:

• Underflow protection: Zero-pads streams that run slow
• Overflow handling: Trims excess samples with warning logs

Key methods include add_samples for ingestion and extract_window for retrieval, implemented in pipeline.rs lines 16–40 and 49–84.

Professional Audio Mixing

The ProfessionalAudioMixer::mix_window method (lines 45–87) combines aligned windows using soft-scaling to prevent clipping:

// Simplified mixing logic from pipeline.rs
let mixed = self.mixer.mix_window(&mic_window, &sys_window);

The mixer scales system audio to leave headroom and applies proportional scaling when the combined signal exceeds ±1.0, ensuring clean output for both transcription and optional recording.

Voice Activity Detection and Segmentation

After mixing, audio flows into the continuous VAD processor (frontend/src-tauri/src/audio/vad.rs) to eliminate silence and non-speech segments.

Continuous VAD Processing

Inside AudioPipeline::run, the pipeline calls self.vad_processor.process_audio(&mixed_with_gain) (lines 34–66) to analyze each mixed window. The VAD implements a streaming architecture that:

1. Maintains internal state across windows
2. Detects speech onset and offset
3. Buffers segments until speech ends or timeout occurs

Speech Segment Filtering

The pipeline filters out segments smaller than 800 samples (approximately 50 ms at 16 kHz), which eliminates transient noises and breathing sounds:

// Inside AudioPipeline::run (pipeline.rs)
match self.vad_processor.process_audio(&mixed) {
    Ok(segments) => for seg in segments {
        if seg.samples.len() >= 800 {
            let transcription_chunk = AudioChunk {
                data: seg.samples,
                sample_rate: 16000,  // Downsampled for Whisper
                timestamp: seg.start_timestamp_ms as f64 / 1000.0,
                chunk_id: self.chunk_id_counter,
                device_type: DeviceType::Microphone,
            };
            self.transcription_sender.send(transcription_chunk).ok();
            self.chunk_id_counter += 1;
        }
    },
    Err(e) => warn!("VAD error: {}", e),
}

When recording stops, self.vad_processor.flush() (lines 99–115) processes any remaining buffered audio to ensure no trailing speech is lost.

Whisper Transcription Provider

Valid speech segments travel via Tokio mpsc::UnboundedChannel to the transcription subsystem in frontend/src-tauri/src/audio/transcription/provider.rs.

Model Management and Inference

The WhisperProvider struct manages the local Whisper engine:

• Model loading: WhisperEngine::load_model initializes the chosen model (base, medium, etc.) once and reuses it for all subsequent chunks
• Inference: WhisperProvider::transcribe converts AudioChunk data into text using the loaded engine
• Resource efficiency: Single model instance serves all transcription requests to minimize memory overhead and loading latency

Each AudioChunk arrives with a 16 kHz sample rate, the optimal input format for Whisper models, converted during the VAD stage.

Optional Mixed Audio Recording

Parallel to transcription, the pipeline supports writing the mixed audio stream to disk. The recording_sender_for_mixed channel (lines 68–78 in pipeline.rs) forwards mixed chunks to frontend/src-tauri/src/audio/recording_saver.rs, which encodes the stream into a standard WAV file for later playback or archival.

This occurs before VAD processing, ensuring the recording contains continuous audio rather than speech-only segments.

Summary

• Dual capture: Meetily captures microphone and system audio separately through AudioCapture instances, applying device-specific enhancements like RNNoise and EBU R128 normalization only to microphone inputs.

• Synchronized mixing: A ring buffer aligns streams into 50 ms windows, handled by ProfessionalAudioMixer to prevent clipping while combining sources.

• Speech isolation: Continuous VAD processing filters out silence and noise, forwarding only segments exceeding 800 samples to reduce Whisper computational load.

• Local transcription: The WhisperProvider manages a persistent model instance, processing 16 kHz audio chunks through the local Whisper engine for privacy-preserving transcription.

• Async architecture: All stages communicate via Tokio unbounded channels, enabling real-time processing with graceful shutdown handling through flush signals.

Frequently Asked Questions

What sample rates does Meetily use throughout the pipeline?

Meetily captures audio at 48 kHz to accommodate high-quality system audio and provide headroom for processing. Before transcription, the audio is resampled to 16 kHz, which is the optimal input rate for OpenAI's Whisper models and reduces computational overhead during inference.

How does Meetily synchronize microphone and system audio captured from different devices?

The AudioMixerRingBuffer in pipeline.rs implements a sliding window approach that aligns streams temporally. When one stream lags, the buffer zero-pads the missing samples; when one runs ahead, it trims the excess. This ensures the ProfessionalAudioMixer always receives temporally aligned windows for combining.

Why does Meetily apply noise suppression only to microphone audio?

The RNNoise suppression and EBU R128 normalization target microphone inputs only because they typically contain more variable acoustic environments, room noise, and inconsistent levels. System audio (loopback capture) generally originates from digital sources with consistent levels and minimal noise, making aggressive processing unnecessary and potentially harmful to audio quality.

How does the VAD filter improve transcription accuracy?

By discarding segments shorter than 800 samples (50 ms) and silences between speech, the continuous VAD processor eliminates non-speech audio from reaching Whisper. This reduces hallucinations caused by background noise, breathing, or keyboard clicks, while lowering CPU usage by avoiding inference on silent passages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}