# Meetily Audio Flow: From Device Capture to Whisper Transcription

> Discover the Meetily audio flow: Capture device audio, process with a Rust pipeline, and achieve real-time Whisper transcription with voice activity detection.

- Repository: [Zackriya Solutions/meetily](https://github.com/Zackriya-Solutions/meetily)
- Tags: internals
- Published: 2026-07-25

---

**Meetily processes audio through a fully asynchronous Rust pipeline that captures microphone and system streams separately, normalizes and mixes them via a ring buffer, applies voice activity detection to isolate speech segments, and streams only voiced audio to a local Whisper model for real-time transcription.**

Meetily (Zackriya-Solutions/meetily) implements a sophisticated audio architecture designed for meeting transcription. Understanding the complete audio flow from capture to transcription in Meetily reveals how the application balances real-time performance with transcription accuracy using local AI models.

## Audio Capture and Pre-Processing

Meetily initiates dual capture streams to record both sides of a conversation. The `AudioCapture` struct in [`frontend/src-tauri/src/audio/audio_capture.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/audio_capture.rs) handles raw input from microphones and system audio devices independently.

### Device Input Handling

Each audio source creates an `AudioCapture` instance configured for its specific device type. The `process_audio_data` method performs several critical transformations:

- **Channel conversion**: Multi-channel audio is converted to mono using `audio_to_mono`
- **Resampling**: Audio is resampled to 48 kHz when necessary using internal buffers
- **Format normalization**: All streams are standardized to floating-point samples for downstream processing

### Real-Time Enhancement Pipeline

For microphone inputs, Meetily applies professional-grade audio enhancement before forwarding to the mixer:

```rust
// frontend/src-tauri/src/audio/audio_capture.rs
pub fn process_audio_data(&self, data: &[f32]) {
    // Convert to mono
    let mut mono = if self.channels > 1 {
        audio_to_mono(data, self.channels)
    } else {
        data.to_vec()
    };

    // Resample to 48 kHz if needed
    if self.needs_resampling {
        mono = self.resample_with_buffer(&mono);
    }

    // Apply high-pass → RNNoise → loudness normalisation (mic only)
    if matches!(self.device_type, DeviceType::Microphone) {
        if let Some(ref mut f) = *self.high_pass_filter.lock().unwrap() {
            mono = f.process(&mono);
        }
        if RNNOISE_APPLY_ENABLED {
            if let Some(ref mut ns) = *self.noise_suppressor.lock().unwrap() {
                mono = ns.process(&mono);
            }
        }
        if let Some(ref mut n) = *self.normalizer.lock().unwrap() {
            mono = n.normalize_loudness(&mono);
        }
    }

    // Send to pipeline
    let chunk = AudioChunk { data: mono, sample_rate: 48000, … };
    self.state.send_audio_chunk(chunk).ok();
}

```

The enhancement chain includes high-pass filtering, RNNoise-based suppression, and **EBU R128 loudness normalization**, ensuring consistent levels regardless of input device characteristics.

## Ring Buffer Synchronization and Mixing

Once captured, audio streams enter the `AudioPipeline` in [`frontend/src-tauri/src/audio/pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/pipeline.rs), which synchronizes asynchronous inputs using a ring buffer architecture.

### Temporal Alignment via Ring Buffer

The `AudioMixerRingBuffer` aligns microphone and system streams into fixed **50 ms windows** (approximately 2400 samples at 48 kHz). The buffer handles clock drift through intelligent padding:

- **Underflow protection**: Zero-pads streams that run slow
- **Overflow handling**: Trims excess samples with warning logs

Key methods include `add_samples` for ingestion and `extract_window` for retrieval, implemented in [`pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/pipeline.rs) lines 16–40 and 49–84.

### Professional Audio Mixing

The `ProfessionalAudioMixer::mix_window` method (lines 45–87) combines aligned windows using soft-scaling to prevent clipping:

```rust
// Simplified mixing logic from pipeline.rs
let mixed = self.mixer.mix_window(&mic_window, &sys_window);

```

The mixer scales system audio to leave headroom and applies proportional scaling when the combined signal exceeds ±1.0, ensuring clean output for both transcription and optional recording.

## Voice Activity Detection and Segmentation

After mixing, audio flows into the **continuous VAD processor** ([`frontend/src-tauri/src/audio/vad.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/vad.rs)) to eliminate silence and non-speech segments.

### Continuous VAD Processing

Inside `AudioPipeline::run`, the pipeline calls `self.vad_processor.process_audio(&mixed_with_gain)` (lines 34–66) to analyze each mixed window. The VAD implements a streaming architecture that:

1. Maintains internal state across windows
2. Detects speech onset and offset
3. Buffers segments until speech ends or timeout occurs

### Speech Segment Filtering

The pipeline filters out segments smaller than **800 samples** (approximately 50 ms at 16 kHz), which eliminates transient noises and breathing sounds:

```rust
// Inside AudioPipeline::run (pipeline.rs)
match self.vad_processor.process_audio(&mixed) {
    Ok(segments) => for seg in segments {
        if seg.samples.len() >= 800 {
            let transcription_chunk = AudioChunk {
                data: seg.samples,
                sample_rate: 16000,  // Downsampled for Whisper
                timestamp: seg.start_timestamp_ms as f64 / 1000.0,
                chunk_id: self.chunk_id_counter,
                device_type: DeviceType::Microphone,
            };
            self.transcription_sender.send(transcription_chunk).ok();
            self.chunk_id_counter += 1;
        }
    },
    Err(e) => warn!("VAD error: {}", e),
}

```

When recording stops, `self.vad_processor.flush()` (lines 99–115) processes any remaining buffered audio to ensure no trailing speech is lost.

## Whisper Transcription Provider

Valid speech segments travel via Tokio `mpsc::UnboundedChannel` to the transcription subsystem in [`frontend/src-tauri/src/audio/transcription/provider.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/transcription/provider.rs).

### Model Management and Inference

The `WhisperProvider` struct manages the local Whisper engine:

- **Model loading**: `WhisperEngine::load_model` initializes the chosen model (base, medium, etc.) once and reuses it for all subsequent chunks
- **Inference**: `WhisperProvider::transcribe` converts `AudioChunk` data into text using the loaded engine
- **Resource efficiency**: Single model instance serves all transcription requests to minimize memory overhead and loading latency

Each `AudioChunk` arrives with a **16 kHz sample rate**, the optimal input format for Whisper models, converted during the VAD stage.

## Optional Mixed Audio Recording

Parallel to transcription, the pipeline supports writing the mixed audio stream to disk. The `recording_sender_for_mixed` channel (lines 68–78 in [`pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/pipeline.rs)) forwards mixed chunks to [`frontend/src-tauri/src/audio/recording_saver.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/audio/recording_saver.rs), which encodes the stream into a standard WAV file for later playback or archival.

This occurs **before** VAD processing, ensuring the recording contains continuous audio rather than speech-only segments.

## Summary

- **Dual capture**: Meetily captures microphone and system audio separately through `AudioCapture` instances, applying device-specific enhancements like RNNoise and EBU R128 normalization only to microphone inputs.

- **Synchronized mixing**: A ring buffer aligns streams into 50 ms windows, handled by `ProfessionalAudioMixer` to prevent clipping while combining sources.

- **Speech isolation**: Continuous VAD processing filters out silence and noise, forwarding only segments exceeding 800 samples to reduce Whisper computational load.

- **Local transcription**: The `WhisperProvider` manages a persistent model instance, processing 16 kHz audio chunks through the local Whisper engine for privacy-preserving transcription.

- **Async architecture**: All stages communicate via Tokio unbounded channels, enabling real-time processing with graceful shutdown handling through flush signals.

## Frequently Asked Questions

### What sample rates does Meetily use throughout the pipeline?

Meetily captures audio at **48 kHz** to accommodate high-quality system audio and provide headroom for processing. Before transcription, the audio is resampled to **16 kHz**, which is the optimal input rate for OpenAI's Whisper models and reduces computational overhead during inference.

### How does Meetily synchronize microphone and system audio captured from different devices?

The `AudioMixerRingBuffer` in [`pipeline.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/pipeline.rs) implements a sliding window approach that aligns streams temporally. When one stream lags, the buffer zero-pads the missing samples; when one runs ahead, it trims the excess. This ensures the `ProfessionalAudioMixer` always receives temporally aligned windows for combining.

### Why does Meetily apply noise suppression only to microphone audio?

The RNNoise suppression and EBU R128 normalization target **microphone inputs only** because they typically contain more variable acoustic environments, room noise, and inconsistent levels. System audio (loopback capture) generally originates from digital sources with consistent levels and minimal noise, making aggressive processing unnecessary and potentially harmful to audio quality.

### How does the VAD filter improve transcription accuracy?

By discarding segments shorter than 800 samples (50 ms) and silences between speech, the **continuous VAD processor** eliminates non-speech audio from reaching Whisper. This reduces hallucinations caused by background noise, breathing, or keyboard clicks, while lowering CPU usage by avoiding inference on silent passages.