Meetily Audio Pipeline Architecture: How It Synchronizes Dual Audio Streams

Meetily captures microphone and system audio as independent PCM streams, buffers them in a lock-free ring buffer, and synchronizes them by extracting fixed 50 ms windows with zero-padding to maintain perfect temporal alignment before mixing and transcription.

The open-source meeting assistant Meetily (Zackriya-Solutions/meetily) implements a sophisticated audio pipeline architecture that records both user voice and desktop audio simultaneously. This Rust-based pipeline, primarily implemented in frontend/src-tauri/src/audio/pipeline.rs and orchestrated by frontend/src-tauri/src/audio/recording_manager.rs, handles real-time resampling, professional-grade mixing, and voice activity detection while ensuring the two independent streams remain perfectly synchronized.

Dual Stream Capture and Resampling

Independent Device Capture

The pipeline captures two distinct Pulse Code Modulation (PCM) sources independently: the microphone (user voice) and the system audio (desktop output). Each stream originates from separate OS audio APIs—such as macOS Core Audio or Windows WASAPI—and enters the pipeline as raw audio chunks via device-specific implementations in frontend/src-tauri/src/audio/capture/microphone.rs.

Persistent Resampling with Rubato

To ensure uniformity, the architecture resamples all input to a target 48 kHz sample rate. The AudioCapture struct maintains a reusable Rubato resampler with a dedicated resampler_input_buffer, allowing variable-size chunks to be correctly up-sampled or down-sampled without energy loss. This logic resides in process_audio_data (lines 1007–1052).

The Synchronization Mechanism

Lock-Free Ring Buffer Architecture

The core synchronization relies on the AudioMixerRingBuffer struct, which creates isolated buffers for microphone and system audio. Upon initialization (AudioMixerRingBuffer::new at lines 27–35), it establishes a fixed mixing window (approximately 50 ms) and a safety limit of eight times that window to prevent unbounded memory growth.

Incoming chunks route through add_samples (lines 49–84), which appends data to the appropriate mic_buffer or system_buffer. If either buffer exceeds the safety limit, the oldest samples drop to maintain real-time performance.

Window-Based Extraction and Zero-Padding

Synchronization occurs in extract_window (lines 92–141). The pipeline checks readiness via can_mix (lines 87–90), which returns true when at least one buffer contains a full window of samples. This non-blocking approach ensures the pipeline never stalls waiting for the slower stream.

When extracting, the system retrieves fixed-size windows from both buffers. If one stream lags, the missing samples are zero-padded (filled with silence) rather than repeating the last sample. This technique prevents audible artifacts and maintains precise temporal alignment between the two sources.

Soft-Scaling Mixing

The ProfessionalAudioMixer::mix_window method (lines 154–188) combines the aligned windows using soft scaling. When the sum of microphone and system samples exceeds ±1.0, the mixer proportionally scales both signals to prevent clipping without aggressive ducking. This yields a clean, undistorted final track written to the WAV file via recording_sender_for_mixed (lines 689–779).

Audio Processing and Enhancement

Microphone Signal Enhancement

Before entering the ring buffer, microphone audio undergoes enhancement in frontend/src-tauri/src/audio/audio_processing.rs. The signal chain includes:

• High-pass filtering to remove low-frequency rumble.
• RNNoise noise suppression for broadband noise reduction.
• EBU R128 loudness normalization to ensure consistent volume levels.

Voice Activity Detection

The mixed audio stream feeds into the ContinuousVadProcessor defined in frontend/src-tauri/src/audio/vad.rs. The process_audio method (lines 335–367) segments the audio into speech and non-speech regions, forwarding only segments exceeding 800 samples (approximately 50 ms at 16 kHz) to the Whisper transcription engine. This dramatically reduces computational load while preserving transcription accuracy.

Pipeline Orchestration and Thread Safety

AudioPipelineManager Implementation

The AudioPipelineManager struct coordinates the entire flow. It initializes the capture interfaces and manages thread-safe communication using RecordingState, which stores an Arc<AtomicBool> and a RwLock-protected sender. This design allows capture threads to push chunks to the pipeline without blocking the Tauri UI thread.

Graceful Shutdown with Flush Markers

To eliminate shutdown delays, the pipeline recognizes special flush chunks marked with chunk_id >= u64::MAX - 10 (lines 781–889). These markers force the pipeline to empty all remaining buffers before exiting, ensuring no audio data is lost during the final write.

Code Implementation

Below are practical examples showing how to instantiate the pipeline and push audio data:

// Create the pipeline manager from a Tauri command context
let mut pipeline_manager = AudioPipelineManager::new();

// Initialize with device metadata and senders
pipeline_manager.start(
    recording_state.clone(),
    transcription_sender.clone(),
    0,  // target_chunk_duration_ms
    48000,  // sample_rate
    Some(mixed_recording_sender.clone()),
    "Built‑in Microphone".to_string(),
    mic_device_kind,
    "BlackHole 2ch".to_string(),
    system_device_kind,
)?;

// Inside the audio callback for each device
let audio_capture = AudioCapture::new(
    device.clone(),
    recording_state.clone(),
    device.sample_rate(),
    device.channels(),
    DeviceType::Microphone,  // or DeviceType::System
    Some(pipeline_manager.audio_sender().clone()),
);
audio_capture.process_audio_data(&pcm_samples);

// Trigger a fast shutdown when stopping recording
pipeline_manager.force_flush_and_stop().await?;

Summary

• Dual independent capture: Microphone and system audio are recorded separately as raw PCM to prevent cross-contamination.
• 48 kHz resampling: The Rubato-based resampler in AudioCapture normalizes all input to a common sample rate.
• Lock-free synchronization: The AudioMixerRingBuffer aligns streams using fixed 50 ms windows with zero-padding for lagging sources.
• Professional mixing: Soft-scaling in ProfessionalAudioMixer prevents clipping while preserving dynamic range.
• Efficient transcription: The VAD processor filters non-speech audio, sending only relevant segments to Whisper.
• Robust shutdown: Flush markers ensure complete buffer drainage before recording termination.

Frequently Asked Questions

How does Meetily handle different sample rates between microphone and system audio?

According to the Meetily source code, each AudioCapture instance maintains a persistent Rubato resampler that converts arbitrary input rates to the pipeline's target 48 kHz. This occurs in process_audio_data (lines 1007–1052), where variable-size input chunks are resampled without energy loss before entering the synchronization buffer.

What happens if one audio stream lags behind the other during recording?

The extract_window method (lines 92–141) handles latency by zero-padding the shorter buffer with silence. This ensures the mixer always receives temporally aligned windows of equal length, preventing the phasing or echo artifacts that would result from repeating samples or waiting for the slower stream.

Why does Meetily use zero-padding instead of sample repetition for synchronization?

Zero-padding inserts literal silence (0.0 amplitude) for missing samples, whereas sample repetition would hold the last known value. According to the implementation in frontend/src-tauri/src/audio/pipeline.rs, zero-padding avoids audible clicks or droning artifacts when one source experiences jitter or temporary dropout, maintaining professional audio quality.

How does the pipeline ensure the transcription engine receives only speech segments?

The ContinuousVadProcessor runs Voice Activity Detection on the mixed audio stream in process_audio (lines 335–367). It segments the audio and forwards only chunks containing speech—specifically those longer than 800 samples (approximately 50 ms at 16 kHz)—to the Whisper transcription sender, filtering out silence, background noise, and system sounds.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}