How the AudioCapturePipeline Processes Incoming Audio Samples Asynchronously in FluidVoice

The AudioCapturePipeline processes incoming audio samples asynchronously by receiving raw buffers on real-time threads, locking mutable state with NSLock, down-mixing to mono, resampling to 16 kHz, and feeding results into a ThreadSafeAudioBuffer without ever blocking the main actor.

FluidVoice's real-time dictation engine relies on the AudioCapturePipeline class to bridge low-level audio capture APIs with asynchronous speech recognition. This Sendable class, implemented in ASRService.swift, receives audio from both Core Audio and AVAudioEngine on dedicated real-time threads, performing format conversion and thread-safe buffering to supply ASR managers with standardized 16 kHz mono Float samples.

Pipeline Architecture and Thread Safety

Sendable Design and Lock Protection

The AudioCapturePipeline is declared as @unchecked Sendable and maintains exclusive access to mutable state through an internal NSLock. According to the FluidVoice source code, this design allows the pipeline to receive audio on high-priority real-time threads while remaining safely accessible from Swift concurrency contexts without requiring @MainActor isolation.

All mutable state—including the recording flag, session timestamps, and audio history—is protected by explicit lock.lock() and lock.unlock() calls. This guarantees thread safety when the low-level Core Audio callback fires at the native hardware rate concurrently with UI-triggered state changes.

Integration with Real-Time Audio Sources

FluidVoice receives raw audio from either Core Audio Capture (optimized for Apple Silicon) or AVAudioEngine (universal fallback). The DirectCoreAudioInput.swift bridge forwards buffers directly to the pipeline's handler methods, ensuring zero-copy handoff from the hardware layer to the processing pipeline.

Instantiating the Pipeline

The pipeline is lazily initialized once per ASRService instance at line 903 of ASRService.swift, retaining a ThreadSafeAudioBuffer that later supplies ASR managers with processed samples:

private lazy var audioCapturePipeline: AudioCapturePipeline = .init(
    audioBuffer: threadSafeAudioBuffer,
    onFirstAudio: { /* ASR session initialization */ },
    onLevel: { /* UI level meter update */ })

This singleton pattern ensures that all audio sources write to the same buffer instance, while the ASR manager consumes data on a separate background async task.

Enabling and Disabling Recording

The setRecordingEnabled(_:sessionID:startHostTime:) method at lines 363-380 manages the recording lifecycle with lock protection:

func setRecordingEnabled(_ enabled: Bool,
                         sessionID: Int = 0,
                         startHostTime: UInt64 = 0) {
    lock.lock()
    if enabled {
        // Initialize new session with precise host time
    } else {
        // Clean up and reset state
    }
    lock.unlock()
}

A per-session ID and host-time stamp are stored atomically under the lock, enabling the pipeline to trim the buffer to exact start/stop boundaries later via markRecordingEnd(atHostTime:).

Receiving Audio from Multiple Sources

The pipeline exposes two entry points at lines 525-540 to accommodate different audio subsystem outputs:

AVAudioEngine Buffer Processing

func handle(buffer: AVAudioPCMBuffer, time: AVAudioTime) { 
    // Convert AVAudioBuffer to raw samples
}

Core Audio Sample Processing

func handle(samples: UnsafePointer<Float>,
            frameCount: Int,
            sampleRate: Double,
            inputHostTime: UInt64,
            inputSampleTime: Int64) { 
    // Process raw interleaved Float buffers
}

Both methods forward data to a common internal routine handleMonoSamples(...), ensuring consistent processing regardless of the audio source.

Audio Processing and Format Conversion

Down-mixing Multi-Channel Audio

When receiving multi-channel audio, the downmixToMono(_:) static method at lines 394-419 uses Accelerate framework routines (vDSP_vadd and vDSP_vsdiv) to mix channels into a single Float-32 array:

private static func downmixToMono(_ buffer: AVAudioPCMBuffer) -> [Float] {
    // vDSP-based mixing and normalization
    // Returns normalized mono samples
}

Resampling and Thread-Safe Buffering

The handleMonoSamples method starting at line 484 implements the critical asynchronous path:

self.lock.lock()
guard self.recordingEnabled else { 
    self.lock.unlock()
    return 
}

// Resample from input sample rate to 16 kHz
let resampled = resampleTo16kHz(samples, from: sampleRate)

// Append to thread-safe FIFO
self.audioBuffer.append(resampled, sampleRate: 16000.0)
self.lock.unlock()

This implementation:

1. Locks the critical section to prevent race conditions with the UI thread
2. Validates recording state (preventing stray callbacks from corrupting finished sessions)
3. Resamples using a linear-interpolation state machine (located after line 480 in the same file)
4. Feeds the converted Float buffer into ThreadSafeAudioBuffer

Real-Time Level Metering

While processing each chunk, the pipeline computes an RMS-based level at lines 390-415, applying smoothing and reporting results through the onLevel closure:

let rms = sqrt(sum / Float(samples.count))
let dbLevel = 20 * log10(max(rms, 1e-10))
let normalized = max(0, min(1, (dbLevel + 55) / 55))
self.onLevel(applySmoothingAndThreshold(normalized))

The UI consumes these normalized levels to drive waveform visualizations without touching the ASR service or main actor.

Completing Recording Sessions

When dictation stops, finishRecording() at lines 441-445 clears the recording flag and resets the level meter:

func finishRecording() {
    setRecordingEnabled(false)
    onLevel(0.0)
}

Because the pipeline is Sendable and uses explicit locking rather than actor isolation, these state changes synchronize safely with concurrent audio callbacks.

Summary

• The AudioCapturePipeline in ASRService.swift is a Sendable class that uses NSLock to protect mutable state from real-time audio threads.
• It receives audio from both Core Audio and AVAudioEngine through handle methods at lines 525-540, supporting asynchronous input from multiple sources.
• Raw samples are down-mixed to mono using vDSP (lines 394-419) and resampled to 16 kHz within a locked critical section.
• Processed data flows into ThreadSafeAudioBuffer without blocking the main actor, enabling concurrent ASR processing.
• RMS level metering (lines 390-415) provides UI feedback while the pipeline manages session boundaries via setRecordingEnabled (lines 363-380).

Frequently Asked Questions

How does AudioCapturePipeline maintain thread safety without using actors?

The class adopts the @unchecked Sendable protocol and manually synchronizes access using an internal NSLock instance. All mutable state modifications—including enabling recording, appending samples, and updating session IDs—are wrapped in lock.lock() and lock.unlock() pairs. This approach allows the pipeline to receive audio on high-priority real-time threads that cannot be isolated to the main actor or other Swift actors.

What audio formats does the pipeline output for ASR processing?

The pipeline converts all incoming audio to 16 kHz sample rate, mono channel, Float-32 format. The downmixToMono method handles multi-channel input using Accelerate framework routines, while an internal resampling state machine converts varying input rates (such as 44.1 kHz or 48 kHz) to the required 16 kHz before appending to the ThreadSafeAudioBuffer.

Can the pipeline handle simultaneous audio from multiple sources?

While the pipeline supports two entry points—handle(buffer:time:) for AVAudioEngine and handle(samples:frameCount:...) for Core Audio—it processes them sequentially through a common handleMonoSamples routine protected by the same NSLock. Only one audio source should be active at a time per pipeline instance, though the design allows rapid switching between sources by calling setRecordingEnabled(false) before changing inputs.

How does the pipeline signal the ASR manager when audio is ready?

The pipeline initializes with an onFirstAudio callback closure (configured at line 903) that triggers when the first samples arrive, and it writes processed data into a shared ThreadSafeAudioBuffer. The ASR manager asynchronously consumes data from this buffer via consumeAll() or similar methods, decoupling the real-time capture thread from the transcription task.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}