# How the AudioCapturePipeline Processes Incoming Audio Samples Asynchronously in FluidVoice

> FluidVoice's AudioCapturePipeline processes audio samples asynchronously, avoiding main thread blockages. Learn how it handles real-time audio processing efficiently.

- Repository: [ALTIC/FluidVoice](https://github.com/altic-dev/FluidVoice)
- Tags: internals
- Published: 2026-07-07

---

**The `AudioCapturePipeline` processes incoming audio samples asynchronously by receiving raw buffers on real-time threads, locking mutable state with `NSLock`, down-mixing to mono, resampling to 16 kHz, and feeding results into a `ThreadSafeAudioBuffer` without ever blocking the main actor.**

FluidVoice's real-time dictation engine relies on the `AudioCapturePipeline` class to bridge low-level audio capture APIs with asynchronous speech recognition. This `Sendable` class, implemented in [`ASRService.swift`](https://github.com/altic-dev/FluidVoice/blob/main/ASRService.swift), receives audio from both Core Audio and AVAudioEngine on dedicated real-time threads, performing format conversion and thread-safe buffering to supply ASR managers with standardized 16 kHz mono Float samples.

## Pipeline Architecture and Thread Safety

### Sendable Design and Lock Protection

The `AudioCapturePipeline` is declared as `@unchecked Sendable` and maintains exclusive access to mutable state through an internal `NSLock`. According to the FluidVoice source code, this design allows the pipeline to receive audio on high-priority real-time threads while remaining safely accessible from Swift concurrency contexts without requiring `@MainActor` isolation.

All mutable state—including the recording flag, session timestamps, and audio history—is protected by explicit `lock.lock()` and `lock.unlock()` calls. This guarantees thread safety when the low-level Core Audio callback fires at the native hardware rate concurrently with UI-triggered state changes.

### Integration with Real-Time Audio Sources

FluidVoice receives raw audio from either **Core Audio Capture** (optimized for Apple Silicon) or **AVAudioEngine** (universal fallback). The [`DirectCoreAudioInput.swift`](https://github.com/altic-dev/FluidVoice/blob/main/DirectCoreAudioInput.swift) bridge forwards buffers directly to the pipeline's handler methods, ensuring zero-copy handoff from the hardware layer to the processing pipeline.

## Instantiating the Pipeline

The pipeline is lazily initialized once per `ASRService` instance at line 903 of [`ASRService.swift`](https://github.com/altic-dev/FluidVoice/blob/main/ASRService.swift), retaining a `ThreadSafeAudioBuffer` that later supplies ASR managers with processed samples:

```swift
private lazy var audioCapturePipeline: AudioCapturePipeline = .init(
    audioBuffer: threadSafeAudioBuffer,
    onFirstAudio: { /* ASR session initialization */ },
    onLevel: { /* UI level meter update */ })

```

This singleton pattern ensures that all audio sources write to the same buffer instance, while the ASR manager consumes data on a separate background async task.

## Enabling and Disabling Recording

The `setRecordingEnabled(_:sessionID:startHostTime:)` method at lines 363-380 manages the recording lifecycle with lock protection:

```swift
func setRecordingEnabled(_ enabled: Bool,
                         sessionID: Int = 0,
                         startHostTime: UInt64 = 0) {
    lock.lock()
    if enabled {
        // Initialize new session with precise host time
    } else {
        // Clean up and reset state
    }
    lock.unlock()
}

```

A per-session ID and host-time stamp are stored atomically under the lock, enabling the pipeline to trim the buffer to exact start/stop boundaries later via `markRecordingEnd(atHostTime:)`.

## Receiving Audio from Multiple Sources

The pipeline exposes two entry points at lines 525-540 to accommodate different audio subsystem outputs:

### AVAudioEngine Buffer Processing

```swift
func handle(buffer: AVAudioPCMBuffer, time: AVAudioTime) { 
    // Convert AVAudioBuffer to raw samples
}

```

### Core Audio Sample Processing

```swift
func handle(samples: UnsafePointer<Float>,
            frameCount: Int,
            sampleRate: Double,
            inputHostTime: UInt64,
            inputSampleTime: Int64) { 
    // Process raw interleaved Float buffers
}

```

Both methods forward data to a common internal routine `handleMonoSamples(...)`, ensuring consistent processing regardless of the audio source.

## Audio Processing and Format Conversion

### Down-mixing Multi-Channel Audio

When receiving multi-channel audio, the `downmixToMono(_:)` static method at lines 394-419 uses Accelerate framework routines (`vDSP_vadd` and `vDSP_vsdiv`) to mix channels into a single Float-32 array:

```swift
private static func downmixToMono(_ buffer: AVAudioPCMBuffer) -> [Float] {
    // vDSP-based mixing and normalization
    // Returns normalized mono samples
}

```

### Resampling and Thread-Safe Buffering

The `handleMonoSamples` method starting at line 484 implements the critical asynchronous path:

```swift
self.lock.lock()
guard self.recordingEnabled else { 
    self.lock.unlock()
    return 
}

// Resample from input sample rate to 16 kHz
let resampled = resampleTo16kHz(samples, from: sampleRate)

// Append to thread-safe FIFO
self.audioBuffer.append(resampled, sampleRate: 16000.0)
self.lock.unlock()

```

This implementation:
1. **Locks** the critical section to prevent race conditions with the UI thread
2. **Validates** recording state (preventing stray callbacks from corrupting finished sessions)
3. **Resamples** using a linear-interpolation state machine (located after line 480 in the same file)
4. **Feeds** the converted Float buffer into `ThreadSafeAudioBuffer`

## Real-Time Level Metering

While processing each chunk, the pipeline computes an RMS-based level at lines 390-415, applying smoothing and reporting results through the `onLevel` closure:

```swift
let rms = sqrt(sum / Float(samples.count))
let dbLevel = 20 * log10(max(rms, 1e-10))
let normalized = max(0, min(1, (dbLevel + 55) / 55))
self.onLevel(applySmoothingAndThreshold(normalized))

```

The UI consumes these normalized levels to drive waveform visualizations without touching the ASR service or main actor.

## Completing Recording Sessions

When dictation stops, `finishRecording()` at lines 441-445 clears the recording flag and resets the level meter:

```swift
func finishRecording() {
    setRecordingEnabled(false)
    onLevel(0.0)
}

```

Because the pipeline is `Sendable` and uses explicit locking rather than actor isolation, these state changes synchronize safely with concurrent audio callbacks.

## Summary

- **The `AudioCapturePipeline` in [`ASRService.swift`](https://github.com/altic-dev/FluidVoice/blob/main/ASRService.swift) is a `Sendable` class that uses `NSLock` to protect mutable state from real-time audio threads.**
- **It receives audio from both Core Audio and AVAudioEngine through `handle` methods at lines 525-540, supporting asynchronous input from multiple sources.**
- **Raw samples are down-mixed to mono using `vDSP` (lines 394-419) and resampled to 16 kHz within a locked critical section.**
- **Processed data flows into `ThreadSafeAudioBuffer` without blocking the main actor, enabling concurrent ASR processing.**
- **RMS level metering (lines 390-415) provides UI feedback while the pipeline manages session boundaries via `setRecordingEnabled` (lines 363-380).**

## Frequently Asked Questions

### How does AudioCapturePipeline maintain thread safety without using actors?

The class adopts the `@unchecked Sendable` protocol and manually synchronizes access using an internal `NSLock` instance. All mutable state modifications—including enabling recording, appending samples, and updating session IDs—are wrapped in `lock.lock()` and `lock.unlock()` pairs. This approach allows the pipeline to receive audio on high-priority real-time threads that cannot be isolated to the main actor or other Swift actors.

### What audio formats does the pipeline output for ASR processing?

The pipeline converts all incoming audio to **16 kHz sample rate**, **mono channel**, **Float-32** format. The `downmixToMono` method handles multi-channel input using Accelerate framework routines, while an internal resampling state machine converts varying input rates (such as 44.1 kHz or 48 kHz) to the required 16 kHz before appending to the `ThreadSafeAudioBuffer`.

### Can the pipeline handle simultaneous audio from multiple sources?

While the pipeline supports two entry points—`handle(buffer:time:)` for AVAudioEngine and `handle(samples:frameCount:...)` for Core Audio—it processes them sequentially through a common `handleMonoSamples` routine protected by the same `NSLock`. Only one audio source should be active at a time per pipeline instance, though the design allows rapid switching between sources by calling `setRecordingEnabled(false)` before changing inputs.

### How does the pipeline signal the ASR manager when audio is ready?

The pipeline initializes with an `onFirstAudio` callback closure (configured at line 903) that triggers when the first samples arrive, and it writes processed data into a shared `ThreadSafeAudioBuffer`. The ASR manager asynchronously consumes data from this buffer via `consumeAll()` or similar methods, decoupling the real-time capture thread from the transcription task.