How FluidVoice Manages Audio Input for Transcription: A Deep Dive into ASRService

FluidVoice manages audio input for transcription through the ASRService class, which orchestrates microphone permissions, real-time buffering via AudioCapturePipeline, and thread-safe sample delivery to multiple transcription providers.

The open-source FluidVoice repository (altic-dev/FluidVoice) implements a sophisticated speech-to-text pipeline designed for low-latency dictation. Understanding how this Swift-based application captures and processes microphone input reveals best practices for bridging CoreAudio's real-time threads with modern Swift concurrency.

The ASRService Architecture

At the center of FluidVoice's audio management strategy sits the ASRService class defined in Sources/Fluid/Services/ASRService.swift. This singleton-style service owns the entire lifecycle of audio capture, from permission checks to final transcription delivery.

Initialization and Permission Handling

Before capturing audio, ASRService.initialize() (lines 729-734) verifies microphone access and prepares the audio subsystem:

• Queries AVCaptureDevice.authorizationStatus(for: .audio) and stores the result in micStatus
• Registers listeners for audio route changes
• Preloads cached ASR models to minimize latency during recording startup

This initialization ensures the app respects user privacy while maintaining readiness for instant dictation triggers.

Starting the Recording Session

When the user initiates dictation, await asrService.start() (lines 842-870) executes a sequential setup process:

1. Permission validation – Returns immediately if micStatus != .authorized
2. State reset – Clears finalText, audioBuffer, and previous transcription state
3. Session configuration – Calls configureSession() to set the AVAudioSession category and sample rate
4. Engine startup – Lazily instantiates AVAudioEngine and invokes engine.start()
5. Tap installation – setupEngineTap() adds a real-time tap on the input node that streams raw PCM to the audio capture pipeline
6. Streaming activation – If the selected model supports it, startStreamingTranscription() begins feeding samples to the TranscriptionProvider

Real-Time Audio Capture Pipeline

The AudioCapturePipeline (inner class within ASRService.swift, lines 618-629) handles the critical path between CoreAudio's real-time thread and the app's transcription logic. This component operates under NSLock protection to prevent race conditions between audio capture and Swift concurrency.

Format Conversion and Resampling

Raw audio from AVAudioEngine rarely arrives in the format required by modern ASR models. The pipeline's toMono16k method (lines 274-286) performs essential preprocessing:

• Direct copy when the buffer is already 16 kHz mono Float32
• Down-mixing to mono via downmixToMono for multi-channel inputs
• Resampling to 16 kHz via resampleTo16k for high-sample-rate devices

Simultaneously, the pipeline computes audio levels using Accelerate framework's vDSP_svesq (lines 335-363), converting RMS values to normalized decibel readings for UI visualization via audioLevelSubject.

Thread-Safe Buffering

Converted samples land in the ThreadSafeAudioBuffer (Sources/Fluid/Services/ThreadSafeAudioBuffer.swift, lines 5-22), an NSLock-protected wrapper around [Float]. This buffer provides atomic operations:

• append(_:) – Called by the real-time audio thread
• getPrefix(_:) and clear() – Called by the transcription task
• getAll() – Retrieves complete recording for final processing

This architecture eliminates data races while maintaining the throughput necessary for real-time transcription.

Streaming Transcription Flow

Once recording begins, startStreamingTranscription() launches a persistent Task that bridges the buffer to ASR providers. The workflow operates in a tight loop:

1. Extracts pending samples from audioBuffer using getPrefix
2. Invokes transcriptionProvider.transcribeStreaming(_:) with the [Float] array
3. Updates partialTranscription and publishes results to the UI

The TranscriptionProvider protocol (implemented in files like FluidAudioProvider.swift and WhisperProvider.swift) abstracts multiple backends including FluidAudio, Parakeet, Whisper, AppleSpeech, and Nemotron. Provider selection occurs via the computed property at lines 667-685 based on SettingsStore.shared.selectedSpeechModel.

Stopping and Finalizing Transcription

When dictation ends, await asrService.stop() (lines 998-1064) performs graceful teardown:

• Disables the capture pipeline via audioCapturePipeline.setRecordingEnabled(false)
• Stops AVAudioEngine and removes the input tap
• Awaits completion of the streaming task with stopStreamingTimerAndAwait
• Retrieves remaining samples via audioBuffer.getAll(), applies padding for short recordings, and executes final transcription via transcriptionProvider.transcribe(_:)

This ensures no audio samples are lost between the last streaming chunk and the recording termination.

UI Integration and Audio Visualization

FluidVoice's interface components consume audio levels through audioLevelPublisher, a Combine publisher exposed by ASRService. Views like BottomOverlayView.swift and CommandModeView.swift subscribe to these normalized CGFloat values (ranging 0.0 to 1.0) to drive animated waveform visualizations that provide immediate feedback during speech capture.

import Fluid

// Initialize the transcription service
let asr = ASRService()
await asr.initialize()

// Begin recording when user activates dictation
Task {
    await asr.start()
}

// Drive UI waveform with real-time audio levels
let cancellable = asr.audioLevelPublisher
    .sink { level in
        // level is CGFloat [0,1] representing normalized dB
        waveformView.updateAmplitude(level)
    }

// Stop and retrieve final text
Task {
    let transcription = await asr.stop()
    textEditor.insert(transcription)
}

Summary

• ASRService (ASRService.swift) centralizes microphone permission handling, engine configuration, and transcription orchestration
• AudioCapturePipeline converts incoming AVAudioPCMBuffer to 16 kHz mono Float32 while computing RMS levels for visualization
• ThreadSafeAudioBuffer provides lock-protected storage shared between real-time audio threads and Swift concurrency transcription tasks
• Multiple TranscriptionProvider implementations (Whisper, FluidAudio, etc.) consume raw [Float] samples through a unified streaming interface
• The system uses AVAudioEngine input taps protected by NSLock to safely bridge CoreAudio callbacks with async/await transcription logic

Frequently Asked Questions

How does FluidVoice handle microphone permissions?

FluidVoice queries AVCaptureDevice.authorizationStatus(for: .audio) during ASRService.initialize() (lines 729-734) and stores the result in the micStatus property. The start() method guards against unauthorized access, ensuring the app complies with iOS privacy requirements before activating AVAudioEngine.

What audio format does FluidVoice use for transcription?

The system standardizes all input to 16 kHz mono Float32 samples. The AudioCapturePipeline.toMono16k method (lines 274-286) automatically down-mixes multi-channel audio and resamples higher sample rates, ensuring compatibility with ASR providers while minimizing processing overhead.

How does FluidVoice prevent audio data loss during transcription?

The ThreadSafeAudioBuffer class wraps a [Float] array with an NSLock to synchronize access between the real-time CoreAudio tap (which appends samples) and the transcription Task (which reads and clears chunks). This lock-based approach prevents race conditions without blocking the audio thread for significant durations.

Can FluidVoice transcribe audio in real-time while recording?

Yes, when using models that support streaming (checked via SettingsStore.shared.selectedSpeechModel.supportsStreaming), ASRService launches a background task via startStreamingTranscription(). This task repeatedly pulls samples from the thread-safe buffer and pushes them to the provider's transcribeStreaming(_:) method, updating partial results on the main actor for immediate UI feedback.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}