How FluidVoice Interacts with macOS System Services for Dictation: Speech, CoreAudio, and Accessibility APIs

FluidVoice integrates three core macOS frameworks—Speech (SFSpeechRecognizer), CoreAudio (AudioObject APIs), and Accessibility (AXUIElement)—to capture audio, convert speech to text, and inject transcriptions into any foreground application.

FluidVoice is an open-source macOS dictation application that leverages native system services to deliver seamless voice-to-text functionality. By combining on-device speech recognition with low-level audio hardware management and accessibility-based text injection, the app provides a universal dictation layer that works across native apps, Electron editors, and web views. This article examines the specific implementation details found in the altic-dev/FluidVoice repository, revealing exactly how the Swift codebase interacts with macOS system services to handle the complete dictation pipeline.

Speech Recognition via SFSpeechRecognizer

FluidVoice's speech-to-text capabilities are encapsulated in Sources/Fluid/Services/AppleSpeechProvider.swift, which wraps Apple's SFSpeechRecognizer framework. The implementation handles authorization, locale-specific model loading, and real-time audio buffer processing.

Authorization and Locale Configuration

Before capturing audio, the provider requests user permission through the standard macOS speech recognition dialog. The prepare(progressHandler:) method initiates this flow by calling requestAuthorization(), wrapped in an async continuation for Swift concurrency compatibility.

let status = await self.requestAuthorization()

The recognizer rebuilds whenever the user changes languages via SettingsStore.shared.selectedAppleSpeechLocale, ensuring the correct phonetic model loads for the selected locale.

let locale = SettingsStore.shared.selectedAppleSpeechLocale
self.recognizer = SFSpeechRecognizer(locale: locale)

Audio Buffer Processing and Recognition

Raw microphone samples arrive as 16 kHz Float32 arrays. The createPCMBuffer(from:) method wraps these in AVAudioPCMBuffer instances, which are appended to a SFSpeechAudioBufferRecognitionRequest. The provider specifically awaits final transcriptions to avoid partial-text noise, returning an ASRTranscriptionResult for downstream AI enhancement or direct insertion.

let request = SFSpeechAudioBufferRecognitionRequest()
request.append(buffer)
request.endAudio()

The final transcript is handed off to FluidVoice’s AI enhancement pipeline (local “Fluid Intelligence” or cloud providers) before being inserted into the target application.

Audio Hardware Management with CoreAudio

To support multiple microphones and hardware configurations, Sources/Fluid/Services/AudioDeviceService.swift interfaces directly with CoreAudio's C-based AudioObject APIs rather than high-level AVFoundation abstractions.

Device Enumeration and Selection

The service queries kAudioHardwarePropertyDevices to enumerate available audio hardware, extracting device names, UIDs, and input channel capabilities through listAllDevices(). For default input selection, getDefaultInputDevice() and setDefaultInputDevice(uid:) wrap kAudioHardwarePropertyDefaultInputDevice, allowing users to switch microphones programmatically.

let devices = AudioDeviceService.listInputDevices()
for d in devices {
    print("🔊 \(d.name) – UID: \(d.uid)")
}

Switching the default input device works via:

let targetUID = "AppleHDAEngineInput:1B,0,1,0:0"
if AudioDeviceService.setDefaultInputDevice(uid: targetUID) {
    print("✅ Default input switched")
}

Hardware Change Monitoring

An internal AudioHardwareObserver class registers property listeners via AudioObjectAddPropertyListenerBlock to detect device plug/unplug events. When hardware changes occur, the observer increments a published changeTick property, enabling SwiftUI views to reactively update device dropdown menus without polling.

Text Injection via macOS Accessibility APIs

Once transcription completes, Sources/Fluid/Services/TypingService.swift handles the critical task of inserting text into the target application. The service implements a tiered fallback strategy to maximize compatibility across different app architectures.

The Tiered Insertion Pipeline

The primary method typeTextInstantly(_:) executes a priority-ordered insertion strategy to interact with macOS system services for text delivery:

1. Direct Accessibility Insertion – insertTextViaAccessibility(_:) queries kAXFocusedUIElementAttribute to capture the current PID, then attempts text insertion via kAXValueAttribute, kAXSelectedTextRangeAttribute, or kAXSelectedTextAttribute depending on whether the target is an AXTextField, AXTextArea, or similar element.
2. CGEvent Unicode Injection – If Accessibility fails, insertTextBulkInstant(_:targetPID:) splits UTF-16 strings into 200-character chunks and posts them as low-level CGEvent keypresses directly to the target process.
3. Clipboard Paste Fallback – The service temporarily writes to the system clipboard, sends a Cmd-V event (or AppleScript menu command), then restores the previous clipboard contents.
4. Character-by-Character Typing – As a last resort for stubborn applications, the service types individual characters with small delays to ensure text appears correctly.

Focus Capture and Debugging

Before insertion, captureSystemFocusedPID() snapshots the focused UI element to handle focus-loss scenarios. Detailed logging controlled by FLUID_TYPING_LOGS or the "Enable Typing Logs" user default helps diagnose permission or focus issues during development.

Permission Requirements and User Onboarding

FluidVoice requires three distinct macOS permissions to function, managed through Sources/Fluid/Views/OnboardingTryoutStepView.swift and runtime checks throughout the codebase.

Speech Recognition Permission

The app requests SFSpeechRecognizer authorization on first use, presenting the standard macOS system dialog. Without this, the AppleSpeechProvider cannot instantiate recognition tasks.

Accessibility Permission

Text injection requires Accessibility access, verified via AXIsProcessTrusted(). If denied, FluidVoice displays an onboarding screen with a button directing users to System Settings → Privacy & Security → Accessibility, as implemented in the onboarding view controller.

if !AXIsProcessTrusted() {
    // Open the system preference pane for the user
    TypingService.activateApp(pid: 0) // will open System Settings > Accessibility
}

Microphone Access

While handled separately in the audio pipeline via AVAudioSession, microphone permission is verified before CoreAudio stream initialization to prevent silent failures during dictation sessions.

Summary

• Speech Framework: AppleSpeechProvider.swift wraps SFSpeechRecognizer for locale-aware, permission-gated speech-to-text conversion with final-transcription-only results.
• CoreAudio Integration: AudioDeviceService.swift uses low-level AudioObject APIs to enumerate devices, manage default inputs, and react to hardware changes through property listeners.
• Accessibility Injection: TypingService.swift implements a four-tier fallback system (AX APIs → CGEvent → Clipboard → Character typing) to ensure text insertion works across native macOS apps, Electron applications, and terminals.
• Permission Management: The app explicitly handles Speech, Accessibility, and Microphone permissions through both runtime checks and dedicated onboarding UI in OnboardingTryoutStepView.swift.

Frequently Asked Questions

Does FluidVoice use cloud-based or on-device speech recognition?

FluidVoice utilizes SFSpeechRecognizer, which supports both on-device and cloud-backed recognition depending on the user's macOS version and settings. The AppleSpeechProvider class configures the recognizer with specific locales via SettingsStore.shared.selectedAppleSpeechLocale, but the actual processing (local vs. cloud) is handled by the macOS Speech framework itself based on device capabilities and network availability.

How does FluidVoice handle text insertion into applications that block Accessibility APIs?

When insertTextViaAccessibility(_:) fails to locate a valid AXTextField or AXTextArea, FluidVoice automatically falls back to insertTextBulkInstant(_:targetPID:), which posts CGEvent unicode keypresses directly to the target process ID. If that fails, it uses a clipboard paste mechanism with Cmd-V simulation, and finally resorts to character-by-character typing with delays. This tiered approach ensures compatibility with secure or non-standard applications that resist standard accessibility insertion.

Can FluidVoice switch microphone devices during active dictation?

Yes. The AudioDeviceService class provides setDefaultInputDevice(uid:) to switch inputs dynamically using CoreAudio's kAudioHardwarePropertyDefaultInputDevice property. The service monitors hardware changes via AudioObjectAddPropertyListenerBlock in the AudioHardwareObserver inner class, allowing the UI to update immediately when users connect new headsets or audio interfaces without restarting the application.

What permissions are required to run FluidVoice?

FluidVoice requires three macOS permissions: Speech Recognition (for SFSpeechRecognizer access), Accessibility (for AXUIElement text injection via TypingService), and Microphone (for audio capture). The app detects missing permissions using AXIsProcessTrusted() for accessibility and standard authorization checks for speech, directing users to the appropriate System Settings panes through the onboarding interface defined in OnboardingTryoutStepView.swift.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}