How FluidVoice Implements the TranscriptionProvider Protocol for Pluggable Speech Recognition

FluidVoice abstracts all speech-to-text functionality behind a single Swift protocol called TranscriptionProvider, enabling runtime-swappable backends ranging from Whisper GGUF models to Apple's native speech APIs.

The TranscriptionProvider protocol in the FluidVoice repository (altic-dev/FluidVoice) defines a clean contract that any speech recognition engine must satisfy. Located at Sources/Fluid/Services/TranscriptionProvider.swift, this protocol enables the app to switch between seven distinct transcription backends without modifying UI code.

Core Protocol Requirements in TranscriptionProvider.swift

The protocol mandates four categories of functionality that every provider must implement:

Identity and Availability Reporting

Each provider exposes runtime metadata through three computed properties:

• name – A human-readable string identifying the backend (e.g., "Whisper", "Apple Speech")
• isAvailable – Whether the provider can execute on the current hardware and OS version
• isReady – Whether the provider's models are loaded and transcription can begin immediately

These properties allow ASRService to filter unavailable providers and display appropriate options in the settings UI.

Model Lifecycle Management

Providers handle their own asset preparation through:

func prepare(progressHandler: ((ModelPreparationProgress) -> Void)?) async throws
func clearCache() async
func modelsExistOnDisk() -> Bool

The prepare(progressHandler:) method downloads, verifies, and loads model files. WhisperProvider uses this to fetch GGUF files from HuggingFace, reporting progress through the ModelPreparationProgress struct defined in the same file. Cache inspection and cleanup enable the app to manage disk usage without provider-specific logic.

Transcription Operations

The core transcription method signature:

func transcribe(_ samples: [Float]) async throws -> ASRTranscriptionResult

This accepts 16 kHz mono PCM audio samples and returns structured results. The protocol provides default implementations for four specialized variants:

• transcribeStreaming(_:) – For real-time dictation with partial results
• transcribeFinal(_:) – For optimized final-pass transcription
• transcribeDictionaryTraining(_:) – For vocabulary customization workflows
• transcribeFile(at:) – For direct file-level processing

Default fallbacks route these to the base transcribe(_:) method, but providers can override for optimized paths.

Behavioral Configuration Flags

Two boolean flags guide ASRService orchestration:

• shouldClearCacheAfterCancellation – Whether to purge model files when transcription is interrupted
• prefersNativeFileTranscription – Whether to use transcribeFile(at:) instead of loading audio into memory

Concrete Provider Implementations

FluidVoice ships seven providers conforming to TranscriptionProvider, each targeting a specific backend architecture:

WhisperProvider

Sources/Fluid/Services/WhisperProvider.swift

Wraps transcribe.cpp for local Whisper inference using GGUF quantized models. Key characteristics:

• Downloads model files from HuggingFace Hub with integrity verification
• Automatically selects Metal GPU on Apple Silicon, CPU fallback on Intel
• Performs memory availability checks before model loading
• Reports granular preparation progress including download, decompression, and initialization phases

FluidAudioProvider

Sources/Fluid/Services/FluidAudioProvider.swift

Integrates the proprietary FluidAudio library optimized for Apple Silicon. Focuses on low-latency streaming transcription with minimal CPU overhead.

ParakeetRealtimeProvider

Sources/Fluid/Services/ParakeetRealtimeProvider.swift

Implements true streaming transcription through the Parakeet end-of-utterance (EOU) pipeline. Suitable for continuous dictation scenarios requiring immediate partial results.

AppleSpeechProvider

Sources/Fluid/Services/AppleSpeechProvider.swift

Wraps Apple's public SFSpeechRecognizer API. Handles permission requests, locale selection, and network-dependent recognition automatically.

AppleSpeechAnalyzerProvider

Sources/Fluid/Services/AppleSpeechAnalyzerProvider.swift

Targets SpeechAnalyzer, a newer API available starting macOS 26. Provides improved accuracy over SFSpeechRecognizer for supported systems.

ExternalCoreMLTranscriptionProvider

Sources/Fluid/Services/ExternalCoreMLTranscriptionProvider.swift

Loads user-supplied or bundled Core ML models. Supports native file-wise transcription when prefersNativeFileTranscription returns true, avoiding unnecessary audio decoding.

NemotronProvider

Sources/Fluid/Services/NemotronProvider.swift

Wraps the Nemotron large language model inference engine for high-quality transcription. Includes optional vocabulary rescoring for domain-specific accuracy improvements.

Runtime Provider Selection via ASRService

The ASRService class at Sources/Fluid/Services/ASRService.swift orchestrates provider instantiation without hardcoding specific implementations:

// ASRService constructs providers based on SettingsStore selection
let asr = ASRService()

// Access the currently selected provider
let provider = asr.transcriptionProvider

// Prepare with progress UI feedback
await provider.prepare { progress in
    print("Phase: \(progress.phase), Progress: \(progress.fractionCompleted ?? 0)")
}

// Execute transcription
let samples: [Float] = // 16 kHz mono PCM buffer
let result = try await provider.transcribe(samples)

Provider selection persists through SettingsStore (Sources/Fluid/Persistence/SettingsStore.swift), which serializes the user's choice to UserDefaults. When the user changes models or switches providers, ASRService.resetTranscriptionProvider() (referenced in WelcomeView.swift) disposes the existing instance and constructs a replacement.

Default Implementation Strategy

The protocol extension in TranscriptionProvider.swift supplies no-op implementations for optional behaviors:

// Providers only override what they optimize
extension TranscriptionProvider {
    func transcribeStreaming(_ samples: [Float]) async throws -> ASRTranscriptionResult {
        try await transcribe(samples)  // Default: treat as regular transcription
    }
    
    var shouldClearCacheAfterCancellation: Bool { false }
    var prefersNativeFileTranscription: Bool { false }
}

This design minimizes boilerplate—AppleSpeechProvider implements only essential methods, while WhisperProvider overrides preparation and caching for its GGUF lifecycle.

Summary

• TranscriptionProvider at Sources/Fluid/Services/TranscriptionProvider.swift defines the complete contract for speech recognition backends
• Seven concrete implementations cover local inference (Whisper, Nemotron), Apple APIs (SFSpeechRecognizer, SpeechAnalyzer), proprietary libraries (FluidAudio, Parakeet), and custom Core ML models
• ASRService manages provider lifecycle and exposes a unified API to SwiftUI views
• Default protocol implementations reduce provider boilerplate while preserving optimization opportunities
• Runtime switching through SettingsStore and resetTranscriptionProvider() enables user-controlled backend selection without code changes

Frequently Asked Questions

What audio format does TranscriptionProvider expect?

All transcribe(_:) implementations receive [Float] arrays containing 16 kHz mono PCM samples. Providers that need different formats—such as ExternalCoreMLTranscriptionProvider with prefersNativeFileTranscription—can opt into file-level processing to handle conversion internally.

Can I add a custom transcription backend to FluidVoice?

Yes. Create a new Swift class conforming to TranscriptionProvider, implement the required properties and methods, then register it in ASRService.swift. The default implementations in the protocol extension handle streaming and final-pass fallbacks automatically.

How does FluidVoice handle model downloads for WhisperProvider?

WhisperProvider.prepare(progressHandler:) downloads GGUF files from HuggingFace using URLSession with background transfer support. Progress updates flow through ModelPreparationProgress, which includes download fraction, decompression status, and Metal shader compilation phases. Failed downloads automatically retry with exponential backoff.

Why does ASRService exist instead of calling providers directly?

ASRService decouples UI code from provider-specific initialization logic. It reads SettingsStore preferences, constructs the appropriate provider instance, manages memory pressure notifications, and coordinates model preparation across view lifecycle changes. This centralization prevents view controllers from holding stateful provider references.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}