How WhisperProvider in FluidVoice Works: On-Device Speech Recognition with whisper.cpp

The WhisperProvider in FluidVoice is an on-device speech recognition backend for Intel Macs that wraps whisper.cpp via SwiftWhisper, supporting six Whisper models from Tiny to Large with automatic downloading and memory-guarded loading.

The WhisperProvider serves as the concrete implementation of the TranscriptionProvider protocol in the FluidVoice macOS application (altic-dev/FluidVoice). It enables fully offline speech-to-text transcription by leveraging the open-source whisper.cpp engine through a Swift wrapper, making it ideal for privacy-conscious users on x86-64 Intel Macs.

Architecture and Protocol Conformance

WhisperProvider is built specifically for Intel (x86-64) Macs as the on-device alternative to cloud-based speech recognition. It conforms to the TranscriptionProvider protocol defined in the FluidVoice codebase, ensuring interchangeable usage with other transcription backends like Apple Speech or Parakeet.

The provider acts as a Swift bridge to SwiftWhisper, which wraps the C++ implementation of OpenAI's Whisper model. This architecture keeps all audio processing local to the machine, requiring no network connectivity after the initial model download.

The Five-Stage Transcription Pipeline

The WhisperProvider manages the entire lifecycle of speech recognition through five distinct operational stages defined in Sources/Fluid/Services/WhisperProvider.swift.

Initialization and Model Configuration

The provider initializes with an optional custom model directory, a URLSession for network operations, and an optional modelOverride that can temporarily force a specific model variant.

According to WhisperProvider.swift (lines 26-30), the init method signature is:

init(modelDirectory: URL? = nil, session: URLSession = .shared, modelOverride: SpeechModel? = nil)

The modelName property dynamically determines which model file to load based on the current SettingsStore selection, defaulting to ggml-base.bin when no override is present (lines 32-41).

Preparation and Memory Guarding

The prepare() method handles model caching, validation, and loading. It performs three critical tasks:

• Cache validation: Creates the cache folder and verifies existing files through size checks
• Conditional downloading: Fetches the model from Hugging Face (https://huggingface.co/ggerganov/whisper.cpp/resolve/main/<model>.bin) if missing
• Memory safety: Checks system RAM availability against the model's requiredMemoryGB before loading

According to the source code in WhisperProvider.swift (lines 77-132), the provider queries available memory via mach APIs in availableMemoryGB() and aborts with a descriptive error if the system cannot satisfy the requirement.

Audio Transcription

Once prepared, the transcribe() method accepts an array of Float PCM samples at 16 kHz. The implementation includes a sanity check that rejects buffers smaller than 16,000 samples (approximately one second of audio).

Because whisper.cpp does not expose confidence metrics, the provider always returns a confidence value of 1.0 in the ASRTranscriptionResult. The method concatenates segment texts from the underlying Whisper object's transcribe(audioFrames:) call and trims whitespace before returning.

Cache Management

The provider exposes modelsExistOnDisk() to verify local availability and clearCache() to purge downloaded files, both defined in WhisperProvider.swift (lines 124-143). These methods enable the UI to show storage status and provide cleanup functionality.

Supported Whisper Models and Specifications

FluidVoice exposes six Whisper variants through the SettingsStore.SpeechModel enum, defined in Sources/Fluid/Persistence/SettingsStore.swift. The provider automatically maps these enum cases to specific GGML binary files.

Enum Case	Model File	Size	Description
whisperTiny	ggml-tiny.bin	~75 MB	Fastest inference, lowest accuracy
whisperBase	ggml-base.bin	~142 MB	Default choice balancing speed and accuracy
whisperSmall	ggml-small.bin	~466 MB	Improved accuracy with moderate speed trade-off
whisperMedium	ggml-medium.bin	~1.5 GB	High accuracy requiring significant RAM
whisperLargeTurbo	ggml-large-v3-turbo.bin	~1.5 GB	Higher quality optimized for speed (currently disabled in UI)
whisperLarge	ggml-large-v3.bin	~2.9 GB	Best accuracy, largest resource requirements

The enum definitions and human-readable titles reside in SettingsStore.swift (lines 56-85). When users switch models via the UI, the provider downloads the new binary on the next prepare() invocation if it is not already cached locally.

Integration with FluidVoice Services

The WhisperProvider integrates into the broader application through two primary touchpoints:

• ASRService lazily instantiates the provider via getWhisperProvider() when the selected speech model matches a Whisper variant (ASRService.swift, lines 152-218).
• AISettingsView renders the model selection interface using the SpeechModel enum, persisting user choice through SettingsStore.shared.selectedSpeechModel (AISettingsView+SpeechRecognition.swift).

Practical Implementation Examples

The following Swift patterns demonstrate typical WhisperProvider usage within the FluidVoice architecture.

Initialization and Preparation

import Fluid

let provider = WhisperProvider()

// Prepare with progress tracking
Task {
    do {
        try await provider.prepare { progress in
            // Handle ModelPreparationProgress enum cases
            print("Preparation status: \(progress)")
        }
    } catch {
        print("Preparation failed: \(error)")
    }
}

Transcribing Audio

// Assuming 'samples' is a [Float] array of 16kHz PCM data
func processAudio(_ samples: [Float]) async {
    do {
        let result = try await provider.transcribe(samples)
        print("Transcribed: \(result.text)")
    } catch {
        print("Transcription error: \(error)")
    }
}

Switching Models

// Change model via SettingsStore (triggers download on next prepare)
SettingsStore.shared.selectedSpeechModel = .whisperSmall

Summary

• WhisperProvider implements the TranscriptionProvider protocol for offline speech recognition on Intel Macs using whisper.cpp via SwiftWhisper.
• The five-stage pipeline includes initialization, model selection, preparation with memory guarding, transcription, and cache management.
• Six models are supported, ranging from Tiny (~75 MB) to Large (~2.9 GB), with Base set as the default.
• Memory safety is enforced by comparing requiredMemoryGB against system-available RAM before loading large models.
• Models are downloaded dynamically from Hugging Face and cached locally, with no network requirements during transcription.

Frequently Asked Questions

What is the default Whisper model in FluidVoice?

The default model is Whisper Base (ggml-base.bin), approximately 142 MB in size. This is determined by the modelName property in WhisperProvider.swift, which falls back to Base when no specific model is selected in SettingsStore.

Does WhisperProvider work on Apple Silicon Macs?

The provider is specifically designed for Intel (x86-64) Macs. The FluidVoice codebase uses different transcription providers for Apple Silicon chips, making WhisperProvider the dedicated backend for Intel-based hardware.

How does FluidVoice handle insufficient RAM for large models?

Before loading any model, the provider calls availableMemoryGB() using mach APIs to query system memory. It compares this against the model's requiredMemoryGB property defined in the SpeechModel enum. If available RAM is insufficient, the prepare() method throws an error, preventing the application from crashing due to memory pressure.

Where are the Whisper models downloaded from?

Models are downloaded from the official Hugging Face repository at https://huggingface.co/ggerganov/whisper.cpp/resolve/main/<model-file>.bin. The download logic includes retry mechanisms with exponential backoff and reports progress through the ModelPreparationProgress enum callback.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}