# How WhisperProvider in FluidVoice Works: On-Device Speech Recognition with whisper.cpp

> Explore how FluidVoice's WhisperProvider enables on-device speech recognition for Intel Macs using whisper.cpp. Discover support for six Whisper models with automatic downloads and safe loading.

- Repository: [ALTIC/FluidVoice](https://github.com/altic-dev/FluidVoice)
- Tags: internals
- Published: 2026-07-07

---

**The WhisperProvider in FluidVoice is an on-device speech recognition backend for Intel Macs that wraps whisper.cpp via SwiftWhisper, supporting six Whisper models from Tiny to Large with automatic downloading and memory-guarded loading.**

The WhisperProvider serves as the concrete implementation of the `TranscriptionProvider` protocol in the FluidVoice macOS application (altic-dev/FluidVoice). It enables fully offline speech-to-text transcription by leveraging the open-source whisper.cpp engine through a Swift wrapper, making it ideal for privacy-conscious users on x86-64 Intel Macs.

## Architecture and Protocol Conformance

WhisperProvider is built specifically for Intel (x86-64) Macs as the on-device alternative to cloud-based speech recognition. It conforms to the `TranscriptionProvider` protocol defined in the FluidVoice codebase, ensuring interchangeable usage with other transcription backends like Apple Speech or Parakeet.

The provider acts as a Swift bridge to **SwiftWhisper**, which wraps the C++ implementation of OpenAI's Whisper model. This architecture keeps all audio processing local to the machine, requiring no network connectivity after the initial model download.

## The Five-Stage Transcription Pipeline

The WhisperProvider manages the entire lifecycle of speech recognition through five distinct operational stages defined in [`Sources/Fluid/Services/WhisperProvider.swift`](https://github.com/altic-dev/FluidVoice/blob/main/Sources/Fluid/Services/WhisperProvider.swift).

### Initialization and Model Configuration

The provider initializes with an optional custom model directory, a `URLSession` for network operations, and an optional `modelOverride` that can temporarily force a specific model variant.

According to [`WhisperProvider.swift`](https://github.com/altic-dev/FluidVoice/blob/main/WhisperProvider.swift) (lines 26-30), the `init` method signature is:

```swift
init(modelDirectory: URL? = nil, session: URLSession = .shared, modelOverride: SpeechModel? = nil)

```

The `modelName` property dynamically determines which model file to load based on the current `SettingsStore` selection, defaulting to `ggml-base.bin` when no override is present (lines 32-41).

### Preparation and Memory Guarding

The `prepare()` method handles model caching, validation, and loading. It performs three critical tasks:

- **Cache validation**: Creates the cache folder and verifies existing files through size checks
- **Conditional downloading**: Fetches the model from Hugging Face (`https://huggingface.co/ggerganov/whisper.cpp/resolve/main/<model>.bin`) if missing
- **Memory safety**: Checks system RAM availability against the model's `requiredMemoryGB` before loading

According to the source code in [`WhisperProvider.swift`](https://github.com/altic-dev/FluidVoice/blob/main/WhisperProvider.swift) (lines 77-132), the provider queries available memory via mach APIs in `availableMemoryGB()` and aborts with a descriptive error if the system cannot satisfy the requirement.

### Audio Transcription

Once prepared, the `transcribe()` method accepts an array of `Float` PCM samples at 16 kHz. The implementation includes a sanity check that rejects buffers smaller than 16,000 samples (approximately one second of audio).

Because whisper.cpp does not expose confidence metrics, the provider always returns a confidence value of `1.0` in the `ASRTranscriptionResult`. The method concatenates segment texts from the underlying `Whisper` object's `transcribe(audioFrames:)` call and trims whitespace before returning.

### Cache Management

The provider exposes `modelsExistOnDisk()` to verify local availability and `clearCache()` to purge downloaded files, both defined in [`WhisperProvider.swift`](https://github.com/altic-dev/FluidVoice/blob/main/WhisperProvider.swift) (lines 124-143). These methods enable the UI to show storage status and provide cleanup functionality.

## Supported Whisper Models and Specifications

FluidVoice exposes six Whisper variants through the `SettingsStore.SpeechModel` enum, defined in [`Sources/Fluid/Persistence/SettingsStore.swift`](https://github.com/altic-dev/FluidVoice/blob/main/Sources/Fluid/Persistence/SettingsStore.swift). The provider automatically maps these enum cases to specific GGML binary files.

| Enum Case | Model File | Size | Description |
|-----------|------------|------|-------------|
| `whisperTiny` | `ggml-tiny.bin` | ~75 MB | Fastest inference, lowest accuracy |
| `whisperBase` | `ggml-base.bin` | ~142 MB | Default choice balancing speed and accuracy |
| `whisperSmall` | `ggml-small.bin` | ~466 MB | Improved accuracy with moderate speed trade-off |
| `whisperMedium` | `ggml-medium.bin` | ~1.5 GB | High accuracy requiring significant RAM |
| `whisperLargeTurbo` | `ggml-large-v3-turbo.bin` | ~1.5 GB | Higher quality optimized for speed (currently disabled in UI) |
| `whisperLarge` | `ggml-large-v3.bin` | ~2.9 GB | Best accuracy, largest resource requirements |

The enum definitions and human-readable titles reside in [`SettingsStore.swift`](https://github.com/altic-dev/FluidVoice/blob/main/SettingsStore.swift) (lines 56-85). When users switch models via the UI, the provider downloads the new binary on the next `prepare()` invocation if it is not already cached locally.

## Integration with FluidVoice Services

The WhisperProvider integrates into the broader application through two primary touchpoints:

- **ASRService** lazily instantiates the provider via `getWhisperProvider()` when the selected speech model matches a Whisper variant ([`ASRService.swift`](https://github.com/altic-dev/FluidVoice/blob/main/ASRService.swift), lines 152-218).
- **AISettingsView** renders the model selection interface using the `SpeechModel` enum, persisting user choice through `SettingsStore.shared.selectedSpeechModel` (`AISettingsView+SpeechRecognition.swift`).

## Practical Implementation Examples

The following Swift patterns demonstrate typical WhisperProvider usage within the FluidVoice architecture.

### Initialization and Preparation

```swift
import Fluid

let provider = WhisperProvider()

// Prepare with progress tracking
Task {
    do {
        try await provider.prepare { progress in
            // Handle ModelPreparationProgress enum cases
            print("Preparation status: \(progress)")
        }
    } catch {
        print("Preparation failed: \(error)")
    }
}

```

### Transcribing Audio

```swift
// Assuming 'samples' is a [Float] array of 16kHz PCM data
func processAudio(_ samples: [Float]) async {
    do {
        let result = try await provider.transcribe(samples)
        print("Transcribed: \(result.text)")
    } catch {
        print("Transcription error: \(error)")
    }
}

```

### Switching Models

```swift
// Change model via SettingsStore (triggers download on next prepare)
SettingsStore.shared.selectedSpeechModel = .whisperSmall

```

## Summary

- **WhisperProvider** implements the `TranscriptionProvider` protocol for offline speech recognition on Intel Macs using whisper.cpp via SwiftWhisper.
- The **five-stage pipeline** includes initialization, model selection, preparation with memory guarding, transcription, and cache management.
- **Six models** are supported, ranging from Tiny (~75 MB) to Large (~2.9 GB), with Base set as the default.
- **Memory safety** is enforced by comparing `requiredMemoryGB` against system-available RAM before loading large models.
- Models are downloaded dynamically from Hugging Face and cached locally, with no network requirements during transcription.

## Frequently Asked Questions

### What is the default Whisper model in FluidVoice?

The default model is **Whisper Base** (`ggml-base.bin`), approximately 142 MB in size. This is determined by the `modelName` property in [`WhisperProvider.swift`](https://github.com/altic-dev/FluidVoice/blob/main/WhisperProvider.swift), which falls back to Base when no specific model is selected in `SettingsStore`.

### Does WhisperProvider work on Apple Silicon Macs?

The provider is specifically designed for **Intel (x86-64) Macs**. The FluidVoice codebase uses different transcription providers for Apple Silicon chips, making WhisperProvider the dedicated backend for Intel-based hardware.

### How does FluidVoice handle insufficient RAM for large models?

Before loading any model, the provider calls `availableMemoryGB()` using mach APIs to query system memory. It compares this against the model's `requiredMemoryGB` property defined in the `SpeechModel` enum. If available RAM is insufficient, the `prepare()` method throws an error, preventing the application from crashing due to memory pressure.

### Where are the Whisper models downloaded from?

Models are downloaded from the official Hugging Face repository at `https://huggingface.co/ggerganov/whisper.cpp/resolve/main/<model-file>.bin`. The download logic includes retry mechanisms with exponential backoff and reports progress through the `ModelPreparationProgress` enum callback.