ASRService Architecture and CoreML Serialization in FluidVoice

FluidVoice's ASRService uses a private Swift actor called TranscriptionExecutor to serialize all CoreML operations into a single-threaded task chain, preventing EXC_BAD_ACCESS crashes while coordinating audio capture, model management, and real-time transcription across multiple speech recognition providers.

The speech recognition engine in the altic-dev/FluidVoice repository centers on ASRService, a self-contained Swift class that orchestrates the entire audio-to-text pipeline. Understanding its ASRService architecture reveals how the framework safely handles concurrent transcription requests while managing multiple CoreML models through strict serialization guarantees. The service implements a robust concurrency strategy that ensures thread-safe access to non-thread-safe CoreML frameworks like FluidAudio, Parakeet, and Whisper.

Core Architectural Components

Entry Point and Public API

The ASRService class serves as the central coordinator, defined as final class ASRService: ObservableObject in Sources/Fluid/Services/ASRService.swift (lines 82-99). It exposes the primary public API including start(), stop(), and downloadModel() while maintaining UI state through @Published properties. The service stores references to various transcription providers—including FluidAudio, Whisper, and Apple Speech—and coordinates device-level work such as audio engine management, microphone permission handling, and media-pause integration.

Dynamic Provider Selection

The service selects concrete TranscriptionProvider implementations based on the user's preference stored in SettingsStore.selectedSpeechModel. This decision logic lives in the computed property transcriptionProvider (lines 70-95). When users switch models mid-session, the resetTranscriptionProvider() method clears cached provider instances and cancels any in-flight model preparation tasks, forcing a fresh instantiation on the next transcription request.

Serializing CoreML Operations

The TranscriptionExecutor Actor

CoreML frameworks utilized by FluidVoice—including FluidAudio and Parakeet—are not thread-safe. Concurrent access to these models corrupts internal buffers and triggers EXC_BAD_ACCESS crashes. To solve this, ASRService encapsulates all model inference within TranscriptionExecutor, a private actor defined at lines 16-30 in ASRService.swift.

This actor maintains a lastTask reference that holds the most recent transcription operation. By isolating CoreML access to this actor, the service guarantees that only one thread can mutate model state at any given time, eliminating data races across the transcription pipeline.

Task Chain Mechanics

The serialization logic relies on a strict task chain pattern implemented in the run<T> method (lines 20-28). When new transcription work arrives, the executor creates a Task that first awaits previous?.result—the prior task's completion—before executing the supplied async closure. This chaining ensures every CoreML call completes sequentially, regardless of how many concurrent requests the UI layer issues.

// All provider-specific transcription calls follow this pattern:
await transcriptionExecutor.run {
    // CoreML inference happens here, guaranteed single-threaded
    return try await fluidAudioProvider.transcribe(buffer)
}

Cancellation Safety

Safe teardown is critical when users abort transcription or switch models. The cancelAndAwaitPending() method (lines 31-39) cancels the currently running operation via currentOperationTask?.cancel() and then awaits the final lastTask result. This prevents dangling references to deallocated model memory and ensures clean resource release before subsequent operations begin.

Audio Pipeline and Real-Time Processing

Thread-Safe Audio Capture

The service captures raw microphone samples via AVAudioEngine, which is lazily instantiated in engineStorage (lines 33-41) to avoid reflection-based initialization crashes. The capture pipeline runs on a background dispatch queue, forwarding samples to a ThreadSafeAudioBuffer that safely shares data between the audio thread and the transcription tasks running on the TranscriptionExecutor actor.

Streaming Transcription

For models supporting streaming inference—such as Parakeet and Whisper—the startStreamingTranscription method activates a periodic timer that slices the audio buffer into fixed-size chunks (lines 626-665). Each chunk routes through the serialized executor using await transcriptionExecutor.run { ... }, applying analytics tracking and word-boost logic while maintaining the single-threaded execution guarantee required by CoreML.

Model Lifecycle Management

Async Preparation and Downloading

The service prevents overlapping preparation work by tracking asynchronous tasks like ensureReadyTask and modelDownloadTask. The download logic (lines 400-470) handles model caching, existence validation through both synchronous checkIfModelsExist and asynchronous checkIfModelsExistAsync APIs, and on-demand loading for providers like Apple Speech Analyzer.

Error Handling and Analytics

Centralized logging via DebugLogger.shared and AnalyticsService.shared provides detailed pipeline visibility. The system categorizes errors at lines 299-315, specifically handling CancellationError to distinguish between user-initiated aborts and genuine transcription failures, ensuring appropriate cleanup for each scenario.

Implementation Examples

Basic Service Usage

import Fluid

// Create the service (usually via dependency injection or shared instance)
let asr = ASRService()

// Initialize after the UI is ready (e.g., in a SwiftUI .onAppear)
asr.initialize()

// Start a recording session
Task {
    await asr.start()
}

// Later, stop the session and retrieve the transcript
Task {
    let transcript = await asr.stop()
    print("Transcript:", transcript)
}

Switching Speech Models

import Fluid

// Change the model in SettingsStore (UI control)
SettingsStore.shared.selectedSpeechModel = .parakeetTDT

// Tell ASRService to reset its internal provider state
asr.resetTranscriptionProvider()

The reset method clears cached providers, cancels any in-flight model preparation, and forces a fresh transcriptionProvider creation on the next use—all CoreML work continues to funnel through the same TranscriptionExecutor regardless of provider changes.

Summary

• ASRService in Sources/Fluid/Services/ASRService.swift acts as the central coordinator, managing multiple transcription providers and audio capture through a public Swift API.
• TranscriptionExecutor (lines 16-30) is a private actor that serializes all CoreML operations by chaining tasks via lastTask awaiting, ensuring single-threaded access to non-thread-safe models.
• Task chain mechanics in the run<T> method (lines 20-28) guarantee that new transcription work awaits the completion of previous operations before executing.
• Cancellation safety is handled by cancelAndAwaitPending() (lines 31-39), which prevents memory corruption by awaiting task completion before releasing resources.
• Streaming transcription routes audio chunks through the serialized executor (lines 626-665), enabling real-time recognition while maintaining thread safety.
• Model lifecycle is managed through tracked async tasks (lines 400-470) that prevent overlapping downloads and handle provider-specific preparation requirements.

Frequently Asked Questions

Why does FluidVoice use an actor to serialize CoreML operations instead of a traditional queue?

The TranscriptionExecutor actor provides compile-time guarantees of single-threaded access that raw dispatch queues cannot enforce. Because CoreML models like Parakeet and FluidAudio crash with EXC_BAD_ACCESS when accessed concurrently, the actor's isolation ensures that the lastTask chaining logic in run<T> (lines 20-28) executes exclusively, preventing race conditions by design rather than by convention.

How does ASRService handle model switching without restarting the app?

When users change SettingsStore.selectedSpeechModel, calling resetTranscriptionProvider() clears the cached provider instance and cancels pending preparation tasks through cancelAndAwaitPending(). The next transcription request triggers the transcriptionProvider computed property (lines 70-95) to instantiate the new provider, with all CoreML operations still funneled through the same TranscriptionExecutor actor for consistent serialization.

What happens if a user cancels transcription while CoreML is processing?

The cancelAndAwaitPending() method (lines 31-39) calls currentOperationTask?.cancel() to signal cancellation, then awaits the lastTask result to ensure the CoreML model has finished its current inference before releasing resources. This prevents dangling pointer references to deallocated model memory and ensures the actor's internal state remains consistent for subsequent transcription requests.

Where does the audio data flow before reaching the CoreML models?

Raw microphone samples flow from AVAudioEngine (lazily created in engineStorage, lines 33-41) into a ThreadSafeAudioBuffer that safely bridges the background capture queue and the TranscriptionExecutor actor. For streaming models, the startStreamingTranscription logic (lines 626-665) chunks this buffer and routes each segment through await transcriptionExecutor.run { ... }, ensuring CoreML receives data only within the serialized execution context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}