NemotronProvider in FluidVoice: Features, Implementation, and Usage Guide

The NemotronProvider is FluidVoice's native transcription engine for NVIDIA Nemotron 3.5 models, delivering high-accuracy offline and ultra-low-latency streaming speech recognition exclusively on Apple Silicon devices.

The NemotronProvider class in the altic-dev/FluidVoice repository implements the TranscriptionProvider protocol from Sources/Fluid/Services/NemotronProvider.swift. It serves as the primary interface for integrating NVIDIA's Nemotron 3.5 speech models into macOS applications, handling everything from model downloading and caching to real-time audio stream processing.

Core Features of NemotronProvider

Multiple Operation Modes

The provider supports three distinct transcription modes defined in the Mode enum:

• .offline – High-accuracy batch transcription for recorded audio files
• .streaming – Ultra-low-latency real-time transcription for live audio
• .streaming320 – Streaming pipeline using an alternative model folder configuration

Each mode exposes specific displayName and folderHint properties that determine model selection and loading behavior.

Apple Silicon Hardware Optimization

The provider is compiled exclusively for arm64 architecture using conditional compilation guards (#if arch(arm64)). On non-Apple-Silicon devices, the class automatically sets isAvailable = false and throws descriptive errors explaining the hardware requirement. This ensures optimal performance by targeting the Neural Engine and GPU compute units available on Apple Silicon chips.

Automatic Model Download and Caching

On first initialization, NemotronProvider automatically downloads required CoreML artifacts from the Hugging Face repository BarathwajAnandan/… via HuggingFaceModelDownloader. The prepare(progressHandler:) method validates cached artifacts using artifactsAreComplete(at:) before loading, checking metadata, file size, and markup filtering to prevent corrupt downloads.

Dynamic Compute Unit Selection

The provider attempts to load models on the Neural Engine first (cpuAndNeuralEngine). If initialization fails, it automatically falls back to cpuAndGPU via the shouldRetryWithoutNeuralEngine(_:) method. This fallback mechanism ensures transcription availability across different device capabilities while prioritizing energy-efficient processing.

Language Selection Support

Users can select from approximately 40 supported locales via SettingsStore.NemotronLanguage. The provider applies language settings dynamically through applySelectedLanguage(to:), updating the CoreML manager only when SettingsStore.shared.selectedNemotronLanguage changes, avoiding unnecessary model reloads.

Streaming and Batch Transcription

Streaming transcription tracks processed audio via streamedSampleCount and feeds only delta buffers to NemotronStreamingAsrManager through consumeDelta(from:manager:).

Batch transcription implements intelligent audio chunking for long files, splitting on energy-quiet boundaries using chunkEnd(in:offset:) and quietestBoundary(...) to avoid cutting mid-speech.

Performance Profiling and Debugging

Optional component profiling records timing data for each model stage (pre-processor, encoder, decoder) when ASRComponentProfilingEnabled is set. Methods startComponentProfilingIfNeeded, finishComponentProfilingIfNeeded, and logComponentProfile provide detailed performance metrics for optimization.

Implementation Architecture

Audio Processing Pipeline

All input audio undergoes resampling to 16 kHz mono Float32 format via resampleBuffer(_:targetSampleRate:) before reaching the Nemotron models. This standardization ensures consistent inference regardless of source audio characteristics.

Error Handling and Recovery

The provider implements robust error management:

• Corrupt cached artifacts (detected via HTML markup filtering) trigger automatic re-download
• Transcription failures in transcribeSinglePass(_:) reset the manager and clear internal counters
• The clearCache() method forces complete model re-download for debugging purposes

Key Source Files

File	Purpose
Sources/Fluid/Services/NemotronProvider.swift	Core provider implementation
Sources/Fluid/Persistence/SettingsStore.swift	Language selection persistence
Sources/Fluid/Persistence/SettingsStore+NemotronLanguage.swift	Supported locale definitions
Sources/Fluid/Services/ThinkingParsers.swift	Thinking flag handling for Nemotron models

Public API and Usage Examples

Initialization and Model Preparation

import Fluid

// Initialize with streaming mode for real-time transcription
let nemotron = NemotronProvider(mode: .streaming)

// Or use offline mode for maximum accuracy
let offlineProvider = NemotronProvider(mode: .offline)

// Prepare downloads and loads models automatically
await nemotron.prepare { progress in
    switch progress {
    case .downloading(let fraction):
        print("Downloading... \(Int(fraction * 100))%")
    case .loading:
        print("Loading CoreML models")
    default: break
    }
}

Batch Transcription

// Process a Float array of 16kHz PCM samples
let samples: [Float] = // ... audio data
let result = try await nemotron.transcribe(samples)
print("Transcribed text: \(result.text)")

Streaming Transcription

var streamingProvider = NemotronProvider(mode: .streaming)
await streamingProvider.prepare()

// Feed audio incrementally
while let nextChunk = getNextAudioChunk() {
    let streamingResult = try await streamingProvider.transcribeStreaming(nextChunk)
    if !streamingResult.text.isEmpty {
        print("Partial: \(streamingResult.text)")
    }
}

File-Based Transcription

let fileURL = URL(fileURLWithPath: "/path/to/audio.wav")
let fileResult = try await nemotron.transcribeFile(at: fileURL)
print("Full transcription: \(fileResult.text)")

Cache Management

// Force re-download for debugging or corruption recovery
try await nemotron.clearCache()

Summary

• NemotronProvider implements the TranscriptionProvider protocol in Sources/Fluid/Services/NemotronProvider.swift to provide native NVIDIA Nemotron 3.5 support in FluidVoice.
• The class offers three operation modes—offline, streaming, and streaming320—catering to different latency and accuracy requirements.
• Apple Silicon exclusivity ensures optimal Neural Engine utilization, with automatic fallback to GPU when necessary.
• Automatic model management handles downloading, caching, and validation from Hugging Face repositories without manual intervention.
• The API provides streaming-aware delta processing and smart audio chunking for efficient handling of both real-time and batch transcription tasks.

Frequently Asked Questions

What hardware is required to use NemotronProvider?

NemotronProvider requires Apple Silicon (ARM64) architecture. The implementation uses #if arch(arm64) compilation guards to ensure the class only compiles on compatible devices. On Intel-based Macs, isAvailable returns false and the provider cannot be instantiated, as the CoreML models are optimized specifically for the Neural Engine and GPU compute units found in Apple Silicon chips.

How does NemotronProvider handle model downloads and storage?

The provider automatically downloads CoreML artifacts from Hugging Face (specifically the BarathwajAnandan/… repository) on first use via HuggingFaceModelDownloader. Downloads are cached locally and validated before reuse through artifactsAreComplete(at:), which checks file integrity, metadata consistency, and filters out HTML markup that might indicate download errors. Users can force re-download using the clearCache() method.

What is the difference between streaming and offline modes?

Offline mode processes complete audio files with maximum accuracy using the full Nemotron 3.5 model, suitable for transcription of recorded meetings or podcasts. Streaming mode provides ultra-low-latency transcription by processing audio deltas incrementally, tracking processed samples via streamedSampleCount and feeding only new audio to NemotronStreamingAsrManager. Streaming320 uses the same streaming pipeline but loads models from an alternative folder configuration optimized for specific latency requirements.

How many languages does NemotronProvider support?

The provider supports approximately 40 languages and locales defined in SettingsStore+NemotronLanguage.swift. Users select their preferred language through SettingsStore.shared.selectedNemotronLanguage, which the provider applies via applySelectedLanguage(to:) to configure the underlying CoreML manager. The language setting persists across app launches and only triggers model reconfiguration when actually changed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}