FluidVoice Performance Characteristics: Low-Latency Streaming Architecture Explained

FluidVoice achieves near-instantaneous speech-to-text transcription through a layered streaming pipeline that keeps CPU usage under 20% on Apple Silicon while delivering sub-30ms latency for supported models.

FluidVoice is a macOS-native dictation application built for speed. Its performance characteristics stem from careful separation of audio capture, neural inference, and UI rendering into distinct, non-blocking stages. This article breaks down the measured latency numbers, resource footprints, and architectural decisions that enable real-time voice transcription.

Streaming ASR Pipeline Architecture

The core performance advantage comes from true streaming inference rather than batched processing. In Sources/Fluid/Services/ASRService.swift, the ASRService class coordinates audio chunks as they arrive, feeding them directly into compatible speech models without waiting for utterance boundaries.

Audio Capture Layer

Raw PCM data flows through DirectCoreAudioInput.swift and CoreAudioCaptureSupportBridge.c on a dedicated background thread. This captures microphone input using Core Audio's low-latency callbacks and pushes samples into ThreadSafeAudioBuffer.swift — a lock-free ring buffer that eliminates contention between capture and consumption threads.

// ThreadSafeAudioBuffer provides lock-free audio handoff
// See: Sources/Fluid/Services/ThreadSafeAudioBuffer.swift
let buffer = ThreadSafeAudioBuffer(capacity: 1024 * 1024)
buffer.write(samples)  // Called from Core Audio callback
// ...
buffer.read(into: outputArray)  // Called by ASRService on inference queue

Model Inference Layer

ASRService.swift dispatches audio chunks to the selected provider using Swift's structured concurrency. Streaming-capable models return partial transcripts incrementally:

Model	Streaming Support	Typical Latency	Use Case
Parakeet Flash	Yes (FluidAudio "true streaming EOU")	< 30 ms	English-only, maximum speed
Nemotron Speech 3.5	Yes	50–100 ms	Multilingual with low latency
Whisper variants	No (batched)	300–500 ms	Higher accuracy, offline capable

Non-streaming models buffer audio until a silence threshold is detected, then perform complete inference — trading latency for accuracy.

Measured Performance Metrics

Latency Characteristics

FluidVoice reports real-world latency through its settings UI. In Sources/Fluid/UI/AISettingsView+AIConfiguration.swift, the privateAILoadState enum captures model initialization time:

// From AISettingsView+AIConfiguration.swift
case let .loaded(provider, latencyMilliseconds):
    if let ms = latencyMilliseconds {
        print("Model ready: \(provider.name) in \(ms)ms")
    }

End-to-end latency — from microphone to displayed text — depends on model selection:

• Parakeet Flash: Near-instant (< 30 ms), with partial words appearing as spoken
• Nemotron 3.5: Low latency (≈ 50–100 ms) with full multilingual support
• Whisper Tiny/Base: Moderate latency (300–500 ms), suitable for non-interactive use

Resource Footprint

Component	Disk	RAM (Active)	CPU (Typical)
Parakeet Flash	~250 MB	400–600 MB	~10% single core
Nemotron Speech 3.5	~670 MB	~1 GB	15–20% single core
Whisper Tiny	~75 MB	300–500 MB	5–10% (CPU-only)
Fluid Intelligence (optional)	~3.5 GB	~1 GB	< 5% (Neural Engine / GPU)

CPU usage remains modest because neural inference runs on Apple Silicon's dedicated Neural Engine when available. The main application thread handles only UI updates and coordinate marshaling.

Optional Post-Processing: Fluid Intelligence

PrivateAIProvider.swift implements on-device AI refinement for punctuation, capitalization, and formatting. This adds a small, bounded latency overhead:

// Toggle in SettingsStore
SettingsStore.shared.enableFluidIntelligence = true

// Typical overhead: ≤ 200ms for local inference
// Runs on Metal Performance Shaders or ANE, never leaves device

The post-processing pipeline operates on completed utterances in parallel with the next audio capture, so perceived interruption is minimized.

Threading and UI Performance

All heavy operations — audio encoding, model inference, and AI post-processing — execute on background queues defined in the service layer. fluidApp.swift wires these together while ensuring the overlay UI (NotchContentViews) refreshes at display rate without blocking:

// From fluidApp.swift - service coordination
ASRService.shared.transcriptionStream
    .receive(on: DispatchQueue.main)  // UI updates only
    .sink { transcript in
        self.overlayWindow.updateText(transcript)
    }

This architecture prevents frame drops even on Intel Macs or during sustained dictation sessions.

Configuration for Performance

Developers and power users can tune behavior through SettingsStore:

// Select lowest-latency model
SettingsStore.shared.selectedModelID = "parakeet-flash"

// Disable AI post-processing for maximum speed
SettingsStore.shared.enableFluidIntelligence = false

// Access the background queue for custom extensions
TranscriptionProvider.shared.queue.async {
    // Runs on transcription thread pool
}

Summary

• Sub-30ms latency achievable with Parakeet Flash through streaming inference in ASRService.swift
• Lock-free audio buffering via ThreadSafeAudioBuffer.swift eliminates capture thread stalls
• Neural Engine delegation keeps CPU usage under 20% during active dictation
• Optional AI refinement in PrivateAIProvider.swift adds ≤200ms bounded overhead
• Background-first architecture ensures responsive UI via dispatch queues in fluidApp.swift

Frequently Asked Questions

What is the fastest speech model in FluidVoice?

Parakeet Flash delivers the lowest latency at under 30 milliseconds end-to-end. It uses FluidAudio's true streaming endpoint-of-utterance pipeline, returning partial transcripts word-by-word rather than waiting for silence. This model is English-only and optimized specifically for dictation speed.

Does FluidVoice work well on older Intel Macs?

Yes, though latency increases. The Core Audio capture layer and threading architecture remain efficient on Intel hardware. Models fall back to CPU inference when Neural Engine is unavailable. Whisper Tiny (CPU-only, ~75 MB) provides acceptable performance on older machines at 300–500 ms latency.

How much memory does FluidVoice use during active dictation?

Typical RAM usage ranges from 400 MB to 1 GB depending on model selection. Parakeet Flash uses 400–600 MB; Nemotron Speech 3.5 requires approximately 1 GB. Enabling Fluid Intelligence adds another ~1 GB for the local language model. The application releases model memory when dictation stops to minimize background footprint.

Can I disable AI post-processing to reduce latency?

Yes. Set SettingsStore.shared.enableFluidIntelligence = false to bypass PrivateAIProvider.swift entirely. This eliminates the ≤200 ms post-processing overhead and streams raw ASR output directly to the overlay. Raw transcripts omit punctuation and capitalization unless the underlying speech model provides them.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}