How to Implement Real-Time Transcription with Streaming ASR in FluidAudio

FluidAudio's StreamingAsrManager actor provides an async-stream API that converts live audio buffers into incremental, time-stamped transcriptions with built-in resampling, sliding-window buffering, and error recovery.

Real-time transcription with streaming ASR enables applications to convert speech to text incrementally as audio arrives, rather than waiting for complete files. The FluidAudio open-source repository implements this through a high-level Swift actor that orchestrates audio preprocessing, neural inference, and vocabulary boosting behind a simple async sequence interface.

Architecture and Data Flow

Core Components

Component	Role	Source File
StreamingAsrManager	An actor that owns the ASR model (AsrManager), an AudioConverter, and a sliding-window buffer. It receives raw AVAudioPCMBuffers, converts them to 16 kHz mono, assembles overlapping windows (chunk + left/right context), and calls the ASR model for each window. The actor publishes StreamingTranscriptionUpdate objects through an AsyncStream.	Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift
StreamingAsrConfig	A pure-Swift struct that defines chunk length, hypothesis chunk length, left/right context, confirmation thresholds, etc. The default (StreamingAsrConfig.streaming) is tuned for low latency and high-quality results.	Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift (lines 441–476)
AsyncStream<StreamingTranscriptionUpdate>	Consumers subscribe to this stream to receive real-time updates. Each update contains the provisional text, an isConfirmed flag (high-confidence, after enough context), confidence score, and per-token timing information.	Same file – transcriptionUpdates property (lines 88–100)
ASRConstants	Holds global constants such as the number of encoder frames per audio sample (samplesPerEncoderFrame). Used by StreamingAsrManager.applyGlobalFrameOffset to align token timestamps across windows.	Sources/FluidAudio/Shared/ASRConstants.swift

Streaming Pipeline Steps

1. Initialization – StreamingAsrManager(config:) creates an AsyncStream for incoming PCM buffers and an AsyncStream for transcription updates.

2. Model loading – start(models:source:) builds an AsrManager (non-Sendable CoreML models) inside the actor, guaranteeing thread safety without @unchecked Sendable.

3. Audio ingestion – The caller repeatedly calls streamAudio(_:). The method simply yields the buffer into the internal inputBuilder.

4. Background recognizer task – The actor runs a Task that iterates over inputSequence. For each buffer:

   • Resample to 16 kHz mono (AudioConverter.resampleBuffer).
   • Append samples to the sliding-window buffer (sampleBuffer).
   • While enough samples exist for chunk + rightContext, extract a window [left … chunk … right] and invoke asrManager.transcribeStreamingChunk.

5. Result handling – The returned tokens and timestamps are offset to the global audio timeline (applyGlobalFrameOffset). The manager:

   • Updates internal transcript state (volatileTranscript / confirmedTranscript).
   • Optionally runs vocabulary rescoring (applyVocabularyRescoring) if a custom vocab was configured.
   • Emits a StreamingTranscriptionUpdate through updateContinuation?.yield(update).

6. Finalisation – finish() closes the input stream, waits for the recognizer task, and builds the final text from accumulated tokens (or the volatile/confirmed buffers if vocab boosting was used).

Swift Implementation Guide

The following example demonstrates a complete real-time transcription implementation using the StreamingAsrManager actor:

import FluidAudio
import AVFoundation

// 1️⃣ Create a streaming manager with the high‑performance config
let streaming = StreamingAsrManager(config: .streaming)

// 2️⃣ Load models (v3 is the default, you can pick v2)
let models = try await AsrModels.downloadAndLoad(version: .v3)

// 3️⃣ Start the engine (microphone is the default source)
try await streaming.start(models: models)

// 4️⃣ Subscribe to transcription updates
let updateTask = Task {
    for await update in streaming.transcriptionUpdates {
        // `update.isConfirmed` tells you whether this is a stable hypothesis
        print("[\(update.isConfirmed ? "✅" : "🕒")] \(update.text)  (conf \(update.confidence))")
    }
}

// 5️⃣ Feed audio chunks (e.g. from a microphone, screen‑capture, or a file)
// Here we simulate a file split into 1‑second buffers:
let url = URL(fileURLWithPath: "audio.wav")
let file = try AVAudioFile(forReading: url)
let format = file.processingFormat
while let buffer = AVAudioPCMBuffer(pcmFormat: format,
                                   frameCapacity: AVAudioFrameCount(format.sampleRate)) {
    try file.read(into: buffer)
    if buffer.frameLength == 0 { break }
    streaming.streamAudio(buffer)          // ← fire‑and‑forget
}

// 6️⃣ When the source ends, request the final transcription
let finalText = try await streaming.finish()
print("🗣️ Final transcription: \(finalText)")

// 7️⃣ Clean up the listener task
updateTask.cancel()

All heavy lifting—resampling, sliding-window management, token-offset handling, and vocabulary boosting—is encapsulated in StreamingAsrManager.

Command-Line Interface Usage

For quick testing or batch processing, the FluidAudio CLI provides a complete streaming interface:

# Basic streaming transcription

fluidaudio transcribe path/to/audio.wav --streaming

# Show per‑update metadata (confidence, timestamps)

fluidaudio transcribe path/to/audio.wav --streaming --metadata

# Export the full session (including word‑level timings) to JSON

fluidaudio transcribe path/to/audio.wav \
    --streaming --metadata --output-json result.json

# Enable domain‑specific vocabulary boosting (hybrid rescoring)

fluidaudio transcribe path/to/audio.wav \
    --streaming --custom-vocab vocab.txt

The CLI internally follows the same steps as the programmatic example: it creates a StreamingAsrManager, configures optional vocabulary boosting, streams the file in chunk-sized buffers, and prints incremental updates.

Advanced Configuration

Custom Vocabulary Boosting

Domain-specific terminology can improve accuracy through optional CTC-based rescoring:

// Load a custom vocabulary file (one term per line)
let (vocab, ctcModels) = try await CustomVocabularyContext.loadWithCtcTokens(
    from: "/path/to/vocab.txt")

// Attach it to the streaming manager *before* `start`
try await streaming.configureVocabularyBoosting(
    vocabulary: vocab,
    ctcModels: ctcModels)

During streaming, every confirmed chunk is passed through the CTC-based rescorer; corrected words appear automatically in later CONFIRMED updates.

Latency and Chunk Configuration

The StreamingAsrConfig struct controls real-time performance characteristics:

• Chunk size (chunkSeconds): Defaults to 11 seconds of audio with 2 seconds of left/right context, yielding approximately 0.5 seconds end-to-end latency on modern Mac hardware.
• Hypothesis updates: Emitted for each processed window (approximately every 1 second if hypothesisChunkSeconds is set to 1).
• Confirmation logic: Results transition from isConfirmed = false (volatile) to true (stable) once confirmationThreshold confidence and minContextForConfirmation duration are satisfied.

Error recovery is built into the manager via attemptErrorRecovery, which can reset the decoder or reload models without crashing the client.

Key Source Files Reference

File	Purpose	Direct Link
StreamingAsrManager.swift	Core streaming actor, async-stream API, sliding-window logic, vocab boosting, error recovery.	https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift
StreamingAsrConfig (inside same file)	Declarative configuration for chunk size, context, thresholds, etc.	https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift#L441-L476
TranscribeCommand.swift	CLI entry point demonstrating full streaming sessions, JSON export, and metadata options.	https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudioCLI/Commands/ASR/TranscribeCommand.swift
ASRConstants.swift	Global constants (e.g., samplesPerEncoderFrame) used for timestamp alignment across windows.	https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Shared/ASRConstants.swift

These files together provide a complete, production-ready solution for real-time transcription with streaming ASR with optional domain adaptation, token-level timing, and robust error handling.

Summary

• FluidAudio provides a high-level StreamingAsrManager actor that encapsulates complex audio preprocessing and neural inference behind a simple async-stream API.
• The architecture uses sliding-window buffering with configurable left/right context to maintain temporal accuracy while processing live audio chunks.
• Consumers receive StreamingTranscriptionUpdate objects through an AsyncStream, with isConfirmed flags distinguishing volatile hypotheses from stable transcription.
• Vocabulary boosting via CTC rescoring can be injected before start() to improve domain-specific accuracy without retraining the base model.
• The TranscribeCommand CLI tool demonstrates production-ready usage including JSON export, metadata logging, and error recovery.

Frequently Asked Questions

How does FluidAudio handle timestamp alignment across overlapping audio windows?

FluidAudio uses global frame offset calculation via ASRConstants.samplesPerEncoderFrame to align token timestamps across windows. When StreamingAsrManager processes each chunk, it calls applyGlobalFrameOffset to adjust the relative timestamps from the current window to the absolute timeline of the continuous audio stream, ensuring that words spanning multiple overlapping buffers maintain consistent timing.

What is the difference between volatile and confirmed transcription updates?

StreamingTranscriptionUpdate objects contain an isConfirmed boolean flag that indicates the stability of the hypothesis. Volatile updates (isConfirmed = false) represent interim results from recent audio windows that may change as more context arrives. Confirmed updates (isConfirmed = true) indicate that the text has met the confirmationThreshold confidence level and minContextForConfirmation duration requirements, making it stable and unlikely to change.

Can I use FluidAudio for multi-stream real-time transcription?

Yes, because StreamingAsrManager is an actor, you can instantiate multiple independent managers to process different audio sources concurrently. Each manager maintains its own internal state, sliding-window buffer, and AsyncStream of updates. For benchmarking and examples of concurrent stream handling, refer to Sources/FluidAudioCLI/Commands/MultiStreamCommand.swift, which demonstrates managing multiple simultaneous transcription sessions.

How do I recover from ASR model errors during a live stream?

StreamingAsrManager includes built-in error recovery via the attemptErrorRecovery method. If the CoreML model fails or the decoder state becomes corrupted, the manager can reset the decoder or reload the models internally without terminating the client connection. To ensure robust production deployments, wrap your transcriptionUpdates loop in appropriate error handling, and rely on the manager's internal recovery mechanisms to maintain stream continuity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}