# How to Implement Real-Time Transcription with Streaming ASR in FluidAudio

> Implement real-time transcription using FluidAudio's StreamingAsrManager. Get incremental, time-stamped transcriptions from live audio with built-in resampling and error recovery.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: how-to-guide
- Published: 2026-03-02

---

**FluidAudio's `StreamingAsrManager` actor provides an async-stream API that converts live audio buffers into incremental, time-stamped transcriptions with built-in resampling, sliding-window buffering, and error recovery.**

Real-time transcription with streaming ASR enables applications to convert speech to text incrementally as audio arrives, rather than waiting for complete files. The FluidAudio open-source repository implements this through a high-level Swift actor that orchestrates audio preprocessing, neural inference, and vocabulary boosting behind a simple async sequence interface.

## Architecture and Data Flow

### Core Components

| Component | Role | Source File |
|-----------|------|-------------|
| **`StreamingAsrManager`** | An actor that owns the ASR model (`AsrManager`), an `AudioConverter`, and a sliding-window buffer. It receives raw `AVAudioPCMBuffer`s, converts them to 16 kHz mono, assembles overlapping windows (`chunk + left/right context`), and calls the ASR model for each window. The actor publishes `StreamingTranscriptionUpdate` objects through an `AsyncStream`. | [`Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift) |
| **`StreamingAsrConfig`** | A pure-Swift struct that defines chunk length, hypothesis chunk length, left/right context, confirmation thresholds, etc. The default (`StreamingAsrConfig.streaming`) is tuned for low latency and high-quality results. | [`Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift) (lines 441–476) |
| **`AsyncStream<StreamingTranscriptionUpdate>`** | Consumers subscribe to this stream to receive real-time updates. Each update contains the provisional text, an `isConfirmed` flag (high-confidence, after enough context), confidence score, and per-token timing information. | Same file – `transcriptionUpdates` property (lines 88–100) |
| **`ASRConstants`** | Holds global constants such as the number of encoder frames per audio sample (`samplesPerEncoderFrame`). Used by `StreamingAsrManager.applyGlobalFrameOffset` to align token timestamps across windows. | [`Sources/FluidAudio/Shared/ASRConstants.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Shared/ASRConstants.swift) |

### Streaming Pipeline Steps

1. **Initialization** – `StreamingAsrManager(config:)` creates an `AsyncStream` for incoming PCM buffers and an `AsyncStream` for transcription updates.

2. **Model loading** – `start(models:source:)` builds an `AsrManager` (non-Sendable CoreML models) **inside the actor**, guaranteeing thread safety without `@unchecked Sendable`.

3. **Audio ingestion** – The caller repeatedly calls `streamAudio(_:)`. The method simply `yield`s the buffer into the internal `inputBuilder`.

4. **Background recognizer task** – The actor runs a `Task` that iterates over `inputSequence`. For each buffer:
   * Resample to 16 kHz mono (`AudioConverter.resampleBuffer`).
   * Append samples to the sliding-window buffer (`sampleBuffer`).
   * While enough samples exist for `chunk + rightContext`, extract a window `[left … chunk … right]` and invoke `asrManager.transcribeStreamingChunk`.

5. **Result handling** – The returned tokens and timestamps are **offset** to the global audio timeline (`applyGlobalFrameOffset`). The manager:
   * Updates internal transcript state (`volatileTranscript` / `confirmedTranscript`).
   * Optionally runs vocabulary rescoring (`applyVocabularyRescoring`) if a custom vocab was configured.
   * Emits a `StreamingTranscriptionUpdate` through `updateContinuation?.yield(update)`.

6. **Finalisation** – `finish()` closes the input stream, waits for the recognizer task, and builds the final text from accumulated tokens (or the volatile/confirmed buffers if vocab boosting was used).

## Swift Implementation Guide

The following example demonstrates a complete real-time transcription implementation using the `StreamingAsrManager` actor:

```swift
import FluidAudio
import AVFoundation

// 1️⃣ Create a streaming manager with the high‑performance config
let streaming = StreamingAsrManager(config: .streaming)

// 2️⃣ Load models (v3 is the default, you can pick v2)
let models = try await AsrModels.downloadAndLoad(version: .v3)

// 3️⃣ Start the engine (microphone is the default source)
try await streaming.start(models: models)

// 4️⃣ Subscribe to transcription updates
let updateTask = Task {
    for await update in streaming.transcriptionUpdates {
        // `update.isConfirmed` tells you whether this is a stable hypothesis
        print("[\(update.isConfirmed ? "✅" : "🕒")] \(update.text)  (conf \(update.confidence))")
    }
}

// 5️⃣ Feed audio chunks (e.g. from a microphone, screen‑capture, or a file)
// Here we simulate a file split into 1‑second buffers:
let url = URL(fileURLWithPath: "audio.wav")
let file = try AVAudioFile(forReading: url)
let format = file.processingFormat
while let buffer = AVAudioPCMBuffer(pcmFormat: format,
                                   frameCapacity: AVAudioFrameCount(format.sampleRate)) {
    try file.read(into: buffer)
    if buffer.frameLength == 0 { break }
    streaming.streamAudio(buffer)          // ← fire‑and‑forget
}

// 6️⃣ When the source ends, request the final transcription
let finalText = try await streaming.finish()
print("🗣️ Final transcription: \(finalText)")

// 7️⃣ Clean up the listener task
updateTask.cancel()

```

All heavy lifting—resampling, sliding-window management, token-offset handling, and vocabulary boosting—is encapsulated in `StreamingAsrManager`.

## Command-Line Interface Usage

For quick testing or batch processing, the FluidAudio CLI provides a complete streaming interface:

```bash

# Basic streaming transcription

fluidaudio transcribe path/to/audio.wav --streaming

# Show per‑update metadata (confidence, timestamps)

fluidaudio transcribe path/to/audio.wav --streaming --metadata

# Export the full session (including word‑level timings) to JSON

fluidaudio transcribe path/to/audio.wav \
    --streaming --metadata --output-json result.json

# Enable domain‑specific vocabulary boosting (hybrid rescoring)

fluidaudio transcribe path/to/audio.wav \
    --streaming --custom-vocab vocab.txt

```

The CLI internally follows the same steps as the programmatic example: it creates a `StreamingAsrManager`, configures optional vocabulary boosting, streams the file in chunk-sized buffers, and prints incremental updates.

## Advanced Configuration

### Custom Vocabulary Boosting

Domain-specific terminology can improve accuracy through optional CTC-based rescoring:

```swift
// Load a custom vocabulary file (one term per line)
let (vocab, ctcModels) = try await CustomVocabularyContext.loadWithCtcTokens(
    from: "/path/to/vocab.txt")

// Attach it to the streaming manager *before* `start`
try await streaming.configureVocabularyBoosting(
    vocabulary: vocab,
    ctcModels: ctcModels)

```

During streaming, every confirmed chunk is passed through the CTC-based rescorer; corrected words appear automatically in later `CONFIRMED` updates.

### Latency and Chunk Configuration

The `StreamingAsrConfig` struct controls real-time performance characteristics:

- **Chunk size** (`chunkSeconds`): Defaults to 11 seconds of audio with 2 seconds of left/right context, yielding approximately 0.5 seconds end-to-end latency on modern Mac hardware.
- **Hypothesis updates**: Emitted for each processed window (approximately every 1 second if `hypothesisChunkSeconds` is set to 1).
- **Confirmation logic**: Results transition from `isConfirmed = false` (volatile) to `true` (stable) once `confirmationThreshold` confidence and `minContextForConfirmation` duration are satisfied.

Error recovery is built into the manager via `attemptErrorRecovery`, which can reset the decoder or reload models without crashing the client.

## Key Source Files Reference

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`StreamingAsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/StreamingAsrManager.swift) | Core streaming actor, async-stream API, sliding-window logic, vocab boosting, error recovery. | <https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift> |
| `StreamingAsrConfig` (inside same file) | Declarative configuration for chunk size, context, thresholds, etc. | <https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift#L441-L476> |
| [`TranscribeCommand.swift`](https://github.com/fluidinference/fluidaudio/blob/main/TranscribeCommand.swift) | CLI entry point demonstrating full streaming sessions, JSON export, and metadata options. | <https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudioCLI/Commands/ASR/TranscribeCommand.swift> |
| [`ASRConstants.swift`](https://github.com/fluidinference/fluidaudio/blob/main/ASRConstants.swift) | Global constants (e.g., `samplesPerEncoderFrame`) used for timestamp alignment across windows. | <https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Shared/ASRConstants.swift> |

These files together provide a complete, production-ready solution for **real-time transcription with streaming ASR** with optional domain adaptation, token-level timing, and robust error handling.

## Summary

- **FluidAudio** provides a high-level `StreamingAsrManager` actor that encapsulates complex audio preprocessing and neural inference behind a simple async-stream API.
- The architecture uses **sliding-window buffering** with configurable left/right context to maintain temporal accuracy while processing live audio chunks.
- Consumers receive `StreamingTranscriptionUpdate` objects through an `AsyncStream`, with `isConfirmed` flags distinguishing volatile hypotheses from stable transcription.
- **Vocabulary boosting** via CTC rescoring can be injected before `start()` to improve domain-specific accuracy without retraining the base model.
- The `TranscribeCommand` CLI tool demonstrates production-ready usage including JSON export, metadata logging, and error recovery.

## Frequently Asked Questions

### How does FluidAudio handle timestamp alignment across overlapping audio windows?

FluidAudio uses global frame offset calculation via `ASRConstants.samplesPerEncoderFrame` to align token timestamps across windows. When `StreamingAsrManager` processes each chunk, it calls `applyGlobalFrameOffset` to adjust the relative timestamps from the current window to the absolute timeline of the continuous audio stream, ensuring that words spanning multiple overlapping buffers maintain consistent timing.

### What is the difference between volatile and confirmed transcription updates?

`StreamingTranscriptionUpdate` objects contain an `isConfirmed` boolean flag that indicates the stability of the hypothesis. **Volatile** updates (`isConfirmed = false`) represent interim results from recent audio windows that may change as more context arrives. **Confirmed** updates (`isConfirmed = true`) indicate that the text has met the `confirmationThreshold` confidence level and `minContextForConfirmation` duration requirements, making it stable and unlikely to change.

### Can I use FluidAudio for multi-stream real-time transcription?

Yes, because `StreamingAsrManager` is an actor, you can instantiate multiple independent managers to process different audio sources concurrently. Each manager maintains its own internal state, sliding-window buffer, and `AsyncStream` of updates. For benchmarking and examples of concurrent stream handling, refer to [`Sources/FluidAudioCLI/Commands/MultiStreamCommand.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudioCLI/Commands/MultiStreamCommand.swift), which demonstrates managing multiple simultaneous transcription sessions.

### How do I recover from ASR model errors during a live stream?

`StreamingAsrManager` includes built-in error recovery via the `attemptErrorRecovery` method. If the CoreML model fails or the decoder state becomes corrupted, the manager can reset the decoder or reload the models internally without terminating the client connection. To ensure robust production deployments, wrap your `transcriptionUpdates` loop in appropriate error handling, and rely on the manager's internal recovery mechanisms to maintain stream continuity.