How Context Biasing Improves ASR Accuracy for Domain-Specific Terms in FluidAudio

Context biasing boosts ASR accuracy by adding a configurable log-probability weight to domain-specific terms during CTC decoding, offsetting the acoustic model's bias against rare or out-of-vocabulary words.

FluidAudio implements an advanced context biasing system that enables real-time recognition of domain-specific terminology—such as product names, medical jargon, or technical commands—without retraining the underlying acoustic model. According to the fluidinference/fluidaudio source code, the system uses a three-component pipeline that dynamically rescores CTC hypotheses against a custom vocabulary during streaming transcription.

The Three-Pillar Architecture of Context Biasing

FluidAudio’s context biasing implementation relies on tightly coupled components that handle data management, algorithmic constants, and execution logic.

CustomVocabularyContext: Runtime Term Management

The CustomVocabularyContext class, defined in Sources/FluidAudio/ASR/CustomVocabulary/CustomVocabularyContext.swift, serves as the runtime container for domain-specific terms. It parses JSON vocabulary files and stores per-term parameters including context biasing weight (CBW), CTC-score thresholds, and optional token IDs. When CustomVocabularyContext.load(from:) parses a vocabulary file, it sanitizes each term and fills missing parameters using defaults from the system's constant registry.

ContextBiasingConstants: Algorithmic Tuning

All magic numbers governing the rescoring behavior live in Sources/FluidAudio/ASR/CustomVocabulary/ContextBiasingConstants.swift. This centralizes critical thresholds such as defaultAlpha (the base CBW), similarity floors, and adaptive scaling parameters. The file provides rescorerConfig(forVocabSize:), which returns vocabulary-size-aware configurations—large vocabularies receive tighter similarity floors (minSimilarity = 0.60) and slightly reduced CBW values (cbw = 2.5) to prevent false positives.

VocabularyRescorer: CTC-Based Execution Engine

The actual biasing logic executes in Sources/FluidAudio/ASR/CustomVocabulary/Rescorer/VocabularyRescorer.swift. This CTC-based rescoring engine compares acoustic evidence for hypothesized words against boosted terms via ctcTokenRescore(...). It enforces hierarchical similarity thresholds and applies the adaptive CBW to log-probabilities, determining whether a domain term should replace the generic hypothesis in the final transcript.

How Context Biasing Works: The Execution Pipeline

The system follows a six-stage pipeline during streaming ASR sessions, orchestrated by StreamingAsrManager in Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift.

1. Vocabulary Loading and Validation

The process begins when CustomVocabularyContext.load(from:) ingests a JSON or text file containing domain terms. The method validates each entry against ContextBiasingConstants defaults, ensuring every term carries required parameters like defaultMinSimilarity for fuzzy matching guards.

2. CTC Keyword Spotting

The CtcKeywordSpotter runs the acoustic model on audio buffers, producing frame-wise log-probabilities for every token in the vocabulary. This provides the raw acoustic scores that the rescoring engine will modify.

3. Rescoring Engine Configuration

StreamingAsrManager.configureVocabularyBoosting(...) constructs a VocabularyRescorer via an async factory. The factory checks ContextBiasingConstants.useBkTree to determine whether to enable BK-tree indexing for O(log V) fuzzy matching performance, and selects the appropriate configuration via rescorerConfig(forVocabSize:).

4. Adaptive Bias Weight Calculation

Inside VocabularyRescorer.Config.adaptiveCbw(baseCbw:tokenCount:), the system calculates a dynamic CBW. The bias weight scales by the ratio of the term's token count to a reference count, giving longer phrases a larger boost to compensate for CTC's cumulative acoustic penalty across multiple tokens.

5. Real-Time Rescoring During Streaming

For each confirmed audio chunk processed by processWindow, the system executes VocabularyRescorer.ctcTokenRescore(...):

• Evaluates string similarity using Levenshtein distance against the original hypothesis
• Enforces hierarchical thresholds: minSimilarityFloor → shortWordSimilarity → stopwordSpanSimilarity
• Adds the adaptive CBW to the term's CTC log-probability
• Replaces the original word if the adjusted score exceeds the baseline by the configured margin

6. Result Propagation

The rescored transcript flows back to StreamingAsrManager, which updates the UI via StreamingTranscriptionUpdate. The final finish() call assembles the complete text using rescored confirmed portions, ensuring domain-specific terms survive the final output even when the raw acoustic model initially missed them.

Why Context Biasing Solves Domain-Specific Recognition Failures

Context biasing addresses specific failure modes inherent to general-purpose acoustic models:

Rare or out-of-vocabulary words. When the acoustic model has never encountered a term like a proprietary product name, its CTC score collapses. The CBW—defaulting to values around 2.5–3.0 plus adaptive scaling—adds a substantial log-probability boost that makes rare terms competitive with generic hypotheses.

Long multi-token sequences. CTC applies a penalty per token, causing cumulative scores to drop over longer phrases. The adaptiveCbw mechanism scales the boost with token count, effectively neutralizing this length-based penalty.

Phonetically similar competitors. Terms like "nvidia" versus "nvida" create confusion. Hierarchical similarity thresholds ensure only candidates meeting Levenshtein-based guards (floor → short-word → stopword span) qualify for boosting, eliminating false positives from near-miss words.

Large vocabulary scalability. As vocabulary size grows, false-positive risk increases. The rescorerConfig(forVocabSize:) method automatically tightens similarity floors and reduces CBW for large vocabularies, maintaining precision without manual tuning.

Computational efficiency. Linear scanning across thousands of terms would introduce latency. The optional BK-tree lookup (ContextBiasingConstants.useBkTree) provides logarithmic-time candidate retrieval, enabling real-time performance even with extensive custom dictionaries.

Implementing Context Biasing in Swift

The following end-to-end example demonstrates enabling context biasing in a streaming session using FluidAudio's public API:

import FluidAudio

// 1️⃣  Load a custom vocabulary file (JSON format produced by FluidAudio tooling)
let vocabURL = URL(fileURLWithPath: "/path/to/myDomainVocab.json")
let customVocab = try CustomVocabularyContext.load(from: vocabURL)

// 2️⃣  Download the CTC models (once per app run)
let ctcModels = try await CtcModels.downloadAndLoad()

// 3️⃣  Create a streaming manager with the default configuration
let streamingMgr = StreamingAsrManager()

// 4️⃣  Enable vocabulary boosting – we keep the default adaptive config
try await streamingMgr.configureVocabularyBoosting(
    vocabulary: customVocab,
    ctcModels: ctcModels,
    config: nil               // uses ContextBiasingConstants.rescorerConfig(...)
)

// 5️⃣  Start streaming from the microphone
try await streamingMgr.start(source: .microphone)

// 6️⃣  Feed audio buffers from AVAudioEngine (example)
audioEngine?.inputNode.installTap(
    onBus: 0,
    bufferSize: 2048,
    format: audioEngine?.inputNode.outputFormat(forBus: 0)
) { buffer, _ in
    streamingMgr.streamAudio(buffer)
}

// 7️⃣  Receive live updates
let updates = streamingMgr.transcriptionUpdates
for await update in updates {
    print("[\(update.isConfirmed ? "✔︎" : "…")] \(update.text)")
}

// 8️⃣  When finished, obtain the final, bias‑aware transcript
let finalTranscript = try await streamingMgr.finish()
print("✅ Final: \(finalTranscript)")

Key implementation details from the fluidinference/fluidaudio source:

• configureVocabularyBoosting instantiates a VocabularyRescorer that internally references ContextBiasingConstants for CBW and threshold values
• The updates stream delivers bias-applied text for every confirmed chunk, allowing UI components to display domain-specific terms immediately upon detection
• The final finish() call guarantees that all boosted terms surviving the rescoring process appear in the complete transcript

Summary

• Context biasing in FluidAudio operates as a shallow-fusion rescoring layer that modifies CTC log-probabilities during streaming decoding, not during model training.
• The system combines three components: CustomVocabularyContext for data management, ContextBiasingConstants for algorithmic tuning, and VocabularyRescorer for execution.
• Adaptive CBW scaling automatically adjusts boost strength based on term length, compensating for CTC's multi-token penalty.
• Hierarchical similarity guards (Levenshtein-based thresholds) prevent false positives while allowing fuzzy matches for domain terms.
• Vocabulary-size-aware configuration automatically tightens precision parameters when large custom dictionaries are loaded.
• The implementation requires no model retraining—users simply load a JSON vocabulary and call configureVocabularyBoosting(...) on their StreamingAsrManager instance.

Frequently Asked Questions

What is context biasing in ASR?

Context biasing is a decoding-time technique that increases the probability of specific words or phrases appearing in transcription output. In FluidAudio, the system adds a context biasing weight (CBW) to the log-probability of domain-specific terms during CTC rescoring, effectively telling the decoder "prefer these words when acoustic evidence is ambiguous." This happens without modifying the underlying acoustic model weights, making it suitable for runtime customization.

How does FluidAudio handle large custom vocabularies?

FluidAudio addresses large-vocabulary challenges through ContextBiasingConstants.rescorerConfig(forVocabSize:). As vocabulary size increases, the system automatically reduces the CBW and raises the minSimilarity threshold to maintain precision. Additionally, when ContextBiasingConstants.useBkTree is enabled, the VocabularyRescorer uses a BK-tree data structure for O(log V) fuzzy matching instead of linear scanning, keeping latency low even with thousands of domain terms.

What are the similarity thresholds for?

The hierarchical similarity thresholds—minSimilarityFloor, shortWordSimilarity, and stopwordSpanSimilarity—act as Levenshtein-based guards in VocabularyRescorer.ctcTokenRescore(...). They ensure that a boosted term only replaces the original hypothesis if the strings are sufficiently alike. This prevents false substitutions where acoustically similar but semantically different words (like "right" and "write") might otherwise trigger incorrect bias injections.

Does context biasing require retraining the acoustic model?

No. FluidAudio’s context biasing is a post-processing rescoring technique applied during the CTC decoding phase. The acoustic model remains unchanged; instead, the VocabularyRescorer manipulates the token log-probabilities emitted by the model. This architecture allows users to swap domain vocabularies instantly without the computational cost or data requirements of fine-tuning or retraining the neural network.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}