How to Integrate Custom Vocabulary and Word Spotting into ASR Using FluidAudio

FluidAudio implements a dual-encoder CTC-based vocabulary boosting pipeline that runs a secondary CTC encoder in parallel with the primary TDT model to detect and rescore domain-specific terms without retraining the base ASR model.

The fluidinference/fluidaudio repository provides a complete word spotting framework that improves recognition of proper nouns, technical jargon, and product names through runtime vocabulary injection. By combining a lightweight CTC encoder with dynamic programming alignment, the system rescores transcripts based on acoustic evidence from your custom term list.

Understanding the CTC-Based Vocabulary Boosting Pipeline

FluidAudio’s architecture processes audio through two parallel neural networks. The TDT (Transducer-DT) encoder in TdtDecoderV3.swift produces the high-quality baseline transcript, while a secondary CTC encoder defined in CtcModels.swift generates per-frame log-probabilities over a 1024-token vocabulary.

The CTC tokenizer (CtcTokenizer.swift) converts each custom term into token IDs compatible with the CTC model’s output space. Then, the keyword spotter (CtcKeywordSpotter.swift) applies the dynamic programming algorithm from the NeMo CTC-WS research (arXiv 2406.07096) via CtcDPAlgorithm.swift to scan the log-probability matrix for high-scoring occurrences of your vocabulary terms.

After spotting, the Vocabulary Rescorer (VocabularyRescorer.swift) aligns detected keyword intervals with TDT transcript words, applies similarity guards using thresholds from ContextBiasingConstants.swift, and conditionally replaces transcript segments when the CTC acoustic evidence exceeds the TDT hypothesis.

Step-by-Step Integration Guide

Create a Custom Vocabulary File

FluidAudio supports two formats for vocabulary definition. For complex configurations with weights and aliases, use JSON matching the CustomVocabularyConfig structure:

{
  "alpha": 0.7,
  "terms": [
    { "text": "NVIDIA", "weight": 12.0 },
    { "text": "TensorRT", "aliases": ["Tensor‑RT"], "weight": 10.0 }
  ],
  "minCtcScore": -14.0,
  "minSimilarity": 0.55
}

For simple use cases, create a text file with one term per line, optionally followed by aliases after a colon:

NVIDIA
TensorRT: Tensor‑RT, TensorRT
“Häagen‑Dazs”: Haagen‑Dazs, Hagen‑Das

Load Vocabulary with CustomVocabularyContext

Import the FluidAudio module and initialize the vocabulary context using the loaders in CustomVocabularyContext.swift:

import FluidAudio

// Load from JSON
let vocabURL = URL(fileURLWithPath: "/path/to/vocab.json")
let customVocab = try CustomVocabularyContext.load(from: vocabURL)

// Or load simple text format
// let customVocab = try CustomVocabularyContext.loadFromSimpleFormat(from: vocabURL)

The loader sanitizes terms, removes empty entries, and populates missing thresholds using default values from ContextBiasingConstants.

Run Transcription with the Swift API

Pass the vocabulary context to AsrManager.swift using the transcribe(_:customVocabulary:) method:

// Obtain the shared manager (loads TDT & CTC models lazily)
let asrManager = try await AsrManager.shared

// Perform transcription with boosting enabled
let result = try await asrManager.transcribe(
    audioSamples,
    customVocabulary: customVocab
)

print(result.text)              // "NVIDIA announced TensorRT optimizations"
print(result.ctcAppliedTerms)   // ["NVIDIA", "TensorRT"]

Omit the customVocabulary parameter to receive the raw TDT transcript without vocabulary boosting.

Use the CLI for Batch Processing

The command-line interface in TranscribeCommand.swift exposes the --custom-vocab flag for file-mode transcription:

fluidaudio transcribe audio.wav \
    --custom-vocab path/to/vocab.txt

This flag is only accepted in file-mode; the CLI parser rejects it when combined with --streaming because the CTC log-probability matrix must be available for the complete audio duration.

Tune Context Biasing Constants

Adjust detection sensitivity by modifying the static properties in ContextBiasingConstants.swift:

ContextBiasingConstants.defaultMinVocabCtcScore = -10.0  // More permissive
ContextBiasingConstants.defaultCbw = 5.0                 // Stronger context bias weight

These constants control the minimum CTC score required for detection, the CBW (Context Biasing Weight) boost applied to matched terms, and similarity thresholds for alignment verification.

Working with Streaming Audio

The vocabulary boosting pipeline requires the complete CTC log-probability matrix before rescoring can occur. Consequently, StreamingAsrManager.swift imposes specific limitations when using customVocabulary with live audio sources:

• Only single-word terms are reliably boosted across streaming chunks
• Cross-chunk detections for multi-word phrases are disabled
• The rescorer runs on finalized transcript segments rather than the complete audio matrix

For streaming sessions, initialize the manager with vocabulary context but expect reduced effectiveness for compound terms:

let session = try await asr.startSession(
    source: streamingSource,
    customVocabulary: customVocab  // Limited to single-word boosting
)

Advanced Configuration Options

For deployments with pre-computed tokenization, the JSON vocabulary format accepts tokenIds and ctcTokenIds fields. When provided, the rescorer bypasses the tokenizer pass in CtcTokenizer.swift and uses your pre-computed IDs directly, reducing inference overhead for static vocabularies.

The CtcKeywordSpotter.swift class exposes spotKeywordsWithLogProbs() for low-level access to the detection algorithm, allowing custom rescoring logic outside the standard VocabularyRescorer pipeline.

Summary

• Dual-encoder architecture: FluidAudio runs TDT and CTC encoders in parallel, using the CTC output exclusively for vocabulary detection while preserving the TDT transcript quality.
• File locations: Core logic resides in Sources/FluidAudio/ASR/CustomVocabulary/, with entry points in AsrManager.swift and TranscribeCommand.swift.
• Integration pattern: Create a vocabulary file → load with CustomVocabularyContext → pass to transcribe(_:customVocabulary:) or use --custom-vocab in CLI.
• Streaming constraints: Full vocabulary boosting requires file-mode transcription; streaming supports only single-word terms due to matrix dependency.
• Tuning controls: Thresholds in ContextBiasingConstants.swift manage the trade-off between detection sensitivity and false insertion rates.

Frequently Asked Questions

Can I use custom vocabulary boosting with live microphone streaming?

Partially. According to the StreamingAsrManager.swift implementation, streaming mode supports vocabulary boosting only for single-word terms because the CTC log-probability matrix must be available for the entire audio duration to run the dynamic programming algorithm. Multi-word phrase detection is disabled across chunk boundaries to prevent fragmentation errors.

Do I need to retrain the ASR model to recognize new technical terms?

No. The CTC-based vocabulary boosting pipeline in CtcKeywordSpotter.swift operates as a post-processing rescoring layer. The primary TDT encoder remains unchanged; the system simply adjusts the final transcript when the CTC encoder provides stronger acoustic evidence for your supplied terms than the base model’s prediction.

How does the system handle aliases or alternative spellings?

The CustomVocabularyContext.swift loader accepts alias arrays in JSON vocabulary files (e.g., "aliases": ["Tensor‑RT"]). During spotting, the rescorer treats all aliases as equivalent acoustic targets for the primary term, allowing recognition of variations like "Haagen-Dazs" when the vocabulary entry includes "Hagen-Das" as an alias.

What is the performance impact of enabling vocabulary boosting?

The pipeline adds minimal latency. The CTC encoder runs in parallel with the TDT encoder, and the DP algorithm in CtcDPAlgorithm.swift processes frames efficiently. The primary cost is memory allocation for the CTC log-probability matrix, which is why the feature is restricted in streaming mode to balance real-time constraints against recognition accuracy.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}