# How to Integrate Custom Vocabulary and Word Spotting into ASR Using FluidAudio

> Integrate custom vocabulary and word spotting into ASR with FluidAudio. Use our dual-encoder CTC pipeline to boost domain-specific terms without retraining the base ASR model.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: how-to-guide
- Published: 2026-03-02

---

**FluidAudio implements a dual-encoder CTC-based vocabulary boosting pipeline that runs a secondary CTC encoder in parallel with the primary TDT model to detect and rescore domain-specific terms without retraining the base ASR model.**

The `fluidinference/fluidaudio` repository provides a complete word spotting framework that improves recognition of proper nouns, technical jargon, and product names through runtime vocabulary injection. By combining a lightweight CTC encoder with dynamic programming alignment, the system rescores transcripts based on acoustic evidence from your custom term list.

## Understanding the CTC-Based Vocabulary Boosting Pipeline

FluidAudio’s architecture processes audio through two parallel neural networks. The **TDT (Transducer-DT)** encoder in [`TdtDecoderV3.swift`](https://github.com/fluidinference/fluidaudio/blob/main/TdtDecoderV3.swift) produces the high-quality baseline transcript, while a secondary **CTC** encoder defined in [`CtcModels.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcModels.swift) generates per-frame log-probabilities over a 1024-token vocabulary.

The **CTC tokenizer** ([`CtcTokenizer.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcTokenizer.swift)) converts each custom term into token IDs compatible with the CTC model’s output space. Then, the **keyword spotter** ([`CtcKeywordSpotter.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcKeywordSpotter.swift)) applies the dynamic programming algorithm from the NeMo CTC-WS research (arXiv 2406.07096) via [`CtcDPAlgorithm.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcDPAlgorithm.swift) to scan the log-probability matrix for high-scoring occurrences of your vocabulary terms.

After spotting, the **Vocabulary Rescorer** ([`VocabularyRescorer.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VocabularyRescorer.swift)) aligns detected keyword intervals with TDT transcript words, applies similarity guards using thresholds from [`ContextBiasingConstants.swift`](https://github.com/fluidinference/fluidaudio/blob/main/ContextBiasingConstants.swift), and conditionally replaces transcript segments when the CTC acoustic evidence exceeds the TDT hypothesis.

## Step-by-Step Integration Guide

### Create a Custom Vocabulary File

FluidAudio supports two formats for vocabulary definition. For complex configurations with weights and aliases, use JSON matching the `CustomVocabularyConfig` structure:

```json
{
  "alpha": 0.7,
  "terms": [
    { "text": "NVIDIA", "weight": 12.0 },
    { "text": "TensorRT", "aliases": ["Tensor‑RT"], "weight": 10.0 }
  ],
  "minCtcScore": -14.0,
  "minSimilarity": 0.55
}

```

For simple use cases, create a text file with one term per line, optionally followed by aliases after a colon:

```text
NVIDIA
TensorRT: Tensor‑RT, TensorRT
“Häagen‑Dazs”: Haagen‑Dazs, Hagen‑Das

```

### Load Vocabulary with CustomVocabularyContext

Import the FluidAudio module and initialize the vocabulary context using the loaders in [`CustomVocabularyContext.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CustomVocabularyContext.swift):

```swift
import FluidAudio

// Load from JSON
let vocabURL = URL(fileURLWithPath: "/path/to/vocab.json")
let customVocab = try CustomVocabularyContext.load(from: vocabURL)

// Or load simple text format
// let customVocab = try CustomVocabularyContext.loadFromSimpleFormat(from: vocabURL)

```

The loader sanitizes terms, removes empty entries, and populates missing thresholds using default values from `ContextBiasingConstants`.

### Run Transcription with the Swift API

Pass the vocabulary context to [`AsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/AsrManager.swift) using the `transcribe(_:customVocabulary:)` method:

```swift
// Obtain the shared manager (loads TDT & CTC models lazily)
let asrManager = try await AsrManager.shared

// Perform transcription with boosting enabled
let result = try await asrManager.transcribe(
    audioSamples,
    customVocabulary: customVocab
)

print(result.text)              // "NVIDIA announced TensorRT optimizations"
print(result.ctcAppliedTerms)   // ["NVIDIA", "TensorRT"]

```

Omit the `customVocabulary` parameter to receive the raw TDT transcript without vocabulary boosting.

### Use the CLI for Batch Processing

The command-line interface in [`TranscribeCommand.swift`](https://github.com/fluidinference/fluidaudio/blob/main/TranscribeCommand.swift) exposes the `--custom-vocab` flag for file-mode transcription:

```bash
fluidaudio transcribe audio.wav \
    --custom-vocab path/to/vocab.txt

```

This flag is only accepted in file-mode; the CLI parser rejects it when combined with `--streaming` because the CTC log-probability matrix must be available for the complete audio duration.

### Tune Context Biasing Constants

Adjust detection sensitivity by modifying the static properties in [`ContextBiasingConstants.swift`](https://github.com/fluidinference/fluidaudio/blob/main/ContextBiasingConstants.swift):

```swift
ContextBiasingConstants.defaultMinVocabCtcScore = -10.0  // More permissive
ContextBiasingConstants.defaultCbw = 5.0                 // Stronger context bias weight

```

These constants control the minimum CTC score required for detection, the CBW (Context Biasing Weight) boost applied to matched terms, and similarity thresholds for alignment verification.

## Working with Streaming Audio

The vocabulary boosting pipeline requires the complete CTC log-probability matrix before rescoring can occur. Consequently, [`StreamingAsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/StreamingAsrManager.swift) imposes specific limitations when using `customVocabulary` with live audio sources:

- Only **single-word terms** are reliably boosted across streaming chunks
- Cross-chunk detections for multi-word phrases are disabled
- The rescorer runs on finalized transcript segments rather than the complete audio matrix

For streaming sessions, initialize the manager with vocabulary context but expect reduced effectiveness for compound terms:

```swift
let session = try await asr.startSession(
    source: streamingSource,
    customVocabulary: customVocab  // Limited to single-word boosting
)

```

## Advanced Configuration Options

For deployments with pre-computed tokenization, the JSON vocabulary format accepts `tokenIds` and `ctcTokenIds` fields. When provided, the rescorer bypasses the tokenizer pass in [`CtcTokenizer.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcTokenizer.swift) and uses your pre-computed IDs directly, reducing inference overhead for static vocabularies.

The [`CtcKeywordSpotter.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcKeywordSpotter.swift) class exposes `spotKeywordsWithLogProbs()` for low-level access to the detection algorithm, allowing custom rescoring logic outside the standard `VocabularyRescorer` pipeline.

## Summary

- **Dual-encoder architecture**: FluidAudio runs TDT and CTC encoders in parallel, using the CTC output exclusively for vocabulary detection while preserving the TDT transcript quality.
- **File locations**: Core logic resides in `Sources/FluidAudio/ASR/CustomVocabulary/`, with entry points in [`AsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/AsrManager.swift) and [`TranscribeCommand.swift`](https://github.com/fluidinference/fluidaudio/blob/main/TranscribeCommand.swift).
- **Integration pattern**: Create a vocabulary file → load with `CustomVocabularyContext` → pass to `transcribe(_:customVocabulary:)` or use `--custom-vocab` in CLI.
- **Streaming constraints**: Full vocabulary boosting requires file-mode transcription; streaming supports only single-word terms due to matrix dependency.
- **Tuning controls**: Thresholds in [`ContextBiasingConstants.swift`](https://github.com/fluidinference/fluidaudio/blob/main/ContextBiasingConstants.swift) manage the trade-off between detection sensitivity and false insertion rates.

## Frequently Asked Questions

### Can I use custom vocabulary boosting with live microphone streaming?

Partially. According to the [`StreamingAsrManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/StreamingAsrManager.swift) implementation, streaming mode supports vocabulary boosting only for single-word terms because the CTC log-probability matrix must be available for the entire audio duration to run the dynamic programming algorithm. Multi-word phrase detection is disabled across chunk boundaries to prevent fragmentation errors.

### Do I need to retrain the ASR model to recognize new technical terms?

No. The CTC-based vocabulary boosting pipeline in [`CtcKeywordSpotter.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcKeywordSpotter.swift) operates as a post-processing rescoring layer. The primary TDT encoder remains unchanged; the system simply adjusts the final transcript when the CTC encoder provides stronger acoustic evidence for your supplied terms than the base model’s prediction.

### How does the system handle aliases or alternative spellings?

The [`CustomVocabularyContext.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CustomVocabularyContext.swift) loader accepts alias arrays in JSON vocabulary files (e.g., `"aliases": ["Tensor‑RT"]`). During spotting, the rescorer treats all aliases as equivalent acoustic targets for the primary term, allowing recognition of variations like "Haagen-Dazs" when the vocabulary entry includes "Hagen-Das" as an alias.

### What is the performance impact of enabling vocabulary boosting?

The pipeline adds minimal latency. The CTC encoder runs in parallel with the TDT encoder, and the DP algorithm in [`CtcDPAlgorithm.swift`](https://github.com/fluidinference/fluidaudio/blob/main/CtcDPAlgorithm.swift) processes frames efficiently. The primary cost is memory allocation for the CTC log-probability matrix, which is why the feature is restricted in streaming mode to balance real-time constraints against recognition accuracy.