How to Configure the Offline Diarization Pipeline with VBx Clustering in FluidAudio

Configure the offline diarization pipeline with VBx clustering by initializing OfflineDiarizerConfig with custom thresholds, warm-start parameters, and optional speaker constraints, then passing it to OfflineDiarizerManager for processing.

FluidAudio provides a high-performance offline diarization system that uses Variational Bayes (VBx) clustering to group speaker embeddings into distinct clusters. The pipeline is fully configurable through the OfflineDiarizerConfig struct defined in Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerTypes.swift. When you configure the offline diarization pipeline with VBx clustering, you control everything from the initial Agglomerative Hierarchical Clustering (AHC) warm-start to the final Expectation-Maximization (EM) iteration limits.

Pipeline Architecture and VBx Integration

The offline diarization pipeline consists of three sequential stages executed by OfflineDiarizerManager.process(...):

1. Segmentation – detects speech activity and creates short audio chunks.
2. Embedding extraction – converts each chunk into a 256‑dimensional speaker embedding plus a 128‑dimensional PLDA "rho" vector.
3. Speaker clustering – groups embeddings using the VBx (Variational Bayes) algorithm implemented in VBxClustering.refine(...).

According to the fluidinference/fluidaudio source code, the VBxClustering class in Sources/FluidAudio/Diarizer/Offline/Clustering/VBxClustering.swift performs the full VBx EM loop, returning a VBxOutput containing the gamma matrix, mixture weights (pi), hard assignments, and ELBO history.

Core Configuration Parameters

All VBx-related knobs are exposed through OfflineDiarizerConfig. The following properties control clustering behavior:

Property	Description	Default (community‑1)
clustering.threshold	Euclidean distance threshold for the initial AHC warm-start	0.6
clustering.warmStartFa	VBx warm-start precision parameter	0.07
clustering.warmStartFb	VBx warm-start recall parameter	0.8
vbx.maxIterations	Maximum EM iterations for VBx	20
vbx.convergenceTolerance	ELBO-based convergence epsilon	1e‑4
clustering.minSpeakers	Minimum speaker constraint	nil
clustering.maxSpeakers	Maximum speaker constraint	nil
clustering.numSpeakers	Exact speaker count override	nil

Configuration Flow Through the Pipeline

CLI Entry Point

The ProcessCommand in Sources/FluidAudioCLI/Commands/ProcessCommand.swift (lines 52‑63) parses --mode offline and constructs an OfflineDiarizerConfig with user-supplied thresholds and optional embedding export paths.

Manager Orchestration

OfflineDiarizerManager.process(...) in Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift (lines 241‑257) validates the configuration via config.validate(), loads CoreML models, runs segmentation and embedding extraction in parallel, and invokes VBxClustering.refineWithConstraints(...).

VBx Clustering Implementation

The VBxClustering.refine(...) method at lines 40‑64 of VBxClustering.swift executes the VBx EM loop. If speaker constraints are present, refineWithConstraints performs an additional K-Means re-clustering step using KMeansClustering, marking the output as adjusted via VBxOutput.wasAdjusted.

Speaker Count Constraints

The SpeakerCountConstraints struct in Sources/FluidAudio/Diarizer/Offline/Clustering/SpeakerCountConstraints.swift handles logic for min/max/exact speaker enforcement, automatically triggering K-Means when VBx output falls outside specified ranges.

Practical Configuration Examples

Use default VBx settings

let cfg = OfflineDiarizerConfig()

Adjust clustering threshold

Lower values create more initial clusters for finer separation:

let cfg = OfflineDiarizerConfig(clusteringThreshold: 0.5)

Tune VBx warm-start parameters

Higher Fa improves precision; higher Fb improves recall:

var cfg = OfflineDiarizerConfig()
cfg.Fa = 0.1
cfg.Fb = 0.6

Increase EM iterations

Allow more convergence steps at the cost of CPU time:

var cfg = OfflineDiarizerConfig()
cfg.maxVBxIterations = 30

Enforce speaker range

Force re-clustering if detected speakers fall outside [2, 5]:

let cfg = OfflineDiarizerConfig().withSpeakers(min: 2, max: 5)

Force exact speaker count

Override AHC/VBx estimates entirely:

let cfg = OfflineDiarizerConfig().withSpeakers(exactly: 3)

Export embeddings for debugging

let cfg = OfflineDiarizerConfig(embeddingExportPath: "/tmp/embeddings.json")

Complete integration example

let manager = OfflineDiarizerManager(config: cfg)
let models = try await OfflineDiarizerModels.load(from: modelDir)
manager.initialize(models: models)
let result = try await manager.process(audioSource: diskSource,
                                      audioLoadingSeconds: loadDuration)

Summary

• Configure the offline diarization pipeline with VBx clustering using the OfflineDiarizerConfig struct in OfflineDiarizerTypes.swift.
• Adjust clustering.threshold, Fa, and Fb to control the AHC warm-start and VBx precision/recall trade-offs.
• Set maxVBxIterations and convergenceTolerance to balance accuracy against computational cost.
• Use withSpeakers() constraints to enforce minimum, maximum, or exact speaker counts via SpeakerCountConstraints.
• Pass the configuration to OfflineDiarizerManager.process(...) to execute the full pipeline including segmentation, embedding extraction, and VBx clustering.

Frequently Asked Questions

What is the default VBx clustering threshold in FluidAudio?

The default Euclidean distance threshold for the initial Agglomerative Hierarchical Clustering warm-start is 0.6, defined in OfflineDiarizerConfig. Lowering this value creates more initial clusters and can improve speaker separation for difficult audio.

How do I force FluidAudio to detect exactly 3 speakers?

Use the withSpeakers(exactly:) convenience method when building your configuration: let cfg = OfflineDiarizerConfig().withSpeakers(exactly: 3). This overrides the VBx estimate and applies K-Means clustering to enforce the exact count.

Where does the VBx clustering logic live in the source code?

The core VBx algorithm is implemented in Sources/FluidAudio/Diarizer/Offline/Clustering/VBxClustering.swift, specifically in the refine(...) method at lines 40‑64. The orchestration happens in OfflineDiarizerManager.swift at lines 241‑257.

Can I export speaker embeddings for external analysis?

Yes. Set the embeddingExportPath property in OfflineDiarizerConfig to a file path (e.g., /tmp/embeddings.json). After processing, OfflineDiarizerManager writes a JSON file containing each embedding, chunk indices, speaker indices, and final cluster assignments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}