FluidAudio Offline Diarization Pipeline Architecture: Core ML Segmentation and VBx Clustering in Swift

The FluidAudio offline diarization pipeline is a fully-asynchronous, on-device processing chain that converts raw audio into speaker-labeled time segments using pre-compiled Core ML models, concurrent segmentation and embedding extraction, agglomerative hierarchical clustering (AHC), and Variational Bayes (VBx) refinement.

The FluidAudio repository provides a complete Swift-native speaker diarization solution that processes audio entirely on Apple devices. The offline diarization pipeline architecture employs a modular, stage-based design where each phase—from neural segmentation to probabilistic clustering—is handled by dedicated types operating on pre-downloaded and pre-warmed Core ML models.

Core Stages of the Offline Diarization Pipeline

The pipeline is implemented as a sequence of discrete stages, each handled by dedicated types in the Sources/FluidAudio/Diarizer/Offline/ directory. All heavy computation uses Core ML models that are downloaded, compiled, and pre-warmed before processing begins.

Model Preparation and Pre-warming

Before audio arrives, OfflineDiarizerManager.swift orchestrates model readiness through prepareModels(...). This method downloads and caches three Core ML models: segmentation, fbank+embedding, and PLDA. To eliminate first-call latency, prewarmModelsIfNeeded(_:) executes prewarmSegmentationModel(_:) and prewarmEmbeddingStack(models:) to run dummy predictions, ensuring the models are resident in memory and thermally prepared.

Concurrent Audio Framing and Segmentation

OfflineSegmentationProcessor.swift implements process(audioSource:segmentationModel:config:chunkHandler:), which creates overlapping windows based on samplesPerWindow and samplesPerStep configuration values. The method produces an AsyncThrowingStream of SegmentationChunks, feeding the segmentation model to generate log-probability chunks and speaker-weight matrices for each window.

Speaker Embedding Extraction

OfflineEmbeddingExtractor.swift handles extractEmbeddings(audioSource:segmentationStream:), consuming the segmentation stream to compute 256-dimensional speaker embeddings and 128-dimensional PLDA-rho vectors for every detected segment. These embeddings form the feature set for subsequent clustering operations.

Clustering: AHC Warm-start and VBx Refinement

The clustering phase combines two algorithms for accuracy and speed. First, OfflineDiarizerManager.swift calls selectTrainingEmbeddings(timedEmbeddings:) (line 20) to filter NaN and infinite values, then executes AHCClustering().cluster(embeddingFeatures:threshold:) (line 24) for agglomerative hierarchical clustering to establish an initial speaker count.

Next, VBxClustering.swift performs refinement via VBxClustering(config:pldaTransform:).refine(rhoFeatures:initialClusters:). This Variational Bayes implementation applies PLDA whitening and EM iterations, respecting speaker-count constraints from the configuration. The output soft posteriors (gamma, pi) are converted to hard centroids by computeCentroids(trainingEmbeddings:vbxOutput:initialClusters:) (line 42).

Segment Reconstruction and Result Packaging

OfflineReconstruction.swift executes buildSegments(segmentation:hardClusters:centroids:) to align hard speaker assignments back to the original timeline, merging adjacent chunks from identical speakers. The buildSpeakerDatabase(segments:) method constructs a SpeakerDatabase containing metadata. Finally, the pipeline returns a DiarizationResult containing segments, speaker database, and PipelineTimings statistics.

Data Flow and Concurrency Model

The orchestration in OfflineDiarizerManager.process(_:) implements a producer-consumer pattern using Swift's structured concurrency:

1. Entry validation – Ensures models are ready via prepareModels and validates OfflineDiarizerConfig.

2. Stream creation – Establishes an AsyncThrowingStream of SegmentationChunks.

3. Concurrent tasks – Launches two parallel Tasks:

   • Segmentation task – Consumes the audio source, feeds windows to the segmentation model, and pushes chunks into the stream via OfflineSegmentationProcessor.swift.
   • Embedding task – Consumes the same audio source and the segmentation stream, running extractEmbeddings in OfflineEmbeddingExtractor.swift to produce TimedEmbedding arrays.

4. Post-processing – After both tasks complete:

   • Filters embeddings (selectTrainingEmbeddings).
   • Executes AHC → VBx → centroid computation.
   • Invokes OfflineReconstruction to generate final segments.

5. Telemetry – Captures all stage timings in PipelineTimings for performance benchmarking.

Configurable Parameters in OfflineDiarizerConfig

Behavior is controlled through OfflineDiarizerConfig (defined in OfflineDiarizerTypes.swift):

• samplesPerWindow / samplesPerStep – Sliding window size and hop length affecting latency versus accuracy trade-offs.
• segmentation.sampleRate – Target sample rate for resampling input audio.
• clusteringThreshold – Distance threshold for initial AHC clustering.
• clustering – Speaker count constraints (numSpeakers, minSpeakers, maxSpeakers).
• vbx.maxIterations / vbx.convergenceTolerance – EM iteration limits and convergence criteria for VBx refinement.
• embeddingExportPath – Optional filesystem path for CSV export of raw embeddings.

Implementation Examples

Diarize an In-Memory Float Array

import FluidAudio

// Configure the pipeline
let config = OfflineDiarizerConfig.default
let diarizer = OfflineDiarizerManager(config: config)

// Prepare Core ML models (download + compile + prewarm)
try await diarizer.prepareModels()

// Process raw 16 kHz Float samples
let audioSamples: [Float] = // ... load from WAV or buffer
let result = try await diarizer.process(audio: audioSamples)

// Output speaker segments
for segment in result.segments {
    print("Speaker \(segment.speakerId): \(segment.startSec)s - \(segment.endSec)s")
}

Key files: OfflineDiarizerManager.swift (initialization, process(audio:)).

Diarize a Local Audio File

import FluidAudio

let fileURL = URL(fileURLWithPath: "/path/to/meeting.wav")
let manager = OfflineDiarizerManager()
try await manager.prepareModels()

// URL overload uses StreamingAudioSourceFactory for memory-mapped reading
let result = try await manager.process(fileURL)

result.segments.forEach { seg in
    print("[\(seg.startSec)-\(seg.endSec)] Speaker \(seg.speakerId)")
}

Key files: OfflineDiarizerManager.swift – the process(_ url: URL) overload leverages StreamingAudioSourceFactory.

Force a Two-Speaker Solution

var customConfig = OfflineDiarizerConfig.default
customConfig = customConfig.withSpeakers(exactly: 2)  // Enforce 2 speakers
customConfig.clusteringThreshold = 1.2                // Tighter AHC threshold

let manager = OfflineDiarizerManager(config: customConfig)
try await manager.prepareModels()
let diarization = try await manager.process(audioSamples)

Key files: OfflineDiarizerTypes.swift (withSpeakers builder); VBxClustering.swift (constraint handling).

Export Raw Embeddings for Research

var exportConfig = OfflineDiarizerConfig.default
exportConfig.embeddingExportPath = URL(fileURLWithPath: "/tmp/embeddings.csv")

let manager = OfflineDiarizerManager(config: exportConfig)
try await manager.prepareModels()
_ = try await manager.process(audioSamples)  // CSV written automatically

Key files: OfflineDiarizerManager.swift (lines invoking exportEmbeddings when embeddingExportPath is set).

Key Source Files

File	Role	Link
Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift	Orchestrates the pipeline, manages model lifecycle, launches concurrent tasks, performs clustering and reconstruction.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Segmentation/OfflineSegmentationProcessor.swift	Sliding-window inference for the segmentation model, yields SegmentationChunks.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Extraction/OfflineEmbeddingExtractor.swift	Extracts 256-dimensional speaker embeddings and 128-dimensional PLDA-rho vectors.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Clustering/AHCClustering.swift	Agglomerative hierarchical clustering for initial speaker count estimation.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Clustering/VBxClustering.swift	Variational Bayes clustering with PLDA whitening and EM updates.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Utils/OfflineReconstruction.swift	Maps cluster assignments to timestamps, merges adjacent segments, builds SpeakerDatabase.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerTypes.swift	Type definitions including OfflineDiarizerConfig, DiarizationResult, and TimedEmbedding.	View on GitHub
Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerModels.swift	Downloads, compiles, and caches the three Core ML models required by the pipeline.	View on GitHub

Summary

• The FluidAudio offline diarization pipeline is a pure-Swift, fully asynchronous system that performs speaker diarization entirely on-device using Core ML.
• Model lifecycle management in OfflineDiarizerManager.swift ensures zero-latency inference through pre-downloading, compilation, and thermal pre-warming of segmentation, embedding, and PLDA models via prepareModels() and prewarmModelsIfNeeded().
• Concurrent processing uses AsyncThrowingStream to parallelize segmentation (OfflineSegmentationProcessor.swift) and embedding extraction (OfflineEmbeddingExtractor.swift), yielding 256-dimensional speaker vectors without blocking the caller.
• Two-stage clustering first applies agglomerative hierarchical clustering (AHCClustering.swift) for initial speaker estimation, then refines results with Variational Bayes (VBxClustering.swift) incorporating PLDA whitening and EM iterations.
• Flexible configuration via OfflineDiarizerConfig allows control over window sizes, clustering thresholds, speaker count constraints, and optional embedding export for research workflows.

Frequently Asked Questions

How does the FluidAudio offline diarization pipeline eliminate first-call latency?

The pipeline implements an explicit pre-warming strategy in OfflineDiarizerManager.swift. Before processing audio, the prepareModels() method downloads and compiles the three required Core ML models (segmentation, embedding, and PLDA). Subsequently, prewarmModelsIfNeeded(_:) executes prewarmSegmentationModel(_:) and prewarmEmbeddingStack(models:) to run dummy predictions, ensuring the models are resident in memory and thermally prepared before real audio arrives.

What clustering algorithms does the offline diarization pipeline use?

The architecture employs a two-stage clustering strategy. First, AHCClustering.swift performs agglomerative hierarchical clustering on filtered embeddings via AHCClustering().cluster(embeddingFeatures:threshold:) to establish an initial speaker count and rough assignments. Then, VBxClustering.swift performs refinement via VBxClustering(config:pldaTransform:).refine(rhoFeatures:initialClusters:). This Variational Bayes implementation applies PLDA whitening and EM iterations to produce the final soft cluster posteriors that are converted to hard speaker labels.

Can I constrain the number of speakers in the diarization output?

Yes, the pipeline accepts speaker constraints through OfflineDiarizerConfig. You can specify exact, minimum, or maximum speaker counts using methods like withSpeakers(exactly:). These constraints are passed to VBxClustering.swift, where the Variational Bayes algorithm adjusts its convergence criteria and cluster initialization to respect the specified bounds during the refinement phase.

How does the pipeline process audio concurrently?

The architecture uses Swift's structured concurrency to parallelize computation. OfflineDiarizerManager.swift creates an AsyncThrowingStream of SegmentationChunks, then launches two concurrent Tasks: one running OfflineSegmentationProcessor.swift to generate segmentation windows, and another running OfflineEmbeddingExtractor.swift to consume those windows and extract 256-dimensional speaker embeddings. This producer-consumer pattern maximizes throughput while maintaining ordered processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}