# FluidAudio Offline Diarization Pipeline Architecture: Core ML Segmentation and VBx Clustering in Swift

> Explore the FluidAudio offline diarization pipeline architecture. Discover how Core ML models, Swift concurrency, and VBx clustering create on-device speaker segmentation for your audio projects.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: architecture
- Published: 2026-03-02

---

**The FluidAudio offline diarization pipeline is a fully-asynchronous, on-device processing chain that converts raw audio into speaker-labeled time segments using pre-compiled Core ML models, concurrent segmentation and embedding extraction, agglomerative hierarchical clustering (AHC), and Variational Bayes (VBx) refinement.**

The **FluidAudio** repository provides a complete Swift-native speaker diarization solution that processes audio entirely on Apple devices. The **offline diarization pipeline** architecture employs a modular, stage-based design where each phase—from neural segmentation to probabilistic clustering—is handled by dedicated types operating on pre-downloaded and pre-warmed Core ML models.

## Core Stages of the Offline Diarization Pipeline

The pipeline is implemented as a sequence of discrete stages, each handled by dedicated types in the `Sources/FluidAudio/Diarizer/Offline/` directory. All heavy computation uses Core ML models that are downloaded, compiled, and pre-warmed before processing begins.

### Model Preparation and Pre-warming

Before audio arrives, [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) orchestrates model readiness through `prepareModels(...)`. This method downloads and caches three Core ML models: segmentation, fbank+embedding, and PLDA. To eliminate first-call latency, `prewarmModelsIfNeeded(_:)` executes `prewarmSegmentationModel(_:)` and `prewarmEmbeddingStack(models:)` to run dummy predictions, ensuring the models are resident in memory and thermally prepared.

### Concurrent Audio Framing and Segmentation

[`OfflineSegmentationProcessor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineSegmentationProcessor.swift) implements `process(audioSource:segmentationModel:config:chunkHandler:)`, which creates overlapping windows based on `samplesPerWindow` and `samplesPerStep` configuration values. The method produces an `AsyncThrowingStream` of `SegmentationChunk`s, feeding the segmentation model to generate log-probability chunks and speaker-weight matrices for each window.

### Speaker Embedding Extraction

[`OfflineEmbeddingExtractor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineEmbeddingExtractor.swift) handles `extractEmbeddings(audioSource:segmentationStream:)`, consuming the segmentation stream to compute **256-dimensional speaker embeddings** and **128-dimensional PLDA-rho vectors** for every detected segment. These embeddings form the feature set for subsequent clustering operations.

### Clustering: AHC Warm-start and VBx Refinement

The clustering phase combines two algorithms for accuracy and speed. First, [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) calls `selectTrainingEmbeddings(timedEmbeddings:)` (line 20) to filter NaN and infinite values, then executes `AHCClustering().cluster(embeddingFeatures:threshold:)` (line 24) for agglomerative hierarchical clustering to establish an initial speaker count.

Next, [`VBxClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VBxClustering.swift) performs refinement via `VBxClustering(config:pldaTransform:).refine(rhoFeatures:initialClusters:)`. This Variational Bayes implementation applies PLDA whitening and EM iterations, respecting speaker-count constraints from the configuration. The output soft posteriors (`gamma`, `pi`) are converted to hard centroids by `computeCentroids(trainingEmbeddings:vbxOutput:initialClusters:)` (line 42).

### Segment Reconstruction and Result Packaging

[`OfflineReconstruction.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineReconstruction.swift) executes `buildSegments(segmentation:hardClusters:centroids:)` to align hard speaker assignments back to the original timeline, merging adjacent chunks from identical speakers. The `buildSpeakerDatabase(segments:)` method constructs a `SpeakerDatabase` containing metadata. Finally, the pipeline returns a `DiarizationResult` containing segments, speaker database, and `PipelineTimings` statistics.

## Data Flow and Concurrency Model

The orchestration in `OfflineDiarizerManager.process(_:)` implements a producer-consumer pattern using Swift's structured concurrency:

1. **Entry validation** – Ensures models are ready via `prepareModels` and validates `OfflineDiarizerConfig`.

2. **Stream creation** – Establishes an `AsyncThrowingStream` of `SegmentationChunk`s.

3. **Concurrent tasks** – Launches two parallel `Task`s:
   - **Segmentation task** – Consumes the audio source, feeds windows to the segmentation model, and pushes chunks into the stream via [`OfflineSegmentationProcessor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineSegmentationProcessor.swift).
   - **Embedding task** – Consumes the same audio source and the segmentation stream, running `extractEmbeddings` in [`OfflineEmbeddingExtractor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineEmbeddingExtractor.swift) to produce `TimedEmbedding` arrays.

4. **Post-processing** – After both tasks complete:
   - Filters embeddings (`selectTrainingEmbeddings`).
   - Executes **AHC** → **VBx** → **centroid computation**.
   - Invokes `OfflineReconstruction` to generate final segments.

5. **Telemetry** – Captures all stage timings in `PipelineTimings` for performance benchmarking.

## Configurable Parameters in OfflineDiarizerConfig

Behavior is controlled through `OfflineDiarizerConfig` (defined in [`OfflineDiarizerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerTypes.swift)):

- **`samplesPerWindow` / `samplesPerStep`** – Sliding window size and hop length affecting latency versus accuracy trade-offs.
- **`segmentation.sampleRate`** – Target sample rate for resampling input audio.
- **`clusteringThreshold`** – Distance threshold for initial AHC clustering.
- **`clustering`** – Speaker count constraints (`numSpeakers`, `minSpeakers`, `maxSpeakers`).
- **`vbx.maxIterations`** / **`vbx.convergenceTolerance`** – EM iteration limits and convergence criteria for VBx refinement.
- **`embeddingExportPath`** – Optional filesystem path for CSV export of raw embeddings.

## Implementation Examples

### Diarize an In-Memory Float Array

```swift
import FluidAudio

// Configure the pipeline
let config = OfflineDiarizerConfig.default
let diarizer = OfflineDiarizerManager(config: config)

// Prepare Core ML models (download + compile + prewarm)
try await diarizer.prepareModels()

// Process raw 16 kHz Float samples
let audioSamples: [Float] = // ... load from WAV or buffer
let result = try await diarizer.process(audio: audioSamples)

// Output speaker segments
for segment in result.segments {
    print("Speaker \(segment.speakerId): \(segment.startSec)s - \(segment.endSec)s")
}

```

*Key files*: [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) (initialization, `process(audio:)`).

### Diarize a Local Audio File

```swift
import FluidAudio

let fileURL = URL(fileURLWithPath: "/path/to/meeting.wav")
let manager = OfflineDiarizerManager()
try await manager.prepareModels()

// URL overload uses StreamingAudioSourceFactory for memory-mapped reading
let result = try await manager.process(fileURL)

result.segments.forEach { seg in
    print("[\(seg.startSec)-\(seg.endSec)] Speaker \(seg.speakerId)")
}

```

*Key files*: [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) – the `process(_ url: URL)` overload leverages `StreamingAudioSourceFactory`.

### Force a Two-Speaker Solution

```swift
var customConfig = OfflineDiarizerConfig.default
customConfig = customConfig.withSpeakers(exactly: 2)  // Enforce 2 speakers
customConfig.clusteringThreshold = 1.2                // Tighter AHC threshold

let manager = OfflineDiarizerManager(config: customConfig)
try await manager.prepareModels()
let diarization = try await manager.process(audioSamples)

```

*Key files*: [`OfflineDiarizerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerTypes.swift) (`withSpeakers` builder); [`VBxClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VBxClustering.swift) (constraint handling).

### Export Raw Embeddings for Research

```swift
var exportConfig = OfflineDiarizerConfig.default
exportConfig.embeddingExportPath = URL(fileURLWithPath: "/tmp/embeddings.csv")

let manager = OfflineDiarizerManager(config: exportConfig)
try await manager.prepareModels()
_ = try await manager.process(audioSamples)  // CSV written automatically

```

*Key files*: [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) (lines invoking `exportEmbeddings` when `embeddingExportPath` is set).

## Key Source Files

| File | Role | Link |
|------|------|------|
| [`Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift) | Orchestrates the pipeline, manages model lifecycle, launches concurrent tasks, performs clustering and reconstruction. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Segmentation/OfflineSegmentationProcessor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Segmentation/OfflineSegmentationProcessor.swift) | Sliding-window inference for the segmentation model, yields `SegmentationChunk`s. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Segmentation/OfflineSegmentationProcessor.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Extraction/OfflineEmbeddingExtractor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Extraction/OfflineEmbeddingExtractor.swift) | Extracts 256-dimensional speaker embeddings and 128-dimensional PLDA-rho vectors. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Extraction/OfflineEmbeddingExtractor.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Clustering/AHCClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Clustering/AHCClustering.swift) | Agglomerative hierarchical clustering for initial speaker count estimation. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Clustering/AHCClustering.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Clustering/VBxClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Clustering/VBxClustering.swift) | Variational Bayes clustering with PLDA whitening and EM updates. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Clustering/VBxClustering.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Utils/OfflineReconstruction.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Utils/OfflineReconstruction.swift) | Maps cluster assignments to timestamps, merges adjacent segments, builds `SpeakerDatabase`. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Utils/OfflineReconstruction.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerTypes.swift) | Type definitions including `OfflineDiarizerConfig`, `DiarizationResult`, and `TimedEmbedding`. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerTypes.swift) |
| [`Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerModels.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerModels.swift) | Downloads, compiles, and caches the three Core ML models required by the pipeline. | [View on GitHub](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerModels.swift) |

## Summary

- The **FluidAudio offline diarization pipeline** is a pure-Swift, fully asynchronous system that performs speaker diarization entirely on-device using Core ML.
- **Model lifecycle management** in [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) ensures zero-latency inference through pre-downloading, compilation, and thermal pre-warming of segmentation, embedding, and PLDA models via `prepareModels()` and `prewarmModelsIfNeeded()`.
- **Concurrent processing** uses `AsyncThrowingStream` to parallelize segmentation ([`OfflineSegmentationProcessor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineSegmentationProcessor.swift)) and embedding extraction ([`OfflineEmbeddingExtractor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineEmbeddingExtractor.swift)), yielding 256-dimensional speaker vectors without blocking the caller.
- **Two-stage clustering** first applies agglomerative hierarchical clustering ([`AHCClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/AHCClustering.swift)) for initial speaker estimation, then refines results with Variational Bayes ([`VBxClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VBxClustering.swift)) incorporating PLDA whitening and EM iterations.
- **Flexible configuration** via `OfflineDiarizerConfig` allows control over window sizes, clustering thresholds, speaker count constraints, and optional embedding export for research workflows.

## Frequently Asked Questions

### How does the FluidAudio offline diarization pipeline eliminate first-call latency?

The pipeline implements an explicit pre-warming strategy in [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift). Before processing audio, the `prepareModels()` method downloads and compiles the three required Core ML models (segmentation, embedding, and PLDA). Subsequently, `prewarmModelsIfNeeded(_:)` executes `prewarmSegmentationModel(_:)` and `prewarmEmbeddingStack(models:)` to run dummy predictions, ensuring the models are resident in memory and thermally prepared before real audio arrives.

### What clustering algorithms does the offline diarization pipeline use?

The architecture employs a two-stage clustering strategy. First, [`AHCClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/AHCClustering.swift) performs agglomerative hierarchical clustering on filtered embeddings via `AHCClustering().cluster(embeddingFeatures:threshold:)` to establish an initial speaker count and rough assignments. Then, [`VBxClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VBxClustering.swift) performs refinement via `VBxClustering(config:pldaTransform:).refine(rhoFeatures:initialClusters:)`. This Variational Bayes implementation applies PLDA whitening and EM iterations to produce the final soft cluster posteriors that are converted to hard speaker labels.

### Can I constrain the number of speakers in the diarization output?

Yes, the pipeline accepts speaker constraints through `OfflineDiarizerConfig`. You can specify exact, minimum, or maximum speaker counts using methods like `withSpeakers(exactly:)`. These constraints are passed to [`VBxClustering.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VBxClustering.swift), where the Variational Bayes algorithm adjusts its convergence criteria and cluster initialization to respect the specified bounds during the refinement phase.

### How does the pipeline process audio concurrently?

The architecture uses Swift's structured concurrency to parallelize computation. [`OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineDiarizerManager.swift) creates an `AsyncThrowingStream` of `SegmentationChunk`s, then launches two concurrent `Task`s: one running [`OfflineSegmentationProcessor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineSegmentationProcessor.swift) to generate segmentation windows, and another running [`OfflineEmbeddingExtractor.swift`](https://github.com/fluidinference/fluidaudio/blob/main/OfflineEmbeddingExtractor.swift) to consume those windows and extract 256-dimensional speaker embeddings. This producer-consumer pattern maximizes throughput while maintaining ordered processing.