# How Sortformer Compares to Pyannote for Speaker Diarization in FluidAudio

> Compare Sortformer vs Pyannote for speaker diarization in FluidAudio. Discover Sortformer's ultra-low latency streaming and Pyannote's flexible offline capabilities for your audio needs.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: deep-dive
- Published: 2026-03-02

---

**Sortformer delivers ultra-low latency streaming diarization for up to four speakers through a single end-to-end neural network, while Pyannote provides flexible offline diarization supporting unlimited speaker counts via a modular pipeline.**

Both Sortformer and Pyannote are speaker diarization solutions shipped with the FluidAudio repository, but they target fundamentally different use cases and operational constraints. Understanding their architectural differences is essential for selecting the right approach for real-time transcription versus batch processing workflows.

## Core Architectural Differences

### Sortformer: End-to-End Streaming Design

Sortformer is implemented as a **single neural network** that directly predicts per-frame speaker probabilities for four fixed speaker slots. Located in `Sources/FluidAudio/Diarizer/Sortformer/`, this architecture eliminates separate stages for voice activity detection (VAD), segmentation, and clustering. The model maintains a **Speaker Cache** and **FIFO queue** to provide temporal context across audio chunks, enabling true streaming inference.

### Pyannote: Modular Pipeline Approach

Pyannote, exposed through `DiarizerManager` in [`Sources/FluidAudio/Diarizer/Core/DiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Core/DiarizerManager.swift), follows a classic multi-stage pipeline: VAD → segmentation → speaker-embedding extraction (WeSpeaker) → clustering with VBx. This modular design in [`Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift) allows unlimited speaker counts but requires offline processing or higher-latency online variants.

## Streaming Capabilities and Latency

Sortformer is explicitly designed for **real-time inference** with tunable latency characteristics. According to [`Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift), the default configuration operates at approximately **1 second latency**, while high-quality mode extends to approximately **30 seconds** for improved accuracy. The streaming state management defined in [`Sources/FluidAudio/Diarizer/Sortformer/SortformerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerTypes.swift) handles frame-by-frame processing through `SortformerStreamingState` and `SortformerChunkResult`.

Pyannote supports online diarization through `DiarizerManager`, but the pipeline remains built around offline-oriented components. It cannot achieve the ultra-low latency of Sortformer because each stage requires complete context windows for VAD and clustering operations.

## Speaker Handling Constraints

**Sortformer** is limited to **four fixed speaker slots** defined statically in the CoreML model shapes. Additional speakers beyond this limit are either merged or missed entirely. Furthermore, Sortformer does **not** maintain persistent speaker embeddings across recordings—each session starts fresh without speaker identity history.

**Pyannote** supports an **unlimited number of speakers** through its clustering stage. The system retains speaker embeddings, allowing the same speaker label to persist across multiple recordings or meeting segments, which is critical for longitudinal analysis.

## Performance Characteristics and Robustness

Sortformer demonstrates superior **noise robustness** and handles overlapping speech effectively due to training on noisy, real-world data. Its typical error mode involves **missed or silent speech**, where quiet voices may be ignored completely.

Pyannote excels in clean acoustic conditions but **degrades in heavy background noise** where the segmentation stage can be fooled by non-speech events. Its primary error mode is **mis-labeling**, where clustering algorithms assign incorrect speaker identities to segments.

Regarding accuracy, Sortformer achieves approximately **32% Diarization Error Rate (DER)** on typical meetings with four or fewer speakers, representing realistic production performance. Pyannote can achieve lower DER on specific benchmarks like AMI when aggressively tuned, but these optimized settings often fail to generalize to real-world audio distributions.

## Model Configuration and Licensing

**Sortformer** distributes as a single end-to-end CoreML model (~90 MB) under the NVIDIA Open Model License with no usage restrictions. Configuration occurs through `SortformerConfig`, where parameters like FIFO depth and right-context must match static CoreML shapes—these cannot be modified at runtime without model regeneration.

**Pyannote** requires multiple bundled models (segmentation, embedding, clustering) with licensing following individual component terms. Configuration involves several tuning knobs including VAD thresholds and clustering hyper-parameters that require dataset-specific calibration.

## Implementation Examples

### Real-Time Streaming with Sortformer

```swift
import FluidAudio

let diarizer = SortformerDiarizer(config: .default)
let models = try await SortformerModels.loadFromHuggingFace(config: .default)
try await diarizer.initialize(models: models)

// Process audio chunks from microphone or network stream
for chunk in audioChunkStream {
    if let result = try? diarizer.processSamples(chunk) {
        for frame in 0..<result.frameCount {
            for speaker in 0..<4 {
                let prob = result.getSpeakerPrediction(speaker: speaker, frame: frame)
                // Drive real-time UI updates or downstream logic
            }
        }
        // Access timeline via SortformerTimeline for visualization
        updateSpeakerDisplay(diarizer.timeline)
    }
}

```

### Batch Processing with Pyannote

```swift
import FluidAudio

let manager = DiarizerManager(config: .default)
let models = try await DiarizerModels.downloadIfNeeded()
manager.initialize(models: models)

let samples = try AudioConverter().resampleAudioFile(path: "meeting.wav")
let result = try manager.performCompleteDiarization(samples)

// Iterate over finalized segments
for segment in result.segments {
    print("\(segment.speakerId): \(segment.startTimeSeconds)s → \(segment.endTimeSeconds)s")
}

```

### Comparing Both Approaches

```swift
let audio = try AudioConverter().resampleAudioFile(path: "meeting.wav")

// Sortformer streaming approach
let sortformer = SortformerDiarizer(config: .default)
let sortModels = try await SortformerModels.loadFromHuggingFace(config: .default)
try await sortformer.initialize(models: sortModels)
let sortTimeline = try sortformer.processComplete(audio)

// Pyannote offline approach
let pyannote = DiarizerManager(config: .default)
let pyModels = try await DiarizerModels.downloadIfNeeded()
pyannote.initialize(models: pyModels)
let pyResult = try pyannote.performCompleteDiarization(audio)

// Calculate DER against reference RTTM for quantitative comparison
// let derSort = computeDER(referenceRTTM, sortTimeline.rttm)
// let derPy = computeDER(referenceRTTM, pyResult.rttm)

```

Post-processing for Sortformer occurs in [`Sources/FluidAudio/Diarizer/Sortformer/SortformerTimeline.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerTimeline.swift), which converts frame-level probabilities into finalized speaker segments suitable for RTTM export.

## Summary

- **Sortformer** trades speaker count flexibility for ultra-low latency and superior noise handling in streaming scenarios.
- **Pyannote** (via `DiarizerManager`) offers unlimited speaker support and persistent identity tracking but requires higher latency and careful tuning.
- Sortformer configurations in [`SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerConfig.swift) must match static CoreML shapes and cannot change at runtime.
- Pyannote's modular pipeline in [`DiarizerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/DiarizerManager.swift) supports batch processing where speaker count varies or historical identity matching is required.
- Both solutions ship with FluidAudio, selecting Sortformer for real-time meeting transcription and Pyannote for archival analysis.

## Frequently Asked Questions

### Can Sortformer handle more than four speakers?

No. Sortformer is architecturally constrained to four fixed speaker slots defined in the CoreML model weights. Additional speakers are merged into existing slots or missed entirely. For scenarios requiring unlimited speaker detection, use Pyannote's `DiarizerManager` which clusters embeddings dynamically without predefined limits.

### Why does Sortformer forget speakers between recordings?

Sortformer operates as a stateful streaming model without persistent embedding storage. The Speaker Cache and FIFO queue defined in [`SortformerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerTypes.swift) maintain context only within a single session. Pyannote retains speaker embeddings across recordings, enabling consistent labeling for recurring participants.

### Which approach yields better accuracy in noisy environments?

Sortformer generally outperforms Pyannote in noisy conditions and high-overlap scenarios because it was trained end-to-end on noisy, reverberant data. Pyannote's segmentation stage often degrades when background noise masks speech boundaries, leading to fragmentation errors that propagate through the clustering stage.

### How do I reduce latency in Sortformer without recompiling the model?

Adjust the `SortformerConfig` parameters exposed in [`Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift). The default configuration targets approximately 1 second latency, while setting `highQuality: false` and reducing right-context parameters can lower this further. However, these parameters must match the compiled CoreML model shapes—significant architectural changes require model regeneration.