How Sortformer Compares to Pyannote for Speaker Diarization in FluidAudio

Sortformer delivers ultra-low latency streaming diarization for up to four speakers through a single end-to-end neural network, while Pyannote provides flexible offline diarization supporting unlimited speaker counts via a modular pipeline.

Both Sortformer and Pyannote are speaker diarization solutions shipped with the FluidAudio repository, but they target fundamentally different use cases and operational constraints. Understanding their architectural differences is essential for selecting the right approach for real-time transcription versus batch processing workflows.

Core Architectural Differences

Sortformer: End-to-End Streaming Design

Sortformer is implemented as a single neural network that directly predicts per-frame speaker probabilities for four fixed speaker slots. Located in Sources/FluidAudio/Diarizer/Sortformer/, this architecture eliminates separate stages for voice activity detection (VAD), segmentation, and clustering. The model maintains a Speaker Cache and FIFO queue to provide temporal context across audio chunks, enabling true streaming inference.

Pyannote: Modular Pipeline Approach

Pyannote, exposed through DiarizerManager in Sources/FluidAudio/Diarizer/Core/DiarizerManager.swift, follows a classic multi-stage pipeline: VAD → segmentation → speaker-embedding extraction (WeSpeaker) → clustering with VBx. This modular design in Sources/FluidAudio/Diarizer/Offline/Core/OfflineDiarizerManager.swift allows unlimited speaker counts but requires offline processing or higher-latency online variants.

Streaming Capabilities and Latency

Sortformer is explicitly designed for real-time inference with tunable latency characteristics. According to Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift, the default configuration operates at approximately 1 second latency, while high-quality mode extends to approximately 30 seconds for improved accuracy. The streaming state management defined in Sources/FluidAudio/Diarizer/Sortformer/SortformerTypes.swift handles frame-by-frame processing through SortformerStreamingState and SortformerChunkResult.

Pyannote supports online diarization through DiarizerManager, but the pipeline remains built around offline-oriented components. It cannot achieve the ultra-low latency of Sortformer because each stage requires complete context windows for VAD and clustering operations.

Speaker Handling Constraints

Sortformer is limited to four fixed speaker slots defined statically in the CoreML model shapes. Additional speakers beyond this limit are either merged or missed entirely. Furthermore, Sortformer does not maintain persistent speaker embeddings across recordings—each session starts fresh without speaker identity history.

Pyannote supports an unlimited number of speakers through its clustering stage. The system retains speaker embeddings, allowing the same speaker label to persist across multiple recordings or meeting segments, which is critical for longitudinal analysis.

Performance Characteristics and Robustness

Sortformer demonstrates superior noise robustness and handles overlapping speech effectively due to training on noisy, real-world data. Its typical error mode involves missed or silent speech, where quiet voices may be ignored completely.

Pyannote excels in clean acoustic conditions but degrades in heavy background noise where the segmentation stage can be fooled by non-speech events. Its primary error mode is mis-labeling, where clustering algorithms assign incorrect speaker identities to segments.

Regarding accuracy, Sortformer achieves approximately 32% Diarization Error Rate (DER) on typical meetings with four or fewer speakers, representing realistic production performance. Pyannote can achieve lower DER on specific benchmarks like AMI when aggressively tuned, but these optimized settings often fail to generalize to real-world audio distributions.

Model Configuration and Licensing

Sortformer distributes as a single end-to-end CoreML model (~90 MB) under the NVIDIA Open Model License with no usage restrictions. Configuration occurs through SortformerConfig, where parameters like FIFO depth and right-context must match static CoreML shapes—these cannot be modified at runtime without model regeneration.

Pyannote requires multiple bundled models (segmentation, embedding, clustering) with licensing following individual component terms. Configuration involves several tuning knobs including VAD thresholds and clustering hyper-parameters that require dataset-specific calibration.

Implementation Examples

Real-Time Streaming with Sortformer

import FluidAudio

let diarizer = SortformerDiarizer(config: .default)
let models = try await SortformerModels.loadFromHuggingFace(config: .default)
try await diarizer.initialize(models: models)

// Process audio chunks from microphone or network stream
for chunk in audioChunkStream {
    if let result = try? diarizer.processSamples(chunk) {
        for frame in 0..<result.frameCount {
            for speaker in 0..<4 {
                let prob = result.getSpeakerPrediction(speaker: speaker, frame: frame)
                // Drive real-time UI updates or downstream logic
            }
        }
        // Access timeline via SortformerTimeline for visualization
        updateSpeakerDisplay(diarizer.timeline)
    }
}

Batch Processing with Pyannote

import FluidAudio

let manager = DiarizerManager(config: .default)
let models = try await DiarizerModels.downloadIfNeeded()
manager.initialize(models: models)

let samples = try AudioConverter().resampleAudioFile(path: "meeting.wav")
let result = try manager.performCompleteDiarization(samples)

// Iterate over finalized segments
for segment in result.segments {
    print("\(segment.speakerId): \(segment.startTimeSeconds)s → \(segment.endTimeSeconds)s")
}

Comparing Both Approaches

let audio = try AudioConverter().resampleAudioFile(path: "meeting.wav")

// Sortformer streaming approach
let sortformer = SortformerDiarizer(config: .default)
let sortModels = try await SortformerModels.loadFromHuggingFace(config: .default)
try await sortformer.initialize(models: sortModels)
let sortTimeline = try sortformer.processComplete(audio)

// Pyannote offline approach
let pyannote = DiarizerManager(config: .default)
let pyModels = try await DiarizerModels.downloadIfNeeded()
pyannote.initialize(models: pyModels)
let pyResult = try pyannote.performCompleteDiarization(audio)

// Calculate DER against reference RTTM for quantitative comparison
// let derSort = computeDER(referenceRTTM, sortTimeline.rttm)
// let derPy = computeDER(referenceRTTM, pyResult.rttm)

Post-processing for Sortformer occurs in Sources/FluidAudio/Diarizer/Sortformer/SortformerTimeline.swift, which converts frame-level probabilities into finalized speaker segments suitable for RTTM export.

Summary

• Sortformer trades speaker count flexibility for ultra-low latency and superior noise handling in streaming scenarios.
• Pyannote (via DiarizerManager) offers unlimited speaker support and persistent identity tracking but requires higher latency and careful tuning.
• Sortformer configurations in SortformerConfig.swift must match static CoreML shapes and cannot change at runtime.
• Pyannote's modular pipeline in DiarizerManager.swift supports batch processing where speaker count varies or historical identity matching is required.
• Both solutions ship with FluidAudio, selecting Sortformer for real-time meeting transcription and Pyannote for archival analysis.

Frequently Asked Questions

Can Sortformer handle more than four speakers?

No. Sortformer is architecturally constrained to four fixed speaker slots defined in the CoreML model weights. Additional speakers are merged into existing slots or missed entirely. For scenarios requiring unlimited speaker detection, use Pyannote's DiarizerManager which clusters embeddings dynamically without predefined limits.

Why does Sortformer forget speakers between recordings?

Sortformer operates as a stateful streaming model without persistent embedding storage. The Speaker Cache and FIFO queue defined in SortformerTypes.swift maintain context only within a single session. Pyannote retains speaker embeddings across recordings, enabling consistent labeling for recurring participants.

Which approach yields better accuracy in noisy environments?

Sortformer generally outperforms Pyannote in noisy conditions and high-overlap scenarios because it was trained end-to-end on noisy, reverberant data. Pyannote's segmentation stage often degrades when background noise masks speech boundaries, leading to fragmentation errors that propagate through the clustering stage.

How do I reduce latency in Sortformer without recompiling the model?

Adjust the SortformerConfig parameters exposed in Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift. The default configuration targets approximately 1 second latency, while setting highQuality: false and reducing right-context parameters can lower this further. However, these parameters must match the compiled CoreML model shapes—significant architectural changes require model regeneration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}