# Limitations of Sortformer for Speaker Diarization: Constraints and Workarounds

> Explore the limitations of Sortformer for speaker diarization including fixed speaker slots and cross-session identity issues. Learn about workarounds for large conferences.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: deep-dive
- Published: 2026-03-02

---

**Sortformer is architecturally limited to four fixed speaker slots, cannot persist speaker identities across separate audio sessions, and relies on static CoreML configurations that prevent runtime parameter adjustments, making it unsuitable for large conferences or cross-session speaker tracking.**

Sortformer is a streaming, end-to-end neural diarization model implemented in the `fluidinference/fluidaudio` repository. While it excels at real-time, on-device speaker diarization with low latency, understanding the **limitations of Sortformer for speaker diarization** is critical for architects deciding whether it fits their use case.

## The Four-Speaker Hard Limit

The most significant constraint is the **fixed output layer architecture** that supports exactly four speaker probability channels. Unlike clustering-based diarizers that can dynamically create new speaker identities, Sortformer’s neural head is hard-wired to four slots.

According to the source documentation in [`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md) (lines 14-18), any fifth or later speaker is either merged with an existing slot or completely missed. This makes the model unsuitable for webinars, large conference calls, or any scenario where more than four people might speak.

## No Persistent Speaker Identity Across Sessions

Sortformer maintains a **speaker cache** that stores short-term embeddings for the current stream only. When the stream ends, the cache is discarded, and there is no mechanism to persist or compare embeddings across sessions.

As noted in [`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md) (lines 73-76), this architectural choice prevents the system from answering questions like "Is this the same person who spoke in yesterday’s meeting?" Each audio file or stream is processed in complete isolation.

## Static CoreML Configuration Constraints

Configuration parameters are **baked into the CoreML model** at conversion time, making them immutable at runtime. Parameters such as `fifoLen`, `spkcacheLen`, and context lengths are fixed to satisfy CoreML’s requirement for static input tensor shapes.

According to [`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md) (lines 66-78), you cannot adjust these parameters on the fly to trade latency for quality. If you need different behavior, you must swap the entire model file (e.g., switching from `SortformerDefault` to `SortformerNvidiaHigh`).

## Coarse Latency vs. Quality Trade-offs

The model provides only **three preset latency/quality points** rather than fine-grained control:

- **Default**: ~1 second latency
- **NVIDIA Low**: ~1 second latency  
- **NVIDIA High**: ~30 seconds latency (best quality)

As documented in [`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md) (lines 91-100 and 124-131), there is no API to adjust chunk size or right-context duration without generating a new CoreML conversion. This limits flexibility for applications that need mid-range latency settings.

## Performance Boundaries in Real-World Audio

### Quiet or Distant Speech

The model was trained to **ignore background conversations** and favor robust detection of dominant speech. Consequently, low-energy speech receives low probabilities and is often filtered out during post-processing ([`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md), lines 17-18).

### Heavy Crosstalk and Overlap

With more than four people talking simultaneously, the fixed-slot output cannot represent overlapping speech patterns, leading to missed or merged segments ([`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md), lines 40-42).

## Implementation Examples

### Real-Time Streaming (Default 4-Speaker Config)

```swift
import FluidAudio

let diarizer = SortformerDiarizer(config: .default)          // 4-speaker, ~1 s latency
let models = try await SortformerModels.loadFromHuggingFace(
    config: .default,
    computeUnits: .all
)

try await diarizer.initialize(models: models)

// Assume `audioEngine` provides microphone buffers
audioEngine.installTap { buffer in
    let samples = buffer.floatChannelData![0]
    if let result = try? diarizer.processSamples(Array(samples)) {
        // `result` contains per-frame probabilities for 4 speakers
        // Handle UI update or further analysis here
    }
}

```

This implementation uses **`SortformerDiarizer`**, **`SortformerModels`**, and the **default configuration** defined in [`Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift).

### Batch Processing with High-Quality (NVIDIA High-Latency) Model

```swift
import FluidAudio

let diarizer = SortformerDiarizer(config: .nvidiaHighLatency)   // 30 s latency, best quality
let models = try await SortformerModels.loadFromHuggingFace(
    config: .default,
    computeUnits: .all
)

try await diarizer.initialize(models: models)

let timeline = try diarizer.processComplete(audioSamples)

// Iterate over finalized segments
for (speakerIdx, segments) in timeline.segments.enumerated() {
    for seg in segments {
        print("Speaker \(speakerIdx): \(seg.startTime)s → \(seg.endTime)s")
    }
}

```

This relies on the **high-latency configuration** defined in [`Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift) and the model file `SortformerNvidiaHigh.mlmodelc`.

## Key Source Files

| File | Role | Link |
|------|------|------|
| [`SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerConfig.swift) | Stores static streaming parameters (`chunkLen`, `fifoLen`, `rightContext`, etc.) that must match the CoreML model shape. | [Link](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift) |
| [`SortformerDiarizerPipeline.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerDiarizerPipeline.swift) (named `SortformerDiarizer` in code) | Orchestrates audio buffering, feature extraction, model inference, state updates, and timeline construction. | [Link](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerDiarizerPipeline.swift) |
| [`SortformerModelInference.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerModelInference.swift) | Wraps the CoreML model, loads it from HuggingFace or local path, and defines input/output tensor formats. | [Link](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerModelInference.swift) |
| [`SortformerStateUpdater.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerStateUpdater.swift) | Implements the streaming state logic (FIFO queue, speaker-cache compression, silence profile) derived from NVIDIA’s NeMo implementation. | [Link](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerStateUpdater.swift) |
| [`SortformerTimeline.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerTimeline.swift) | Accumulates per-frame predictions, applies median filtering, and creates final `SortformerSegment` objects. | [Link](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Sortformer/SortformerTimeline.swift) |
| [`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md) | Human-readable description of architecture, parameters, and listed limitations. | [Link](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md) |

## Summary

- **Fixed four-speaker ceiling**: The output layer in `SortformerDiarizer` is hard-coded to four probability channels, making the model unsuitable for meetings with five or more participants.

- **No cross-session speaker persistence**: The speaker cache is ephemeral and discarded when streams end, preventing identity tracking across multiple recordings.

- **Immutable runtime configuration**: CoreML static tensor requirements bake parameters like `fifoLen` and `spkcacheLen` into the model file, requiring model swaps to change behavior.

- **Coarse latency controls**: Only three preset configurations exist (default, NVIDIA low, NVIDIA high), offering limited flexibility for latency tuning.

- **Audio sensitivity boundaries**: The model filters low-energy speech and struggles with heavy crosstalk or overlapping conversations involving more than four speakers.

## Frequently Asked Questions

### Can Sortformer handle meetings with more than four speakers?

No. The model’s neural output layer is architecturally constrained to exactly four speaker probability channels. According to the source documentation in [`Documentation/Diarization/Sortformer.md`](https://github.com/fluidinference/fluidaudio/blob/main/Documentation/Diarization/Sortformer.md), any fifth or subsequent speaker is either merged with an existing slot or completely omitted from the diarization output. For larger meetings, you must use a clustering-based diarization approach instead.

### Why doesn't Sortformer remember speakers across different audio files?

Sortformer maintains a transient **speaker cache** that stores short-term embeddings only for the current audio stream. When processing ends, this cache is discarded, and the architecture lacks any mechanism to persist embeddings or compare them across sessions. As documented in [`SortformerStateUpdater.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerStateUpdater.swift), this design prioritizes low-latency streaming over cross-session speaker identity management.

### Can I adjust the latency settings at runtime?

No. Sortformer’s configuration parameters—including `fifoLen`, `spkcacheLen`, and context lengths—are baked into the CoreML model at conversion time to satisfy static tensor shape requirements. To change latency characteristics, you must swap the entire model file (e.g., switching from `SortformerDefault` to `SortformerNvidiaHigh`) rather than adjusting parameters programmatically. This limitation is defined in [`SortformerConfig.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerConfig.swift) and [`SortformerModelInference.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SortformerModelInference.swift).