Limitations of Sortformer for Speaker Diarization: Constraints and Workarounds

Sortformer is architecturally limited to four fixed speaker slots, cannot persist speaker identities across separate audio sessions, and relies on static CoreML configurations that prevent runtime parameter adjustments, making it unsuitable for large conferences or cross-session speaker tracking.

Sortformer is a streaming, end-to-end neural diarization model implemented in the fluidinference/fluidaudio repository. While it excels at real-time, on-device speaker diarization with low latency, understanding the limitations of Sortformer for speaker diarization is critical for architects deciding whether it fits their use case.

The Four-Speaker Hard Limit

The most significant constraint is the fixed output layer architecture that supports exactly four speaker probability channels. Unlike clustering-based diarizers that can dynamically create new speaker identities, Sortformer’s neural head is hard-wired to four slots.

According to the source documentation in Documentation/Diarization/Sortformer.md (lines 14-18), any fifth or later speaker is either merged with an existing slot or completely missed. This makes the model unsuitable for webinars, large conference calls, or any scenario where more than four people might speak.

No Persistent Speaker Identity Across Sessions

Sortformer maintains a speaker cache that stores short-term embeddings for the current stream only. When the stream ends, the cache is discarded, and there is no mechanism to persist or compare embeddings across sessions.

As noted in Documentation/Diarization/Sortformer.md (lines 73-76), this architectural choice prevents the system from answering questions like "Is this the same person who spoke in yesterday’s meeting?" Each audio file or stream is processed in complete isolation.

Static CoreML Configuration Constraints

Configuration parameters are baked into the CoreML model at conversion time, making them immutable at runtime. Parameters such as fifoLen, spkcacheLen, and context lengths are fixed to satisfy CoreML’s requirement for static input tensor shapes.

According to Documentation/Diarization/Sortformer.md (lines 66-78), you cannot adjust these parameters on the fly to trade latency for quality. If you need different behavior, you must swap the entire model file (e.g., switching from SortformerDefault to SortformerNvidiaHigh).

Coarse Latency vs. Quality Trade-offs

The model provides only three preset latency/quality points rather than fine-grained control:

• Default: ~1 second latency
• NVIDIA Low: ~1 second latency
• NVIDIA High: ~30 seconds latency (best quality)

As documented in Documentation/Diarization/Sortformer.md (lines 91-100 and 124-131), there is no API to adjust chunk size or right-context duration without generating a new CoreML conversion. This limits flexibility for applications that need mid-range latency settings.

Performance Boundaries in Real-World Audio

Quiet or Distant Speech

The model was trained to ignore background conversations and favor robust detection of dominant speech. Consequently, low-energy speech receives low probabilities and is often filtered out during post-processing (Documentation/Diarization/Sortformer.md, lines 17-18).

Heavy Crosstalk and Overlap

With more than four people talking simultaneously, the fixed-slot output cannot represent overlapping speech patterns, leading to missed or merged segments (Documentation/Diarization/Sortformer.md, lines 40-42).

Implementation Examples

Real-Time Streaming (Default 4-Speaker Config)

import FluidAudio

let diarizer = SortformerDiarizer(config: .default)          // 4-speaker, ~1 s latency
let models = try await SortformerModels.loadFromHuggingFace(
    config: .default,
    computeUnits: .all
)

try await diarizer.initialize(models: models)

// Assume `audioEngine` provides microphone buffers
audioEngine.installTap { buffer in
    let samples = buffer.floatChannelData![0]
    if let result = try? diarizer.processSamples(Array(samples)) {
        // `result` contains per-frame probabilities for 4 speakers
        // Handle UI update or further analysis here
    }
}

This implementation uses SortformerDiarizer, SortformerModels, and the default configuration defined in Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift.

Batch Processing with High-Quality (NVIDIA High-Latency) Model

import FluidAudio

let diarizer = SortformerDiarizer(config: .nvidiaHighLatency)   // 30 s latency, best quality
let models = try await SortformerModels.loadFromHuggingFace(
    config: .default,
    computeUnits: .all
)

try await diarizer.initialize(models: models)

let timeline = try diarizer.processComplete(audioSamples)

// Iterate over finalized segments
for (speakerIdx, segments) in timeline.segments.enumerated() {
    for seg in segments {
        print("Speaker \(speakerIdx): \(seg.startTime)s → \(seg.endTime)s")
    }
}

This relies on the high-latency configuration defined in Sources/FluidAudio/Diarizer/Sortformer/SortformerConfig.swift and the model file SortformerNvidiaHigh.mlmodelc.

Key Source Files

File	Role	Link
SortformerConfig.swift	Stores static streaming parameters (chunkLen, fifoLen, rightContext, etc.) that must match the CoreML model shape.	Link
SortformerDiarizerPipeline.swift (named SortformerDiarizer in code)	Orchestrates audio buffering, feature extraction, model inference, state updates, and timeline construction.	Link
SortformerModelInference.swift	Wraps the CoreML model, loads it from HuggingFace or local path, and defines input/output tensor formats.	Link
SortformerStateUpdater.swift	Implements the streaming state logic (FIFO queue, speaker-cache compression, silence profile) derived from NVIDIA’s NeMo implementation.	Link
SortformerTimeline.swift	Accumulates per-frame predictions, applies median filtering, and creates final SortformerSegment objects.	Link
Documentation/Diarization/Sortformer.md	Human-readable description of architecture, parameters, and listed limitations.	Link

Summary

• Fixed four-speaker ceiling: The output layer in SortformerDiarizer is hard-coded to four probability channels, making the model unsuitable for meetings with five or more participants.

• No cross-session speaker persistence: The speaker cache is ephemeral and discarded when streams end, preventing identity tracking across multiple recordings.

• Immutable runtime configuration: CoreML static tensor requirements bake parameters like fifoLen and spkcacheLen into the model file, requiring model swaps to change behavior.

• Coarse latency controls: Only three preset configurations exist (default, NVIDIA low, NVIDIA high), offering limited flexibility for latency tuning.

• Audio sensitivity boundaries: The model filters low-energy speech and struggles with heavy crosstalk or overlapping conversations involving more than four speakers.

Frequently Asked Questions

Can Sortformer handle meetings with more than four speakers?

No. The model’s neural output layer is architecturally constrained to exactly four speaker probability channels. According to the source documentation in Documentation/Diarization/Sortformer.md, any fifth or subsequent speaker is either merged with an existing slot or completely omitted from the diarization output. For larger meetings, you must use a clustering-based diarization approach instead.

Why doesn't Sortformer remember speakers across different audio files?

Sortformer maintains a transient speaker cache that stores short-term embeddings only for the current audio stream. When processing ends, this cache is discarded, and the architecture lacks any mechanism to persist embeddings or compare them across sessions. As documented in SortformerStateUpdater.swift, this design prioritizes low-latency streaming over cross-session speaker identity management.

Can I adjust the latency settings at runtime?

No. Sortformer’s configuration parameters—including fifoLen, spkcacheLen, and context lengths—are baked into the CoreML model at conversion time to satisfy static tensor shape requirements. To change latency characteristics, you must swap the entire model file (e.g., switching from SortformerDefault to SortformerNvidiaHigh) rather than adjusting parameters programmatically. This limitation is defined in SortformerConfig.swift and SortformerModelInference.swift.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}