Multi-Speaker Diarization with Overlapping Speech: Best Practices in FluidAudio

FluidAudio handles multi-speaker diarization with overlapping speech by processing audio in overlapping chunks, filtering ambiguous frames where multiple speakers are active, and merging speaker embeddings across chunk boundaries using cosine-distance thresholds.

Multi-speaker diarization with overlapping speech remains one of the most challenging scenarios in audio processing, requiring sophisticated chunk management and speaker tracking to maintain consistent identities. The FluidAudio open-source framework implements a robust pipeline that addresses these challenges through configurable overlap parameters and intelligent masking strategies. This guide examines the source code architecture in fluidinference/fluidaudio and provides actionable best practices for configuring the diarization system when speakers talk simultaneously.

Chunk Overlap Strategy for Boundary Stability

The diarization pipeline processes long recordings in overlapping segments to ensure speech crossing chunk boundaries is captured consistently. In DiarizerManager.swift, the chunkOverlap value defined in DiarizerConfig is converted to samples and used to calculate the stride between consecutive chunks.

The step size calculation guarantees that each successive chunk contains a shared region with the previous chunk:

let overlapDuration = Int(config.chunkOverlap.rounded())
let stepSize = chunkSize - (sampleRate * overlapDuration)

As implemented in DiarizerManager.swift (lines 24–27), this approach ensures that any speech segment appearing at a chunk boundary is processed twice, providing redundant embedding extraction that stabilizes speaker identity assignment across transitions.

Mask Cleaning for Overlapping Frames

To prevent noisy embeddings from frames containing multiple simultaneous speakers, the pipeline implements a cleaning mask in processChunkWithSpeakerTracking. After segmentation, the system generates per-speaker binary masks and explicitly filters frames where the sum of speaker probabilities exceeds a single-speaker threshold.

In DiarizerManager.swift (lines 74–79), the logic discards ambiguous frames:

let isClean: Float = speakerSum < 2.0 ? 1.0 : 0.0
speakerMask.append(speakerProb * isClean)

This mask cleaning step ensures that only frames containing a single active speaker contribute to the embedding extraction, eliminating the false speaker switches typically caused by overlapping speech artifacts.

Speaker Assignment and Embedding Merging

The SpeakerManager class maintains an in-memory database that assigns stable speaker IDs and resolves conflicts when speakers overlap across chunks. According to SpeakerManager.swift (lines 21–33), the system stores speaker embeddings and updates them gradually to preserve identity consistency.

When processing each segment, the assignSpeaker method (lines 35–50) implements the following logic:

1. Attempts to match the new embedding to an existing speaker within speakerThreshold cosine distance
2. If matched, updates the existing speaker's embedding (subject to embeddingThreshold to prevent overwriting high-confidence vectors)
3. If no match is found and the segment exceeds minSpeechDuration, creates a new speaker ID

This threshold-based merging naturally handles overlapping speech scenarios because the clean mask ensures only one speaker's embedding is generated per frame, while the overlap strategy guarantees redundant processing of boundary-crossing speech.

Recommended Configuration for Overlap-Heavy Recordings

Based on the source code analysis, these parameters optimize performance for recordings with frequent speaker overlap:

• chunkDuration: 10.0 seconds (default) — Balances memory usage with segmentation model context requirements
• chunkOverlap: 2.0 seconds (minimum) — Guarantees boundary-crossing speech is captured twice; increase to 3.0 seconds for very dense overlapping speech
• speakerThreshold: 0.60–0.70 — Tighter distance threshold prevents the same speaker from being split into multiple IDs during overlap
• embeddingThreshold: 0.40–0.45 — Allows gradual embedding refinement without destabilizing high-confidence vectors
• minSpeechDuration: 1.0 second — Filters out short bursts often caused by overlap artifacts
• debugMode: true (development only) — Exposes the full speakerDatabase for inspecting speaker tracking decisions

As defined in DiarizerConfig (DiarizerTypes.swift, lines 29–34), these values control the chunking behavior and clustering sensitivity throughout the pipeline.

Implementation Examples

Basic Diarization with Overlap Handling

This example demonstrates the standard configuration for processing recordings with overlapping speech:

import FluidAudio

let config = DiarizerConfig(
    chunkDuration: 10.0,
    chunkOverlap: 2.0,
    speakerThreshold: 0.65,
    embeddingThreshold: 0.45,
    debugMode: true
)

let diarizer = DiarizerManager(config: config)
let samples: [Float] = try AudioLoader.loadWavFile(at: url)

if let models = try? await DiarizerModels.download() {
    diarizer.initialize(models: models)
}

let result = try diarizer.performCompleteDiarization(samples)

for segment in result.segments {
    print("Speaker \(segment.speakerId): \(segment.startTimeSeconds)s - \(segment.endTimeSeconds)s")
}

if let db = result.speakerDatabase {
    print("Total speakers discovered: \(db.count)")
}

Tuning Parameters for Dense Overlap

Increase overlap duration and tighten matching thresholds for recordings with heavy simultaneous speech:

let customConfig = DiarizerConfig(
    chunkOverlap: 3.0,                     // Larger overlap for redundancy
    speakerThreshold: 0.58,                // Tighter matching to prevent splits
    embeddingThreshold: 0.42,
    minSpeechDuration: 0.8,
    debugMode: false
)

let manager = DiarizerManager(config: customConfig)

The 3-second overlap provides additional redundancy when speech is dense, while the reduced speakerThreshold minimizes the risk of assigning multiple IDs to the same person during overlapping sections.

Post-Processing to Collapse Near-Duplicate Speakers

When automatic tracking produces separate IDs for the same speaker (rare but possible with heavy overlap), apply post-hoc merging based on embedding similarity:

let minDistanceForMerge: Float = 0.25

var mergedSegments = result.segments
for i in 0..<mergedSegments.count {
    for j in (i+1)..<mergedSegments.count {
        let idI = mergedSegments[i].speakerId
        let idJ = mergedSegments[j].speakerId
        guard idI != idJ,
              let embI = result.speakerDatabase?[idI],
              let embJ = result.speakerDatabase?[idJ] else { continue }
        
        let dist = VDSPOperations.cosineDistance(embI, embJ)
        if dist < minDistanceForMerge {
            let target = mergedSegments[i].speakerId
            mergedSegments[j].speakerId = target
        }
    }
}

This cleanup step iterates through the speakerDatabase available in debug mode and consolidates speakers whose embeddings fall within the cosine distance threshold.

Summary

• Implement 2-second minimum chunk overlap to ensure speech crossing chunk boundaries is processed redundantly, stabilizing speaker embeddings across transitions
• Enable mask cleaning via the speakerSum < 2.0 logic in DiarizerManager.swift to exclude frames containing multiple active speakers from embedding calculation
• Configure speakerThreshold between 0.60–0.70 to prevent a single speaker from being fragmented into multiple IDs during overlapping segments
• Activate debugMode during development to access the speakerDatabase and audit how the SpeakerManager assigns and merges embeddings
• Apply post-hoc embedding merging for final cleanup when the automatic clustering produces near-duplicate speakers in heavily overlapped recordings

Frequently Asked Questions

How does FluidAudio prevent the same speaker from being assigned multiple IDs during overlapping speech?

The pipeline uses a combination of mask cleaning and embedding thresholds. In DiarizerManager.swift, frames where the sum of speaker probabilities exceeds 2.0 are filtered out before embedding extraction, ensuring only single-speaker frames contribute to identity vectors. Additionally, SpeakerManager.assignSpeaker uses the speakerThreshold (recommended 0.60–0.70) to determine whether a new embedding matches an existing speaker or requires a new ID, preventing fragmentation during simultaneous speech.

What is the optimal chunk overlap duration for recordings with frequent speaker overlap?

The default 2.0 seconds provides sufficient redundancy for most recordings, as implemented in the step size calculation in DiarizerManager.swift (lines 24–27). For recordings with very dense overlapping speech or noisy environments, increasing chunkOverlap to 3.0 seconds in DiarizerConfig provides additional boundary coverage, ensuring that speech segments are captured in at least two consecutive chunks for stable embedding comparison.

Can I adjust the sensitivity for detecting overlapping speech frames?

Yes, though the threshold is hardcoded in the current implementation. The mask cleaning logic in DiarizerManager.swift (lines 74–79) uses speakerSum < 2.0 to identify clean frames. This value filters frames where more than one speaker probability exceeds 1.0 (indicating overlapping activity). To modify sensitivity, you would need to adjust the isClean calculation in processChunkWithSpeakerTracking within the source code.

How do I debug speaker tracking decisions in overlap-heavy audio?

Enable debugMode: true in your DiarizerConfig. This setting exposes the full speakerDatabase in the diarization result, allowing you to inspect how many speakers were created, how embeddings evolved across chunks, and whether overlapping speech caused spurious ID generation. The database contains the final embedding vectors stored in SpeakerManager.swift (lines 21–33), which you can analyze to determine if threshold adjustments are needed for your specific audio conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}