How to Implement Speaker Identification with Known Speaker Embeddings in FluidAudio

FluidAudio provides a thread-safe, in-memory pipeline for speaker identification with known speaker embeddings through the SpeakerManager class, which matches 256-dimensional vectors using cosine distance and supports EMA updates for recognized speakers.

FluidAudio is an open-source Swift audio processing framework that delivers a full-stack speaker diarization pipeline. This guide explains how to implement speaker identification with known speaker embeddings using the library's Speaker and SpeakerManager types, based on the actual source code in the fluidinference/fluidaudio repository.

Architecture Overview

The speaker identification system centers on two core components that manage 256-dimensional embeddings in a thread-safe concurrent environment:

• Speaker (defined in [SpeakerTypes.swift](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Clustering/SpeakerTypes.swift)): An immutable identifier paired with a mutable embedding history. Each instance stores the current embedding, accumulated duration, and raw embedding history, implementing exponential moving average (EMA) updates and speaker merging capabilities.

• SpeakerManager (defined in [SpeakerManager.swift](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Clustering/SpeakerManager.swift)): A thread-safe in-memory database ([String:Speaker]) running on a dedicated concurrent queue (speaker.manager.queue) with barrier synchronization. It handles initialization of known speakers, cosine-distance matching, assignment logic, and persistence operations.

All embeddings undergo L2 normalization via VDSPOperations.l2Normalize before storage or comparison. The system uses cosine distance for similarity scoring, with a default speakerThreshold of 0.65 for assignment decisions.

Loading and Initializing Known Speakers

To bootstrap the system with pre-computed embeddings, deserialize your speaker data and populate the manager using initializeKnownSpeakers(_:mode:). This method accepts an array of Speaker objects and a mode flag (.skip avoids overwriting existing entries).

import FluidAudio

// JSON structure: [{ "id": "123", "name": "Alice", "embedding": [0.12, …, 0.98] }, …]
struct KnownSpeaker: Decodable {
    let id: String
    let name: String
    let embedding: [Float]
}

let data = try Data(contentsOf: URL(fileURLWithPath: "known_speakers.json"))
let known = try JSONDecoder().decode([KnownSpeaker].self, from: data)

// Map to FluidAudio Speaker types (256-D vectors expected)
let speakers = known.map { ks in
    Speaker(
        id: ks.id,
        name: ks.name,
        currentEmbedding: ks.embedding
    )
}

let speakerManager = SpeakerManager()
speakerManager.initializeKnownSpeakers(speakers, mode: .skip)

The Speaker initializer in SpeakerTypes.swift expects a 256-element Float array representing the L2-normalized speaker embedding vector.

Real-Time Speaker Identification

During streaming diarization, pass each new embedding to assignSpeaker(_:speechDuration:confidence:newName:). This method implements the core matching logic: it calculates cosine distance against all known speakers and either updates an existing record or creates a new identity.

func processEmbedding(_ embedding: [Float], segmentDuration: Float) {
    guard let speaker = speakerManager.assignSpeaker(
            embedding,
            speechDuration: segmentDuration,
            confidence: 1.0,
            newName: nil) else {
        print("Assignment failed: segment too short or error")
        return
    }
    
    print("Identified: \(speaker.name) (ID: \(speaker.id))")
}

If the embedding's cosine distance to the closest known speaker is below speakerThreshold (default 0.65), the method returns that speaker and triggers an EMA update if the distance is also below embeddingThreshold (default 0.45). If no match exists and the segment exceeds minSpeechDuration (default 1.0 seconds), it instantiates a new Speaker with a generated UUID.

Tuning Recognition Thresholds

Customize the matching behavior at initialization by adjusting three key parameters:

• speakerThreshold: Maximum cosine distance for assigning an embedding to an existing speaker (default 0.65).
• embeddingThreshold: Maximum distance to trigger an EMA update of the stored embedding (default 0.45).
• minSpeechDuration: Minimum segment duration in seconds required to create a new speaker entry (default 1.0).

let manager = SpeakerManager(
    speakerThreshold: 0.60,
    embeddingThreshold: 0.40,
    minSpeechDuration: 0.8,
    minEmbeddingUpdateDuration: 1.5
)

Lower speakerThreshold values enforce stricter matching, reducing false positives but potentially fragmenting single speakers into multiple entries.

Managing Speakers During Sessions

FluidAudio provides administrative methods for long-running diarization sessions:

Mark speakers as permanent to prevent automatic merging or cleanup:

speakerManager.makeSpeakerPermanent("42")

Merge duplicate identities after manual review or clustering refinement:

speakerManager.mergeSpeaker("17", into: "3", mergedName: "CombinedSpeaker")

Remove inactive speakers to reclaim memory:

let cutoff = Date().addingTimeInterval(-30)
speakerManager.removeSpeakersInactive(since: cutoff)

Call reset(keepIfPermanent: true) to clear transient speakers while preserving permanent entries.

Summary

• FluidAudio implements speaker identification with known speaker embeddings through the SpeakerManager class in SpeakerManager.swift, backed by the Speaker type in SpeakerTypes.swift.
• The system operates on 256-dimensional L2-normalized embeddings and uses cosine distance for similarity comparisons.
• Initialize known speakers via initializeKnownSpeakers(_:mode:) before processing audio streams.
• Use assignSpeaker(_:speechDuration:confidence:newName:) to match or create speakers during real-time diarization, with configurable thresholds for speakerThreshold (assignment) and embeddingThreshold (EMA updates).
• Thread safety is guaranteed through a dedicated concurrent queue with barrier synchronization, suitable for high-throughput streaming applications.

Frequently Asked Questions

What embedding dimension does FluidAudio require for speaker identification?

FluidAudio expects 256-dimensional Float vectors for all speaker embeddings. The Speaker type defined in SpeakerTypes.swift is optimized for this fixed dimensionality, and the cosine distance calculations in SpeakerUtilities assume normalized 256-element arrays.

How does FluidAudio handle thread safety during real-time diarization?

All SpeakerManager operations execute on a dedicated concurrent queue (speaker.manager.queue) using barrier synchronization. This ensures that read-modify-write cycles—such as EMA updates during assignSpeaker calls—remain atomic even when multiple audio streams or processing threads submit embeddings concurrently.

What is the difference between speakerThreshold and embeddingThreshold?

SpeakerManager uses speakerThreshold (default 0.65) as the maximum cosine distance to consider an embedding as belonging to an existing speaker. If the distance is below this, the speaker is assigned. The stricter embeddingThreshold (default 0.45) controls whether the stored embedding undergoes an EMA update; only embeddings within this tighter radius modify the speaker's centroid, preventing drift from outlier segments.

How do I prevent a known speaker from being removed during long sessions?

Call makeSpeakerPermanent(id) on your SpeakerManager instance. Permanent speakers survive reset(keepIfPermanent:) calls and are exempt from automatic removal heuristics, ensuring that pre-enrolled identities persist throughout the application lifecycle regardless of activity gaps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}