# How to Implement Speaker Identification with Known Speaker Embeddings in FluidAudio

> Implement speaker identification with known speaker embeddings using FluidAudio. Learn how the SpeakerManager class efficiently matches vectors with cosine distance and EMA updates.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: how-to-guide
- Published: 2026-03-02

---

**FluidAudio provides a thread-safe, in-memory pipeline for speaker identification with known speaker embeddings through the `SpeakerManager` class, which matches 256-dimensional vectors using cosine distance and supports EMA updates for recognized speakers.**

FluidAudio is an open-source Swift audio processing framework that delivers a full-stack speaker diarization pipeline. This guide explains how to implement speaker identification with known speaker embeddings using the library's `Speaker` and `SpeakerManager` types, based on the actual source code in the [fluidinference/fluidaudio](https://github.com/fluidinference/fluidaudio) repository.

## Architecture Overview

The speaker identification system centers on two core components that manage **256-dimensional embeddings** in a thread-safe concurrent environment:

- **`Speaker`** (defined in [[`SpeakerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SpeakerTypes.swift)](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Clustering/SpeakerTypes.swift)): An immutable identifier paired with a mutable embedding history. Each instance stores the current embedding, accumulated duration, and raw embedding history, implementing exponential moving average (EMA) updates and speaker merging capabilities.

- **`SpeakerManager`** (defined in [[`SpeakerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SpeakerManager.swift)](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/Diarizer/Clustering/SpeakerManager.swift)): A thread-safe in-memory database (`[String:Speaker]`) running on a dedicated concurrent queue (`speaker.manager.queue`) with barrier synchronization. It handles initialization of known speakers, cosine-distance matching, assignment logic, and persistence operations.

All embeddings undergo L2 normalization via `VDSPOperations.l2Normalize` before storage or comparison. The system uses **cosine distance** for similarity scoring, with a default `speakerThreshold` of `0.65` for assignment decisions.

## Loading and Initializing Known Speakers

To bootstrap the system with pre-computed embeddings, deserialize your speaker data and populate the manager using `initializeKnownSpeakers(_:mode:)`. This method accepts an array of `Speaker` objects and a mode flag (`.skip` avoids overwriting existing entries).

```swift
import FluidAudio

// JSON structure: [{ "id": "123", "name": "Alice", "embedding": [0.12, …, 0.98] }, …]
struct KnownSpeaker: Decodable {
    let id: String
    let name: String
    let embedding: [Float]
}

let data = try Data(contentsOf: URL(fileURLWithPath: "known_speakers.json"))
let known = try JSONDecoder().decode([KnownSpeaker].self, from: data)

// Map to FluidAudio Speaker types (256-D vectors expected)
let speakers = known.map { ks in
    Speaker(
        id: ks.id,
        name: ks.name,
        currentEmbedding: ks.embedding
    )
}

let speakerManager = SpeakerManager()
speakerManager.initializeKnownSpeakers(speakers, mode: .skip)

```

The `Speaker` initializer in [`SpeakerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SpeakerTypes.swift) expects a 256-element `Float` array representing the L2-normalized speaker embedding vector.

## Real-Time Speaker Identification

During streaming diarization, pass each new embedding to `assignSpeaker(_:speechDuration:confidence:newName:)`. This method implements the core matching logic: it calculates cosine distance against all known speakers and either updates an existing record or creates a new identity.

```swift
func processEmbedding(_ embedding: [Float], segmentDuration: Float) {
    guard let speaker = speakerManager.assignSpeaker(
            embedding,
            speechDuration: segmentDuration,
            confidence: 1.0,
            newName: nil) else {
        print("Assignment failed: segment too short or error")
        return
    }
    
    print("Identified: \(speaker.name) (ID: \(speaker.id))")
}

```

If the embedding's cosine distance to the closest known speaker is below `speakerThreshold` (default `0.65`), the method returns that speaker and triggers an EMA update if the distance is also below `embeddingThreshold` (default `0.45`). If no match exists and the segment exceeds `minSpeechDuration` (default `1.0` seconds), it instantiates a new `Speaker` with a generated UUID.

## Tuning Recognition Thresholds

Customize the matching behavior at initialization by adjusting three key parameters:

- **`speakerThreshold`**: Maximum cosine distance for assigning an embedding to an existing speaker (default `0.65`).
- **`embeddingThreshold`**: Maximum distance to trigger an EMA update of the stored embedding (default `0.45`).
- **`minSpeechDuration`**: Minimum segment duration in seconds required to create a new speaker entry (default `1.0`).

```swift
let manager = SpeakerManager(
    speakerThreshold: 0.60,
    embeddingThreshold: 0.40,
    minSpeechDuration: 0.8,
    minEmbeddingUpdateDuration: 1.5
)

```

Lower `speakerThreshold` values enforce stricter matching, reducing false positives but potentially fragmenting single speakers into multiple entries.

## Managing Speakers During Sessions

FluidAudio provides administrative methods for long-running diarization sessions:

**Mark speakers as permanent** to prevent automatic merging or cleanup:

```swift
speakerManager.makeSpeakerPermanent("42")

```

**Merge duplicate identities** after manual review or clustering refinement:

```swift
speakerManager.mergeSpeaker("17", into: "3", mergedName: "CombinedSpeaker")

```

**Remove inactive speakers** to reclaim memory:

```swift
let cutoff = Date().addingTimeInterval(-30)
speakerManager.removeSpeakersInactive(since: cutoff)

```

Call `reset(keepIfPermanent: true)` to clear transient speakers while preserving permanent entries.

## Summary

- **FluidAudio** implements speaker identification with known speaker embeddings through the `SpeakerManager` class in [`SpeakerManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SpeakerManager.swift), backed by the `Speaker` type in [`SpeakerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SpeakerTypes.swift).
- The system operates on **256-dimensional L2-normalized embeddings** and uses **cosine distance** for similarity comparisons.
- Initialize known speakers via `initializeKnownSpeakers(_:mode:)` before processing audio streams.
- Use `assignSpeaker(_:speechDuration:confidence:newName:)` to match or create speakers during real-time diarization, with configurable thresholds for `speakerThreshold` (assignment) and `embeddingThreshold` (EMA updates).
- Thread safety is guaranteed through a dedicated concurrent queue with barrier synchronization, suitable for high-throughput streaming applications.

## Frequently Asked Questions

### What embedding dimension does FluidAudio require for speaker identification?

FluidAudio expects **256-dimensional Float vectors** for all speaker embeddings. The `Speaker` type defined in [`SpeakerTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/SpeakerTypes.swift) is optimized for this fixed dimensionality, and the cosine distance calculations in `SpeakerUtilities` assume normalized 256-element arrays.

### How does FluidAudio handle thread safety during real-time diarization?

All `SpeakerManager` operations execute on a dedicated concurrent queue (`speaker.manager.queue`) using barrier synchronization. This ensures that read-modify-write cycles—such as EMA updates during `assignSpeaker` calls—remain atomic even when multiple audio streams or processing threads submit embeddings concurrently.

### What is the difference between `speakerThreshold` and `embeddingThreshold`?

`SpeakerManager` uses `speakerThreshold` (default `0.65`) as the maximum cosine distance to consider an embedding as belonging to an existing speaker. If the distance is below this, the speaker is assigned. The stricter `embeddingThreshold` (default `0.45`) controls whether the stored embedding undergoes an EMA update; only embeddings within this tighter radius modify the speaker's centroid, preventing drift from outlier segments.

### How do I prevent a known speaker from being removed during long sessions?

Call `makeSpeakerPermanent(id)` on your `SpeakerManager` instance. Permanent speakers survive `reset(keepIfPermanent:)` calls and are exempt from automatic removal heuristics, ensuring that pre-enrolled identities persist throughout the application lifecycle regardless of activity gaps.