FluidAudio Performance Metrics: ASR Timing and Diarization Quality Explained

FluidAudio exposes two families of performance metrics—ASR timing and resource utilization via PerformanceMonitor, and diarization quality via DiarizationMetricsCalculator—enabling precise measurement of real-time factors, memory usage, and error rates.

FluidAudio provides comprehensive instrumentation for evaluating both automatic speech recognition (ASR) pipeline efficiency and speaker diarization accuracy. These FluidAudio performance metrics are designed to be human-readable via summary properties and machine-consumable through Codable conformance, making them suitable for production monitoring and benchmarking workflows.

ASR Performance Metrics: Real-Time Factor and Resource Usage

The ASR instrumentation focuses on latency, throughput, and resource consumption during transcription sessions.

Core Metrics and Data Structures

The primary data structures reside in Sources/FluidAudio/ASR/PerformanceMetrics.swift. The ASRPerformanceMetrics struct captures:

• Component-level timing – Pre-processor, encoder, and decoder durations (currently implemented as placeholders via trackComponent calls)
• Total processing time – Wall-clock duration for the complete session
• Real-time factor (RTFx) – Ratio of audio duration to processing time
• Peak memory usage – Resident memory in megabytes captured via Mach APIs
• Optional GPU utilization – Hardware acceleration metrics when available

The AggregatedMetrics struct provides statistical rollups across multiple sessions, including average RTFx, average total time, maximum memory observed, and sample counts.

How ASR Metrics Are Calculated

The PerformanceMonitor actor collects metrics through the trackSession(_:audioLengthSeconds:block:) method. The calculation workflow proceeds as follows:

1. Timing capture – The monitor records wall-clock timestamps immediately before and after executing the ASR block
2. Memory sampling – Resident memory is queried using Mach kernel APIs at the end of the session
3. RTFx computation – The real-time factor derives from audioLengthSeconds / totalProcessingTime
4. Component tracking – Individual stage timings (preprocessor, encoder, decoder) are captured via trackComponent calls, though these require manual instrumentation in the calling code

Tracking ASR Sessions in Code

To instrument an ASR pipeline, wrap the transcription call with the monitor:

let monitor = PerformanceMonitor()
let audioLength: Float = 12.3   // seconds of input audio

// Wrap the ASR pipeline in a tracked session
let (transcript, metrics) = try await monitor.trackSession(
    operation: "TranscribeAudio",
    audioLengthSeconds: audioLength
) {
    // … put the actual ASR call here, e.g.:
    return try await asrEngine.transcribe(audioBuffer)
}

// Inspect the metrics
print(metrics.summary)               // human‑readable text
print("RTFx =", metrics.rtfx)       // numeric real‑time factor

Diarization Quality Metrics: DER and JER Calculation

For speaker diarization evaluation, FluidAudio implements standard academic metrics including Diarization Error Rate (DER) and Jaccard Error Rate (JER).

Understanding Diarization Error Rate (DER)

The DiarizationMetricsCalculator class in Sources/FluidAudioCLI/Utils/DiarizationMetrics.swift computes DER through segment-level overlap analysis. The static method offlineMetrics(predicted:groundTruth:frameSize:audioDurationSeconds:logger:) executes the following pipeline:

1. Scoring collar application – A default 0.25-second collar is applied to segment boundaries, with optional removal of overlapping regions
2. Interval clipping – Both reference and hypothesis segments are clipped to the evaluation intervals
3. Overlap computation – The algorithm calculates total speech duration for reference and hypothesis, plus their intersection
4. Error derivation:
   • Miss = reference duration – overlap
   • False alarm = hypothesis duration – overlap
   • Confusion = overlap – correctly-assigned speaker overlap
5. Rate calculation – Errors are normalized as percentages of reference speech:
   • missRate = (miss / reference) * 100
   • falseAlarmRate = (falseAlarm / reference) * 100
   • speakerErrorRate = (confusion / reference) * 100
6. DER aggregation – DER = missRate + falseAlarmRate + speakerErrorRate

Jaccard Error Rate (JER) and Speaker Mapping

Beyond DER, the calculator implements Jaccard Error Rate (JER) through per-speaker Jaccard similarity:

1. For each ground-truth speaker, the algorithm identifies the best-matching hypothesis speaker
2. It calculates intersection over union for their respective speech segments
3. These Jaccard scores are averaged across all reference speakers
4. JER derives as (1 – averageJaccard) * 100

The system also exposes the speaker mapping between predicted and ground-truth IDs, enabling detailed error analysis.

Computing Offline Diarization Metrics

To evaluate diarization quality programmatically:

// Assume `predicted` and `groundTruth` are arrays of TimedSpeakerSegment
let metrics = DiarizationMetricsCalculator.offlineMetrics(
    predicted: predicted,
    groundTruth: groundTruth,
    audioDurationSeconds: 180.0,
    logger: AppLogger(category: "Demo")
)

// Human‑readable overview
print(metrics.der)          // DER in percent
print(metrics.jer)          // JER in percent
print(metrics.speakerMapping) // best speaker ID correspondence

Summary

• ASR metrics in Sources/FluidAudio/ASR/PerformanceMetrics.swift track real-time factors, memory usage, and component timing via the PerformanceMonitor actor
• RTFx calculates as audioLengthSeconds / totalProcessingTime, indicating processing speed relative to audio duration
• Diarization metrics in Sources/FluidAudioCLI/Utils/DiarizationMetrics.swift implement standard academic measures including DER and JER
• DER combines miss rate, false-alarm rate, and speaker-error rate as percentages of reference speech duration
• Both metric families support programmatic access through Codable conformance and human-readable summary properties

Frequently Asked Questions

What is RTFx in FluidAudio?

RTFx (Real-Time Factor) measures ASR processing efficiency by dividing the audio duration by the actual processing time. A value of 1.0 indicates real-time processing, while values less than 1.0 indicate faster-than-real-time performance. FluidAudio calculates this automatically when using PerformanceMonitor.trackSession(_:audioLengthSeconds:block:).

How is Diarization Error Rate calculated?

FluidAudio calculates DER by first applying a 0.25-second scoring collar to segment boundaries, then computing three error components: miss (reference speech missed by the hypothesis), false alarm (hypothesis speech without reference), and speaker error (correct time but wrong speaker label). These are summed and normalized as percentages of total reference speech duration.

Where are the performance metrics defined in the codebase?

ASR performance metrics reside in Sources/FluidAudio/ASR/PerformanceMetrics.swift, containing ASRPerformanceMetrics, AggregatedMetrics, and the PerformanceMonitor actor. Diarization metrics are defined in Sources/FluidAudioCLI/Utils/DiarizationMetrics.swift, housing DiarizationMetrics and DiarizationMetricsCalculator.

Can I monitor GPU utilization with FluidAudio?

Yes, the ASRPerformanceMetrics struct includes an optional field for GPU utilization, allowing hardware acceleration metrics to be captured when available. However, this requires the underlying ASR engine to report GPU statistics, as the PerformanceMonitor provides the container but relies on the engine implementation to supply the actual utilization values.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}