How to Handle the Last Chunk in Streaming ASR Processing in FluidAudio

To handle the last chunk in streaming ASR processing, set the isLastChunk flag to true when calling transcribeStreamingChunk, which triggers a final decoding loop in TdtDecoderV3 that emits all pending tokens and clears the timeJump offset.

Streaming automatic speech recognition (ASR) pipelines process audio incrementally, but the final segment demands special treatment to ensure no tokens are lost. In the fluidinference/fluidaudio repository, the last chunk handling is implemented through a dedicated finalization path that forces token emission and resets timestamp alignment state. This mechanism guarantees that transcripts are complete and temporally accurate when the input stream ends.

Why the Final Chunk Requires Special Handling

Standard streaming chunks use overlapping windows to maintain context, with the decoder holding back tokens until it sees sufficient future context to confirm predictions. When processing the last chunk, there is no subsequent audio to provide this context, so the system must:

• Emit any pending tokens that were deferred while waiting for additional encoder frames.
• Clear the timeJump offset used for timestamp alignment across chunks, since no further chunks exist to apply the offset.

Failure to trigger this finalization results in truncated transcripts and misaligned timestamps for the final utterance.

The Streaming ASR Pipeline Components

The fluidaudio repository implements last-chunk handling across three primary components that coordinate through the isLastChunk boolean flag.

AsrManager.transcribeStreamingChunk

The entry point for chunk-level processing resides in Sources/FluidAudio/ASR/AsrTranscription.swift within the transcribeStreamingChunk method (lines 188–221). This method prepares the audio buffer, runs the encoder-decoder inference loop, and accepts the isLastChunk parameter. When true, it forwards this flag to the decoder to enable finalization logic.

TdtDecoderV3.decodeWithTimings

The core finalization logic lives in Sources/FluidAudio/ASR/TDT/TdtDecoderV3.swift inside decodeWithTimings (lines 418–447, marked as "LAST CHUNK FINALIZATION"). When isLastChunk is true, this method executes a supplementary decoding loop that continues until either a configurable blank-run limit (config.tdtConfig.consecutiveBlankLimit) or a hard step limit (maxSymbolsPerStep) is reached. It then calls decoderState.finalizeLastChunk() to clear cached predictor outputs and nullifies decoderState.timeJump.

StreamingAsrManager.processWindow and flushRemaining

High-level stream management is handled in Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift. The processWindow method (lines 65–84) receives the isLastChunk flag and passes it to transcribeStreamingChunk. When the input stream ends, flushRemaining (lines 124–152) constructs the final partial window and automatically invokes processWindow(..., isLastChunk: true), ensuring clients do not need to manually detect stream termination.

Internal Logic of Last-Chunk Processing

When the final chunk flag is active, the pipeline executes the following sequence:

1. Chunk preparation – transcribeStreamingChunk pads the audio to frame-aligned lengths and invokes executeMLInferenceWithTimings.
2. Decoder invocation – The inference method forwards isLastChunk to tdtDecodeWithTimings.
3. Last-chunk loop – Inside TdtDecoderV3, a while loop runs after encoder frames are exhausted. It repeatedly:
   • Runs a decoder step using cached predictor output when available.
   • Calls the joint network for candidate frames.
   • Emits non-blank tokens, updates timestamps, and resets the consecutive-blank counter.
   • Terminates when blank-run or step limits are reached.
4. State finalization – The decoder state clears cached predictors to prevent duplicate punctuation and sets timeJump to nil, finalizing the timestamp alignment.

Implementation Example

When using the low-level AsrManager directly, you must explicitly set isLastChunk: true for the final audio segment:

import FluidAudio

// Assume `asrManager` has already been initialised and models loaded.
let audioSamples: [Float] = …          // Your final audio chunk (resampled to 16 kHz)

do {
    // `isLastChunk: true` triggers the final-chunk loop.
    let (tokens, timestamps, confidences, _) = try await asrManager.transcribeStreamingChunk(
        audioSamples,
        source: .microphone,
        previousTokens: [],
        isLastChunk: true
    )

    let result = asrManager.processTranscriptionResult(
        tokenIds: tokens,
        timestamps: timestamps,
        confidences: confidences,
        encoderSequenceLength: 0,
        audioSamples: audioSamples,
        processingTime: 0.0
    )

    print("Final transcript: \(result.text)")
} catch {
    print("ASR failed: \(error)")
}

When using StreamingAsrManager, this happens automatically. The flushRemaining() method detects end-of-stream and passes isLastChunk: true through processWindow, requiring no manual flag management from the caller. A similar flow exists in StreamingEouAsrManager.swift for end-of-utterance scenarios.

Summary

• Set isLastChunk: true when calling transcribeStreamingChunk to force emission of all pending tokens.
• The finalization logic resides in TdtDecoderV3.decodeWithTimings, which runs an extended decoding loop and clears the timeJump offset.
• StreamingAsrManager automates this via flushRemaining(), making it the preferred interface for most streaming applications.
• State clearing prevents duplicate punctuation and ensures timestamps align with the complete audio timeline.

Frequently Asked Questions

What happens if isLastChunk is not set to true?

If the flag is omitted or set to false, the decoder assumes more audio is coming and may hold back pending tokens while waiting for additional encoder frames. This results in truncated transcripts where the final words or punctuation are never emitted, and the timeJump offset remains active, causing timestamp misalignment relative to the full audio duration.

How does the decoder know when to stop the final loop?

The decoder stops when it encounters a configurable number of consecutive blank predictions (config.tdtConfig.consecutiveBlankLimit) or hits a hard step limit (maxSymbolsPerStep). These safeguards prevent infinite loops while ensuring all meaningful tokens are extracted from the final encoder state, as implemented in the "LAST CHUNK FINALIZATION" block of TdtDecoderV3.swift.

Can I use this with StreamingAsrManager or do I need to call it manually?

You do not need to set the flag manually when using StreamingAsrManager. The flushRemaining() method automatically detects when the input stream ends and invokes processWindow(..., isLastChunk: true). Only use the manual approach when working directly with the lower-level AsrManager interface outside the streaming manager context.

What is the timeJump offset and why must it be cleared?

timeJump is an internal offset used to align timestamps across overlapping chunks in the streaming pipeline. It compensates for the look-ahead context windows used during processing. When the last chunk is processed, this offset must be set to nil because there are no subsequent chunks to apply the remaining offset to, ensuring the final transcript timestamps reflect absolute positions in the complete audio file rather than relative offsets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}