How to Stream Generated Text Progressively in Transformers.js Text-Generation Pipelines

The transformers.js library enables real-time text streaming by passing a TextStreamer instance to the generation pipeline, which invokes callback functions after each token batch to incrementally decode and emit partial outputs.

The huggingface/transformers.js library supports streaming generated text progressively in text-generation pipelines through a specialized streaming architecture. By wiring a TextStreamer object into the generation loop, you can receive decoded words or raw tokens as they are produced rather than waiting for the full sequence to complete. This approach leverages specific callback hooks implemented in the model generation utilities to deliver low-latency, responsive output for CLI tools and web applications.

How Streaming Works Under the Hood

The TextStreamer Class Architecture

Located in packages/transformers/src/generation/streamers.js, the TextStreamer class (lines 38-68) serves as the primary interface for progressive output. It accumulates incoming token IDs in a cache, decodes them incrementally using the tokenizer, and forwards complete words to a configurable callback.

The constructor accepts several critical parameters:

• skip_prompt — Boolean to ignore the initial prompt tokens
• skip_special_tokens — Boolean to filter out special tokens like <pad> or <eos>
• callback_function — Handler for decoded text strings (defaults to process.stdout.write)
• token_callback_function — Handler for raw token ID arrays for advanced use cases

Pipeline Integration

The TextGenerationPipeline in packages/transformers/src/pipelines/text-generation.js (lines 94-99) exposes the streamer option directly in its generation API. When you pass a streamer instance to the pipeline, it forwards this object to the underlying model's generate() method as part of the keyword arguments.

Model Generation Loop Hooks

The actual streaming invocation occurs within packages/transformers/src/models/modeling_utils.js. After each forward pass during generation (lines 1095-1100), the model checks for a streamer and calls streamer.put() with the newly produced token IDs. When generation terminates (lines 1165-1170), the model invokes streamer.end() to flush any remaining cached tokens and finalize the output.

Implementing Progressive Text Streaming

Basic Console Output

The simplest implementation uses the default stdout callback to print words as they form:

import { pipeline, TextStreamer } from '@huggingface/transformers';

async function demoConsole() {
  // Load a small causal LM (ONNX or TF.js back‑end)
  const generator = await pipeline(
    'text-generation',
    'onnx-community/SmolLM2-135M-ONNX',
  );

  // Create a streamer that prints each completed word
  const streamer = new TextStreamer(generator.tokenizer, {
    skip_prompt: true,           // don't echo the prompt
    skip_special_tokens: true,   // hide <eos>, <pad>, …
  });

  // Generate 30 new tokens, streaming to stdout as they appear
  await generator('Once upon a time', {
    max_new_tokens: 30,
    streamer,                      // <‑‑ hook into the generation loop
  });
}

demoConsole();

Streaming to a Web UI

For React applications or browser environments, supply a custom callback_function to update state progressively:

import { pipeline, TextStreamer } from '@huggingface/transformers';
import { useState } from 'react';

export default function TextGeneration() {
  const [output, setOutput] = useState('');

  async function run() {
    const generator = await pipeline('text-generation', 'onnx-community/Qwen3-0.6B-ONNX');

    const streamer = new TextStreamer(generator.tokenizer, {
      skip_prompt: true,
      skip_special_tokens: true,
      // Append each chunk to React state
      callback_function: (chunk) => setOutput((prev) => prev + chunk),
    });

    await generator('Explain transformers.js in one sentence', {
      max_new_tokens: 80,
      streamer,
    });
  }

  return (
    <>
      <button onClick={run}>Generate</button>
      <pre>{output}</pre>
    </>
  );
}

Capturing Raw Token IDs

For analytics or custom decoding logic, use the token_callback_function to intercept token IDs before detokenization:

import { pipeline, TextStreamer } from '@huggingface/transformers';

async function logTokenIds() {
  const generator = await pipeline('text-generation', 'meta-llama/Llama-2-7b-chat');

  const streamer = new TextStreamer(generator.tokenizer, {
    token_callback_function: (ids) => console.log('New token IDs:', ids),
    skip_prompt: true,
    skip_special_tokens: true,
  });

  await generator('What is the capital of France?', {
    max_new_tokens: 10,
    streamer,
  });
}

Customizing Streamer Behavior

Word Boundary Detection and Caching

The put() method implements intelligent buffering to avoid splitting words. When processing new tokens, it decodes the entire cache and applies the following heuristics:

• Newline characters — Immediately flush the entire cache
• CJK characters (Chinese, Japanese, Korean) — Emit immediately since they are not space-delimited
• Standard text — Print only up to the last space character, keeping partial words in the cache until complete

Special Token Handling

The streamer maintains a set of special token IDs (this.special_ids) to handle control tokens. When a single special token arrives, the cache flushes first, then the token decodes separately. If skip_special_tokens is enabled, these are silently dropped rather than emitted to the callback.

Summary

• The TextStreamer class in src/generation/streamers.js provides the core infrastructure for progressive output
• Pass the streamer instance to TextGenerationPipeline via the streamer option in generation kwargs
• The model's generation loop in src/models/modeling_utils.js invokes put() after each token and end() upon completion
• Configure skip_prompt and skip_special_tokens to control which tokens appear in the output stream
• Use callback_function for text updates and token_callback_function for raw ID access
• Only batch size 1 is supported; the streamer receives a 2-D array but expects single-batch input

Frequently Asked Questions

What is the difference between TextStreamer and WhisperTextStreamer?

WhisperTextStreamer extends TextStreamer to handle timestamp tokens specific to the Whisper speech-to-text model. While TextStreamer focuses on general text generation, the Whisper variant processes audio timestamp markers in addition to standard vocabulary tokens, located in the same src/generation/streamers.js file (lines 71-92).

How does transformers.js handle incomplete words during streaming?

The streamer maintains an internal token cache that buffers incomplete words. It only emits text up to the last detected space character (or newlines/CJK characters) to ensure words are not split mid-generation. Remaining tokens stay cached until the next put() call provides enough context to complete the word.

Can I use streaming with batch generation?

No. The TextStreamer explicitly supports only batch size 1. The put() method receives a 2-D array with shape (batch × sequence) but expects the batch dimension to equal 1. Attempting to stream with larger batches will likely cause errors or undefined behavior in the current implementation.

Which model files support the streamer parameter?

All text-generation models in transformers.js that utilize the standard generate() method in src/models/modeling_utils.js support the streamer parameter. This includes ONNX and TensorFlow.js backends for models like Llama, Qwen, SmolLM, and other causal language models exposed through the TextGenerationPipeline in src/pipelines/text-generation.js.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}