Does FreeLLMAPI Support Streaming Responses? A Complete Technical Guide

Yes, FreeLLMAPI natively supports streaming responses through asynchronous generators implemented across all provider classes, delivering Server-Sent Events (SSE) to clients when the stream: true parameter is included in requests.

FreeLLMAPI is an open-source unified API gateway that aggregates multiple language model providers into a single interface. If you are building real-time applications that require low-latency token generation, understanding how FreeLLMAPI streaming works is essential for optimizing user experience. The codebase implements streaming at the provider abstraction layer, ensuring consistent SSE delivery regardless of whether you are using OpenAI-compatible endpoints, Google Gemini, or other supported services.

Architecture of Streaming in FreeLLMAPI

The streaming implementation follows a provider-pattern architecture that abstracts provider-specific differences behind a unified asynchronous generator interface.

The Base Provider Contract

In [server/src/providers/base.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/base.ts), the codebase defines an abstract method streamChatCompletion that every provider must implement. This method returns an asynchronous generator yielding partial response chunks, establishing a contract for streaming behavior across the entire system.

// Conceptual signature from base.ts (around line 234)
abstract streamChatCompletion(
  apiKey: string,
  messages: Message[],
  model: string,
  options: RequestOptions
): AsyncGenerator<StreamChunk, void, unknown>;

Provider-Specific Implementations

Concrete implementations reside in provider-specific files that handle upstream API formatting differences while maintaining the same yield contract:

• OpenAI-compatible providers: [server/src/providers/openai-compat.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts) parses SSE lines from OpenAI-format responses and yields standardized chunks.
• Google Gemini: [server/src/providers/google.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/google.ts) implements streaming for Google's generation methods.
• Additional providers: Cohere, Cloudflare, AI-Horde, and other integrations follow the same pattern, ensuring FreeLLMAPI streaming works universally across all supported backends.

Route Handling and SSE Forwarding

The routing layer in [server/src/routes/proxy.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (approximately line 1169) invokes the selected provider's streamChatCompletion method and immediately pipes the resulting asynchronous iterator to the client as Server-Sent Events. This architecture avoids buffering entire responses in memory, minimizing time-to-first-token latency.

How to Enable Streaming in API Requests

To activate FreeLLMAPI streaming, include "stream": true in your JSON request body. Streaming is the default behavior for chat completions when this parameter is set or omitted.

Direct Provider Usage

When working with the provider layer directly, iterate over the async generator:

import { getProvider } from './server/src/providers/index.js';

const provider = getProvider('openai');
const messages = [{ role: 'user', content: 'Explain quantum tunnelling' }];

for await (const chunk of provider.streamChatCompletion(
  'my-openai-key',
  messages,
  'gpt-4o',
  { stream: true }
)) {
  // chunk.delta.content contains the partial token
  process.stdout.write(chunk.delta?.content ?? '');
}

HTTP API Endpoint

Consume the streaming endpoint from any HTTP client using the SSE protocol:

fetch('https://api.freellmapi.com/v1/chat/completions', {
  method: 'POST',
  headers: { 
    'Content-Type': 'application/json', 
    'Authorization': 'Bearer YOUR_KEY'
  },
  body: JSON.stringify({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: 'Write a haiku about sunrise' }],
    stream: true
  })
})
.then(res => {
  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  
  function pump() {
    return reader.read().then(({ done, value }) => {
      if (done) return;
      console.log(decoder.decode(value, { stream: true }));
      return pump();
    });
  }
  
  return pump();
});

CLI with Streaming

The command-line interface supports real-time output through the --stream flag:

freellmapi chat --model=gpt-4o --stream "Give me a step-by-step plan to bake a chocolate cake"

Key Source Files for Streaming Implementation

Understanding these locations helps when debugging or extending streaming functionality:

• [server/src/providers/base.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/base.ts): Defines the abstract streamChatCompletion method that standardizes streaming interfaces.
• [server/src/providers/openai-compat.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts): Implements SSE parsing and chunk generation for OpenAI-compatible APIs.
• [server/src/providers/google.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/google.ts): Google-specific streaming implementation for Gemini models.
• [server/src/routes/proxy.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts): Forwards provider streams to clients through the proxy endpoint.
• [server/src/routes/responses.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/responses.ts): Handles non-proxy chat endpoints with equivalent streaming logic.
• [cli/src/index.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/cli/src/index.ts): CLI entry point parsing the --stream argument.
• [shared/types.ts](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts): TypeScript definitions for the stream boolean parameter in request schemas.

Summary

• FreeLLMAPI streaming is implemented through an abstract streamChatCompletion method defined in the base provider class.
• All supported providers (OpenAI, Google, Cohere, Cloudflare, AI-Horde, etc.) implement this method to yield asynchronous chunks.
• The routing layer in proxy.ts forwards Server-Sent Events directly to clients without buffering entire responses.
• Enable streaming by setting stream: true in API requests or using the --stream CLI flag.
• The architecture guarantees consistent streaming behavior regardless of the underlying LLM provider's native protocol.

Frequently Asked Questions

Does FreeLLMAPI support streaming for all providers?

Yes, the codebase requires every provider to implement the streamChatCompletion method. Whether you are using OpenAI-compatible endpoints, Google Gemini, Cohere, Cloudflare, or AI-Horde services, the streaming interface remains consistent through the abstraction layer defined in server/src/providers/base.ts.

How do I disable streaming if I need a complete response?

Omit the stream parameter or explicitly set stream: false in your request payload. When streaming is disabled, the API aggregates the entire response server-side before returning it as a single JSON object, which is useful for batch processing or when complete context is required before parsing.

What format does FreeLLMAPI use for streaming responses?

The API uses Server-Sent Events (SSE) format, following industry standards for LLM APIs. Each chunk contains a partial JSON object with a delta field containing the incremental content. Client applications concatenate these deltas to reconstruct the complete response progressively as tokens arrive.

Is streaming available in the FreeLLMAPI CLI tool?

Yes, the CLI supports real-time token display through the --stream flag. When enabled, the CLI consumes the SSE stream from the backend and prints tokens to stdout as they are generated, providing immediate visual feedback without waiting for the full completion to finish.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}