How FreeLLMAPI Supports Streaming Responses: An OpenAI-Compatible SSE Implementation

FreeLLMAPI enables streaming responses by exposing an OpenAI-compatible Server-Sent Events (SSE) endpoint that detects the stream flag in incoming requests, sets Content-Type: text/event-stream, and pipes AsyncGenerator chunks from provider-specific implementations through a unified proxy layer that normalizes them into standard SSE frames.

FreeLLMAPI is an open-source proxy aggregation layer that unifies multiple free LLM providers under a single OpenAI-compatible API. Understanding how the project handles streaming responses reveals its sophisticated SSE pipeline that transforms disparate provider implementations into a consistent real-time token delivery system, as implemented in the tashfeenahmed/freellmapi repository.

The Streaming Architecture Overview

FreeLLMAPI implements streaming through a layered approach that transforms provider-native streams into standardized SSE frames. The architecture consists of route-level detection, provider-specific AsyncGenerators, and centralized chunk normalization that ensures compatibility with OpenAI SDK clients.

Route Detection and Header Configuration

In server/src/routes/proxy.ts, the HTTP handler for POST /v1/chat/completions inspects the incoming request body for the stream boolean flag (lines 30-38). When detected, the proxy immediately configures the response headers for SSE transmission by setting Content-Type: text/event-stream and withholding the initial data frame until actual content arrives (lines 38-49).

This header preparation guarantees that clients receive a well-formed SSE stream from the first byte, preventing premature connection termination and ensuring compatibility with standard SSE parsers.

Provider Streaming Implementation

Each provider implements the abstract streamChatCompletion method defined in server/src/providers/base.ts, which returns an AsyncGenerator yielding ChatCompletionChunk objects. The base class supplies a reusable SSE parser via the readSseStream helper (lines 18-70), which handles the low-level byte parsing of provider responses.

For OpenAI-compatible providers (including Groq, Mistral, and Cohere), the concrete implementation resides in server/src/providers/openai-compat.ts (lines 167-194). This adapter forwards the raw HTTP response to the generic SSE reader while maintaining provider-specific authentication and endpoint routing.

Chunk Processing and Error Resilience

As the generator yields chunks, the proxy iterates through them in server/src/routes/proxy.ts (lines 60-71). For each chunk, the helper streamChunkText extracts incremental text content, updates internal token counters, and constructs a legacyCompletionChunk object that conforms to OpenAI's delta format.

If a provider aborts mid-stream, the proxy catches the error in the same file (lines 96-113), writes an error frame to the SSE stream, and finalizes the response. This ensures clients receive graceful termination signals rather than abrupt connection drops, allowing for proper fallback logic to alternate providers.

Stream Termination Protocol

When the AsyncGenerator exhausts, the proxy writes the terminating data: [DONE] line and ends the HTTP response (lines 73-79). This final frame signals to OpenAI SDK clients that the generation is complete, matching the behavior of the official OpenAI API and ensuring seamless integration with existing client libraries.

Client Implementation Examples

You can consume FreeLLMAPI's streaming endpoint using any HTTP client that supports Server-Sent Events, or through official SDKs configured with the local base URL.

Raw curl Request

Use the -N flag to disable buffering and view tokens as they arrive:

curl -N -X POST http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer $FREELLMAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-3.5-turbo",
        "messages": [{"role":"user","content":"Explain streaming in FreeLLMAPI"}],
        "stream": true
      }'

The server streams chunks formatted as:

data: {"id":"...","choices":[{"delta":{"content":"Free"},"finish_reason":null}],"object":"chat.completion.chunk"}
...
data: [DONE]

Node.js with OpenAI SDK

import OpenAI from "openai";

const client = new OpenAI({ 
  apiKey: process.env.FREELLMAPI_KEY, 
  baseURL: "http://localhost:3001/v1" 
});

const stream = await client.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [{ role: "user", content: "Show me streaming in FreeLLMAPI" }],
  stream: true,
});

for await (const part of stream) {
  if (part.choices[0].delta?.content) {
    process.stdout.write(part.choices[0].delta.content);
  }
}

The SDK automatically parses the SSE stream into async-iterable objects, exposing the same delta.content fields that FreeLLMAPI constructs in its proxy layer.

Python Implementation

import openai, os

openai.api_key = os.getenv("FREELLMAPI_KEY")
openai.base_url = "http://localhost:3001/v1"

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Give a streaming demo"}],
    stream=True,
)

for chunk in response:
    if delta := chunk.choices[0].delta.get("content"):
        print(delta, end="", flush=True)

All three examples target the same /v1/chat/completions endpoint with stream: true, triggering the server-side logic described above to deliver real-time token chunks.

Key Files in the Streaming Pipeline

• server/src/routes/proxy.ts – Handles incoming requests, detects the stream flag, manages SSE headers, and forwards chunks to clients while handling mid-stream errors.
• server/src/providers/base.ts – Defines the abstract streamChatCompletion signature and provides the readSseStream generic parser for SSE byte streams.
• server/src/providers/openai-compat.ts – Implements streaming for OpenAI-compatible providers, returning AsyncGenerators that yield standardized chunks.
• shared/types.ts – Contains TypeScript definitions for ChatCompletionChunk and the request schema including the stream boolean flag.

Summary

• FreeLLMAPI detects streaming requests by parsing the stream boolean in POST /v1/chat/completions at server/src/routes/proxy.ts (lines 30-38).
• SSE headers are configured immediately to set Content-Type: text/event-stream before any data transmission begins (lines 38-49).
• Provider adapters return AsyncGenerators yielding ChatCompletionChunk objects, with OpenAI-compatible implementations in openai-compat.ts (lines 167-194).
• Chunk normalization occurs via streamChunkText and legacyCompletionChunk construction, ensuring OpenAI SDK compatibility (lines 60-71).
• Graceful error handling catches provider aborts mid-stream and writes error frames before closing the connection (lines 96-113).
• Standard termination follows OpenAI conventions with a final data: [DONE] frame signaling completion (lines 73-79).

Frequently Asked Questions

What protocol does FreeLLMAPI use for streaming responses?

FreeLLMAPI uses Server-Sent Events (SSE), an HTTP-based protocol where the server pushes data to the client using the text/event-stream content type. The implementation follows OpenAI's streaming specification, emitting JSON-encoded completion chunks prefixed with data: and terminating with data: [DONE].

How does FreeLLMAPI handle errors during an active stream?

If a provider connection fails mid-stream, the proxy layer in server/src/routes/proxy.ts (lines 96-113) catches the exception, writes an error frame to the SSE stream in the standard format, and finalizes the response. This allows client applications to detect failures gracefully and potentially retry with fallback providers.

Is FreeLLMAPI's streaming implementation compatible with the official OpenAI SDK?

Yes. FreeLLMAPI implements the exact SSE format that OpenAI SDKs expect, including the delta field structure within choices arrays and the [DONE] termination signal. Both Node.js and Python OpenAI clients can consume FreeLLMAPI streams by simply changing the baseURL to point to the local proxy.

Can I use FreeLLMAPI streaming without an OpenAI SDK?

Absolutely. Because FreeLLMAPI uses standard HTTP Server-Sent Events, any HTTP client that can handle streaming responses (such as curl with the -N flag, fetch with the ReadableStream API, or Python's requests with iter_lines) can consume the streaming endpoint directly by parsing the data: prefixed JSON lines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}