# How FreeLLMAPI Supports Streaming Responses: An OpenAI-Compatible SSE Implementation

> Discover how FreeLLMAPI enables streaming responses with its OpenAI-compatible SSE endpoint. Learn how it processes AsyncGenerator chunks for efficient, real-time data delivery.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-27

---

**FreeLLMAPI enables streaming responses by exposing an OpenAI-compatible Server-Sent Events (SSE) endpoint that detects the `stream` flag in incoming requests, sets `Content-Type: text/event-stream`, and pipes AsyncGenerator chunks from provider-specific implementations through a unified proxy layer that normalizes them into standard SSE frames.**

FreeLLMAPI is an open-source proxy aggregation layer that unifies multiple free LLM providers under a single OpenAI-compatible API. Understanding how the project handles **streaming responses** reveals its sophisticated SSE pipeline that transforms disparate provider implementations into a consistent real-time token delivery system, as implemented in the `tashfeenahmed/freellmapi` repository.

## The Streaming Architecture Overview

FreeLLMAPI implements streaming through a layered approach that transforms provider-native streams into standardized SSE frames. The architecture consists of route-level detection, provider-specific AsyncGenerators, and centralized chunk normalization that ensures compatibility with OpenAI SDK clients.

### Route Detection and Header Configuration

In [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts), the HTTP handler for `POST /v1/chat/completions` inspects the incoming request body for the `stream` boolean flag (lines 30-38). When detected, the proxy immediately configures the response headers for SSE transmission by setting `Content-Type: text/event-stream` and withholding the initial data frame until actual content arrives (lines 38-49).

This header preparation guarantees that clients receive a well-formed SSE stream from the first byte, preventing premature connection termination and ensuring compatibility with standard SSE parsers.

### Provider Streaming Implementation

Each provider implements the abstract `streamChatCompletion` method defined in [`server/src/providers/base.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/base.ts), which returns an **AsyncGenerator** yielding `ChatCompletionChunk` objects. The base class supplies a reusable SSE parser via the `readSseStream` helper (lines 18-70), which handles the low-level byte parsing of provider responses.

For OpenAI-compatible providers (including Groq, Mistral, and Cohere), the concrete implementation resides in [`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts) (lines 167-194). This adapter forwards the raw HTTP response to the generic SSE reader while maintaining provider-specific authentication and endpoint routing.

### Chunk Processing and Error Resilience

As the generator yields chunks, the proxy iterates through them in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 60-71). For each chunk, the helper `streamChunkText` extracts incremental text content, updates internal token counters, and constructs a `legacyCompletionChunk` object that conforms to OpenAI's delta format.

If a provider aborts mid-stream, the proxy catches the error in the same file (lines 96-113), writes an error frame to the SSE stream, and finalizes the response. This ensures clients receive graceful termination signals rather than abrupt connection drops, allowing for proper fallback logic to alternate providers.

### Stream Termination Protocol

When the AsyncGenerator exhausts, the proxy writes the terminating `data: [DONE]` line and ends the HTTP response (lines 73-79). This final frame signals to OpenAI SDK clients that the generation is complete, matching the behavior of the official OpenAI API and ensuring seamless integration with existing client libraries.

## Client Implementation Examples

You can consume FreeLLMAPI's streaming endpoint using any HTTP client that supports Server-Sent Events, or through official SDKs configured with the local base URL.

### Raw curl Request

Use the `-N` flag to disable buffering and view tokens as they arrive:

```bash
curl -N -X POST http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer $FREELLMAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-3.5-turbo",
        "messages": [{"role":"user","content":"Explain streaming in FreeLLMAPI"}],
        "stream": true
      }'

```

The server streams chunks formatted as:

```

data: {"id":"...","choices":[{"delta":{"content":"Free"},"finish_reason":null}],"object":"chat.completion.chunk"}
...
data: [DONE]

```

### Node.js with OpenAI SDK

```javascript
import OpenAI from "openai";

const client = new OpenAI({ 
  apiKey: process.env.FREELLMAPI_KEY, 
  baseURL: "http://localhost:3001/v1" 
});

const stream = await client.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [{ role: "user", content: "Show me streaming in FreeLLMAPI" }],
  stream: true,
});

for await (const part of stream) {
  if (part.choices[0].delta?.content) {
    process.stdout.write(part.choices[0].delta.content);
  }
}

```

The SDK automatically parses the SSE stream into async-iterable objects, exposing the same `delta.content` fields that FreeLLMAPI constructs in its proxy layer.

### Python Implementation

```python
import openai, os

openai.api_key = os.getenv("FREELLMAPI_KEY")
openai.base_url = "http://localhost:3001/v1"

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Give a streaming demo"}],
    stream=True,
)

for chunk in response:
    if delta := chunk.choices[0].delta.get("content"):
        print(delta, end="", flush=True)

```

All three examples target the same `/v1/chat/completions` endpoint with `stream: true`, triggering the server-side logic described above to deliver real-time token chunks.

## Key Files in the Streaming Pipeline

- **[`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts)** – Handles incoming requests, detects the `stream` flag, manages SSE headers, and forwards chunks to clients while handling mid-stream errors.
- **[`server/src/providers/base.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/base.ts)** – Defines the abstract `streamChatCompletion` signature and provides the `readSseStream` generic parser for SSE byte streams.
- **[`server/src/providers/openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/providers/openai-compat.ts)** – Implements streaming for OpenAI-compatible providers, returning AsyncGenerators that yield standardized chunks.
- **[`shared/types.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/shared/types.ts)** – Contains TypeScript definitions for `ChatCompletionChunk` and the request schema including the `stream` boolean flag.

## Summary

- **FreeLLMAPI detects streaming requests** by parsing the `stream` boolean in `POST /v1/chat/completions` at [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 30-38).
- **SSE headers are configured** immediately to set `Content-Type: text/event-stream` before any data transmission begins (lines 38-49).
- **Provider adapters return AsyncGenerators** yielding `ChatCompletionChunk` objects, with OpenAI-compatible implementations in [`openai-compat.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/openai-compat.ts) (lines 167-194).
- **Chunk normalization** occurs via `streamChunkText` and `legacyCompletionChunk` construction, ensuring OpenAI SDK compatibility (lines 60-71).
- **Graceful error handling** catches provider aborts mid-stream and writes error frames before closing the connection (lines 96-113).
- **Standard termination** follows OpenAI conventions with a final `data: [DONE]` frame signaling completion (lines 73-79).

## Frequently Asked Questions

### What protocol does FreeLLMAPI use for streaming responses?

FreeLLMAPI uses **Server-Sent Events (SSE)**, an HTTP-based protocol where the server pushes data to the client using the `text/event-stream` content type. The implementation follows OpenAI's streaming specification, emitting JSON-encoded completion chunks prefixed with `data:` and terminating with `data: [DONE]`.

### How does FreeLLMAPI handle errors during an active stream?

If a provider connection fails mid-stream, the proxy layer in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 96-113) catches the exception, writes an error frame to the SSE stream in the standard format, and finalizes the response. This allows client applications to detect failures gracefully and potentially retry with fallback providers.

### Is FreeLLMAPI's streaming implementation compatible with the official OpenAI SDK?

Yes. FreeLLMAPI implements the exact SSE format that OpenAI SDKs expect, including the `delta` field structure within `choices` arrays and the `[DONE]` termination signal. Both Node.js and Python OpenAI clients can consume FreeLLMAPI streams by simply changing the `baseURL` to point to the local proxy.

### Can I use FreeLLMAPI streaming without an OpenAI SDK?

Absolutely. Because FreeLLMAPI uses standard HTTP Server-Sent Events, any HTTP client that can handle streaming responses (such as `curl` with the `-N` flag, `fetch` with the `ReadableStream` API, or Python's `requests` with `iter_lines`) can consume the streaming endpoint directly by parsing the `data:` prefixed JSON lines.