# How FreeLLMAPI Handles SSE Streaming for Real-Time LLM Responses

> Discover how FreeLLMAPI uses SSE streaming for real-time LLM responses. Learn about its lazy header strategy, buffering techniques, and support for tool-call dialect rescue.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-29

---

**FreeLLMAPI implements Server-Sent Events (SSE) streaming through a lazy header emission strategy that only commits the response after the upstream provider yields the first chunk, buffering content through event types like `response.output_text.delta` and `response.function_call_arguments.delta` while supporting inline tool-call dialect rescue.**

The tashfeenahmed/freellmapi repository provides a unified OpenAI-compatible API for large language models with robust SSE streaming capabilities. This article examines how the `/responses` endpoint manages real-time token streaming, including request validation, chunk processing, and error recovery mechanisms implemented in the TypeScript source code.

## SSE Streaming Architecture in FreeLLMAPI

### Entry Point and Request Validation

The streaming pipeline begins in [`server/src/routes/responses.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/responses.ts) where the `POST /responses` handler processes incoming requests. The system validates the request body against `responsesRequestSchema` using `responsesRequestSchema.safeParse` before determining the response mode. The boolean flag `const stream = reqData.stream ?? false;` (line 223) controls whether the response returns as a single JSON object or as an SSE stream.

### The SSE Helper Function

At lines 75-78, a dedicated helper function formats Server-Sent Event frames with proper event delimiters:

```typescript
const sse = (event: string, payload: Record<string, unknown>) => {
  res.write(`event: ${event}\n`);
  res.write(`data: ${JSON.stringify({ type: event, sequence_number: seq++, ...payload })}\n\n`);
};

```

This function prefixes each payload with `event:` and `data:` fields, maintaining an incrementing sequence number that ensures clients can track message ordering.

### Lazy Header Emission Strategy

FreeLLMAPI employs **lazy header emission** to avoid committing to an SSE stream before confirming upstream provider connectivity. The code tracks a `streamStarted` boolean flag, only setting `Content-Type: text/event-stream` and emitting initial `response.created` and `response.in_progress` events after receiving the first provider chunk (lines 68-80). This prevents hanging connections when providers fail at connection time.

## Processing Streaming Chunks

### Text Delta Handling

The provider's async generator `route.provider.streamChatCompletion` yields chunks that the system processes in a loop starting at line 83. For each chunk, the code extracts `delta` from `chunk.choices?.[0]?.delta`. When handling text content (`delta.content`), the system either passes it through directly in `passthrough` mode or buffers it in a dialect window to detect inline tool calls. Text deltas emit via `response.output_text.delta` events (lines 93-96).

### Tool Call Accumulation

When `delta.tool_calls` appear, FreeLLMAPI creates a per-index accumulator (`toolAcc`) to track function call state. The first occurrence triggers a `response.output_item.added` event, while subsequent argument fragments stream as `response.function_call_arguments.delta` events (lines 28-34 and 36-39). This allows real-time streaming of partial JSON arguments before the complete tool call finalizes.

### Dialect Rescue Mechanism

After the provider stream completes, the system examines any held text for inline tool-call dialects using `rescueInlineToolCalls` (lines 45-48). If `dialectMode === 'dialect'` or the text contains dialect markers, the function extracts synthetic function calls from prose text, emitting them as `response.output_item.added` events even when the provider didn't structure them as formal tool calls.

## Stream Finalization and Error Handling

### Completion Events

When streaming concludes, FreeLLMAPI emits a sequence of `.done` events to signal completion. The system calls `sse('response.output_text.done', …)` for text items and `sse('response.function_call_arguments.done', …)` followed by `sse('response.output_item.done', …)` for tool calls after repairing arguments with `repairToolArguments` (defined in [`server/src/lib/tool-args.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-args.ts)). Finally, `buildResponseObject` assembles the complete response and emits `response.completed` before closing the connection (lines 41-43).

### Retry Logic and Mid-Stream Failures

The SSE implementation includes sophisticated error handling through [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts). If a provider error occurs before any SSE frames are sent, the request retries up to `MAX_RETRIES` with fallback logic. However, once the stream starts (`streamStarted` is true), errors trigger a `response.failed` event and immediate termination (lines 54-58). Exhausted retries similarly result in a final `response.failed` event.

## Client-Side Integration

Clients consume the SSE stream by setting `stream: true` in the request body and parsing the event stream using the Fetch API:

```javascript
fetch('https://api.example.com/v1/responses', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer YOUR_KEY' },
  body: JSON.stringify({
    model: 'gpt-oss-120b',
    input: [{ role: 'user', content: 'Explain SSE streaming' }],
    stream: true
  })
})
.then(res => {
  const reader = res.body.getReader();
  const decoder = new TextDecoder('utf-8');
  let buffer = '';

  function parseEvents(chunk) {
    buffer += decoder.decode(chunk, { stream: true });
    const events = buffer.split('\n\n');
    buffer = events.pop();
    for (const ev of events) {
      const lines = ev.split('\n');
      const type = lines[0].replace('event: ', '');
      const data = JSON.parse(lines[1].replace('data: ', ''));
      console.log('Event:', type, data);
    }
  }

  function pump() {
    return reader.read().then(({ done, value }) => {
      if (done) return;
      parseEvents(value);
      return pump();
    });
  }
  return pump();
});

```

The client receives ordered events including `response.created`, `response.output_text.delta`, `response.function_call_arguments.delta`, and `response.completed`, each containing structured JSON payloads with sequence numbers.

## Summary

- FreeLLMAPI implements SSE streaming in [`server/src/routes/responses.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/responses.ts) with lazy header emission that only commits after successful provider connection
- The system uses a dedicated `sse` helper function to format Server-Sent Event frames with incrementing sequence numbers
- Text deltas stream through `response.output_text.delta` events while tool calls accumulate via `response.function_call_arguments.delta` using the `toolAcc` accumulator
- The dialect rescue mechanism in [`server/src/lib/tool-call-rescue.js`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/lib/tool-call-rescue.js) extracts inline tool calls from plain text when providers embed them in prose
- Error handling distinguishes between pre-stream retries (up to `MAX_RETRIES`) and mid-stream failures, emitting `response.failed` events for the latter

## Frequently Asked Questions

### What triggers SSE mode in FreeLLMAPI?

SSE mode activates when the request body contains `stream: true`. The handler validates this via `responsesRequestSchema` and sets `const stream = reqData.stream ?? false;` to determine whether to invoke the streaming pipeline or return a complete JSON response.

### How does FreeLLMAPI prevent empty SSE connections?

The implementation uses lazy header emission tracked by a `streamStarted` flag. Headers and initial events only transmit after the upstream provider yields the first chunk, preventing committed connections when providers fail at connection time.

### Can FreeLLMAPI handle tool calls embedded in streaming text?

Yes. The dialect rescue mechanism examines held text after streaming completes, using `rescueInlineToolCalls` to detect and extract inline tool-call patterns. These are converted to formal `response.output_item.added` and `response.function_call_arguments.delta` events as if they were native tool calls.

### What happens if a provider error occurs mid-stream?

Once SSE headers are sent and streaming begins, the system cannot retry. Instead, it emits a `response.failed` event and terminates the connection. Errors occurring before the first chunk trigger automatic retries up to `MAX_RETRIES` through the router service in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts).