How FreeLLMAPI Handles SSE Streaming for Real-Time LLM Responses

FreeLLMAPI implements Server-Sent Events (SSE) streaming through a lazy header emission strategy that only commits the response after the upstream provider yields the first chunk, buffering content through event types like response.output_text.delta and response.function_call_arguments.delta while supporting inline tool-call dialect rescue.

The tashfeenahmed/freellmapi repository provides a unified OpenAI-compatible API for large language models with robust SSE streaming capabilities. This article examines how the /responses endpoint manages real-time token streaming, including request validation, chunk processing, and error recovery mechanisms implemented in the TypeScript source code.

SSE Streaming Architecture in FreeLLMAPI

Entry Point and Request Validation

The streaming pipeline begins in server/src/routes/responses.ts where the POST /responses handler processes incoming requests. The system validates the request body against responsesRequestSchema using responsesRequestSchema.safeParse before determining the response mode. The boolean flag const stream = reqData.stream ?? false; (line 223) controls whether the response returns as a single JSON object or as an SSE stream.

The SSE Helper Function

At lines 75-78, a dedicated helper function formats Server-Sent Event frames with proper event delimiters:

const sse = (event: string, payload: Record<string, unknown>) => {
  res.write(`event: ${event}\n`);
  res.write(`data: ${JSON.stringify({ type: event, sequence_number: seq++, ...payload })}\n\n`);
};

This function prefixes each payload with event: and data: fields, maintaining an incrementing sequence number that ensures clients can track message ordering.

Lazy Header Emission Strategy

FreeLLMAPI employs lazy header emission to avoid committing to an SSE stream before confirming upstream provider connectivity. The code tracks a streamStarted boolean flag, only setting Content-Type: text/event-stream and emitting initial response.created and response.in_progress events after receiving the first provider chunk (lines 68-80). This prevents hanging connections when providers fail at connection time.

Processing Streaming Chunks

Text Delta Handling

The provider's async generator route.provider.streamChatCompletion yields chunks that the system processes in a loop starting at line 83. For each chunk, the code extracts delta from chunk.choices?.[0]?.delta. When handling text content (delta.content), the system either passes it through directly in passthrough mode or buffers it in a dialect window to detect inline tool calls. Text deltas emit via response.output_text.delta events (lines 93-96).

Tool Call Accumulation

When delta.tool_calls appear, FreeLLMAPI creates a per-index accumulator (toolAcc) to track function call state. The first occurrence triggers a response.output_item.added event, while subsequent argument fragments stream as response.function_call_arguments.delta events (lines 28-34 and 36-39). This allows real-time streaming of partial JSON arguments before the complete tool call finalizes.

Dialect Rescue Mechanism

After the provider stream completes, the system examines any held text for inline tool-call dialects using rescueInlineToolCalls (lines 45-48). If dialectMode === 'dialect' or the text contains dialect markers, the function extracts synthetic function calls from prose text, emitting them as response.output_item.added events even when the provider didn't structure them as formal tool calls.

Stream Finalization and Error Handling

Completion Events

When streaming concludes, FreeLLMAPI emits a sequence of .done events to signal completion. The system calls sse('response.output_text.done', …) for text items and sse('response.function_call_arguments.done', …) followed by sse('response.output_item.done', …) for tool calls after repairing arguments with repairToolArguments (defined in server/src/lib/tool-args.ts). Finally, buildResponseObject assembles the complete response and emits response.completed before closing the connection (lines 41-43).

Retry Logic and Mid-Stream Failures

The SSE implementation includes sophisticated error handling through server/src/services/router.ts. If a provider error occurs before any SSE frames are sent, the request retries up to MAX_RETRIES with fallback logic. However, once the stream starts (streamStarted is true), errors trigger a response.failed event and immediate termination (lines 54-58). Exhausted retries similarly result in a final response.failed event.

Client-Side Integration

Clients consume the SSE stream by setting stream: true in the request body and parsing the event stream using the Fetch API:

fetch('https://api.example.com/v1/responses', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer YOUR_KEY' },
  body: JSON.stringify({
    model: 'gpt-oss-120b',
    input: [{ role: 'user', content: 'Explain SSE streaming' }],
    stream: true
  })
})
.then(res => {
  const reader = res.body.getReader();
  const decoder = new TextDecoder('utf-8');
  let buffer = '';

  function parseEvents(chunk) {
    buffer += decoder.decode(chunk, { stream: true });
    const events = buffer.split('\n\n');
    buffer = events.pop();
    for (const ev of events) {
      const lines = ev.split('\n');
      const type = lines[0].replace('event: ', '');
      const data = JSON.parse(lines[1].replace('data: ', ''));
      console.log('Event:', type, data);
    }
  }

  function pump() {
    return reader.read().then(({ done, value }) => {
      if (done) return;
      parseEvents(value);
      return pump();
    });
  }
  return pump();
});

The client receives ordered events including response.created, response.output_text.delta, response.function_call_arguments.delta, and response.completed, each containing structured JSON payloads with sequence numbers.

Summary

• FreeLLMAPI implements SSE streaming in server/src/routes/responses.ts with lazy header emission that only commits after successful provider connection
• The system uses a dedicated sse helper function to format Server-Sent Event frames with incrementing sequence numbers
• Text deltas stream through response.output_text.delta events while tool calls accumulate via response.function_call_arguments.delta using the toolAcc accumulator
• The dialect rescue mechanism in server/src/lib/tool-call-rescue.js extracts inline tool calls from plain text when providers embed them in prose
• Error handling distinguishes between pre-stream retries (up to MAX_RETRIES) and mid-stream failures, emitting response.failed events for the latter

Frequently Asked Questions

What triggers SSE mode in FreeLLMAPI?

SSE mode activates when the request body contains stream: true. The handler validates this via responsesRequestSchema and sets const stream = reqData.stream ?? false; to determine whether to invoke the streaming pipeline or return a complete JSON response.

How does FreeLLMAPI prevent empty SSE connections?

The implementation uses lazy header emission tracked by a streamStarted flag. Headers and initial events only transmit after the upstream provider yields the first chunk, preventing committed connections when providers fail at connection time.

Can FreeLLMAPI handle tool calls embedded in streaming text?

Yes. The dialect rescue mechanism examines held text after streaming completes, using rescueInlineToolCalls to detect and extract inline tool-call patterns. These are converted to formal response.output_item.added and response.function_call_arguments.delta events as if they were native tool calls.

What happens if a provider error occurs mid-stream?

Once SSE headers are sent and streaming begins, the system cannot retry. Instead, it emits a response.failed event and terminates the connection. Errors occurring before the first chunk trigger automatic retries up to MAX_RETRIES through the router service in server/src/services/router.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}