Does Fabric Support Streaming AI Responses? A Complete Technical Guide

Yes, Fabric supports streaming AI responses natively via server-sent events (SSE) when the Stream flag is enabled on a Chatter instance, routing incremental chunks through provider-specific SendStream implementations.

The danielmiessler/fabric repository treats real-time output as a first-class architectural feature. By leveraging Go channels and a unified StreamUpdate domain model, the framework enables token-by-token consumption across OpenAI, Anthropic, Ollama, and other supported vendors without requiring changes to higher-level application code.

How Streaming Works in Fabric

Fabric’s streaming pipeline operates across four distinct architectural layers, each responsible for a specific phase of the data flow from AI provider to end client.

The Domain Layer: StreamUpdate Messages

At the foundation, internal/domain/stream.go defines the StreamUpdate struct—a standardized payload that carries incremental content, usage statistics, or error states from any vendor:

type StreamUpdate struct {
    Type    StreamType // StreamTypeContent, StreamTypeUsage, StreamTypeError
    Content string
    Usage   *Usage
}

This abstraction allows the core chatter logic to remain vendor-agnostic while handling real-time data.

The Core Chatter Logic

In internal/core/chatter.go, the Chatter struct orchestrates the request flow. When initialized with Stream: true (set via registry.GetChatter(..., stream=true, ...)), the Send method opens a responseChan and delegates to the vendor’s SendStream implementation:

responseChan := make(chan domain.StreamUpdate)
go vendor.SendStream(messages, opts, responseChan)

for upd := range responseChan {
    // Forward to UpdateChan and aggregate final message
    opts.UpdateChan <- upd
}

The caller receives each chunk through opts.UpdateChan while the chatter simultaneously builds the complete response for final return.

Vendor Implementations

Every AI provider in internal/plugins/ai/ implements the SendStream method defined in internal/plugins/ai/vendor.go. For example, in internal/plugins/ai/openai/openai.go, the method opens the provider’s streaming endpoint (e.g., chat/completions?stream=true), reads Server-Sent Events from OpenAI, and pushes domain.StreamUpdate structs onto the channel. The same pattern appears in internal/plugins/ai/anthropic/anthropic.go, internal/plugins/ai/ollama/ollama.go, and internal/plugins/ai/vertexai/vertexai.go, ensuring consistent behavior across local and cloud models.

REST API SSE Handler

The HTTP layer in internal/server/chat.go exposes a /chat endpoint that returns Content-Type: text/event-stream. It creates a per-request streamChan, passes it to the Chatter, and converts each StreamUpdate into a JSON StreamResponse written to the HTTP response:

for update := range streamChan {
    writeSSEResponse(w, StreamResponse{
        Type:    string(update.Type),
        Content: update.Content,
        Usage:   update.Usage,
    })
}

Clients receive a continuous stream of JSON lines that can be rendered progressively in real time.

Enabling Streaming in Your Code

You can consume streaming AI responses through the Go library directly or via the REST API.

Using the Go Library

Configure a Chatter with streaming enabled and read from the UpdateChan:

chatter, err := registry.GetChatter("gpt-4o-mini", 0, "openai", "", true, false)
if err != nil {
    log.Fatal(err)
}

opts := &domain.ChatOptions{
    Model:      "gpt-4o-mini",
    UpdateChan: make(chan domain.StreamUpdate, 10),
}

// Consume streaming chunks
go func() {
    for upd := range opts.UpdateChan {
        switch upd.Type {
        case domain.StreamTypeContent:
            fmt.Print(upd.Content)
        case domain.StreamTypeUsage:
            fmt.Printf("\nTokens: %d in, %d out\n", 
                upd.Usage.InputTokens, upd.Usage.OutputTokens)
        case domain.StreamTypeError:
            log.Printf("Stream error: %s", upd.Content)
        }
    }
}()

_, err = chatter.Send(req, opts)

Key parameters:

• Stream flag: The fifth argument to GetChatter(..., stream=true, ...) activates streaming mode.
• UpdateChan: A buffered or unbuffered channel receiving domain.StreamUpdate messages.

Calling the HTTP Streaming Endpoint

Use curl with the -N flag to disable buffering and receive SSE lines immediately:

curl -N -H "Content-Type: application/json" \
  -d '{
    "prompts":[{
      "userInput":"Explain quantum computing",
      "vendor":"anthropic",
      "model":"claude-3-5-sonnet-20241022"
    }]
  }' \
  http://localhost:8080/chat

The response emits incremental JSON objects:

{"type":"content","format":"markdown","content":"Quantum computing leverages..."}
{"type":"content","format":"markdown","content":" superposition and entanglement..."}
{"type":"usage","usage":{"input_tokens":12,"output_tokens":150,"total_tokens":162}}
{"type":"complete","format":"plain","content":""}

Implementing Streaming in a Custom Vendor

To add support for a new LLM provider, implement the Vendor interface in internal/plugins/ai/vendor.go:

type Vendor interface {
    Send([]*chat.ChatCompletionMessage, *domain.ChatOptions) (string, error)
    SendStream([]*chat.ChatCompletionMessage, *domain.ChatOptions, chan domain.StreamUpdate) error
}

The SendStream method must push StreamUpdate structs to the provided channel for each incremental chunk received from the provider’s API. Once implemented, the core and REST layers automatically handle SSE transmission without additional modifications.

Supported AI Providers with Streaming

Fabric’s plugin architecture enables streaming across the entire ecosystem of supported models:

• OpenAI (internal/plugins/ai/openai/openai.go) – GPT-4, GPT-4o, GPT-4o-mini
• Anthropic (internal/plugins/ai/anthropic/anthropic.go) – Claude 3.5 Sonnet, Claude 3 Opus
• Ollama (internal/plugins/ai/ollama/ollama.go) – Local Llama, Mistral, and other open-source models
• Google Vertex AI (internal/plugins/ai/vertexai/vertexai.go) – Gemini models
• AWS Bedrock, Perplexity, GitHub Copilot, and additional providers via the same interface

Because all vendors conform to the SendStream contract, switching between providers requires only changing the vendor name and model identifier—streaming behavior remains consistent.

Summary

• Fabric supports streaming AI responses through a dedicated Stream flag on Chatter instances that activates SSE-based transmission.
• Architecture: The flow moves from vendor-specific SendStream implementations through domain.StreamUpdate channels to the HTTP layer’s SSE handler.
• Key files: internal/core/chatter.go orchestrates streaming, internal/domain/stream.go defines the message format, and internal/server/chat.go handles the HTTP SSE endpoint.
• Universal support: All major providers (OpenAI, Anthropic, Ollama, Vertex AI) implement the same SendStream interface, enabling seamless provider switching.
• Integration: Use registry.GetChatter(..., stream=true, ...) in Go or POST to /chat with -N in curl to receive real-time token streams.

Frequently Asked Questions

How do I enable streaming in a Fabric chat request?

Set the Stream parameter to true when calling registry.GetChatter(model, temperature, vendor, pattern, stream, dryRun). In the REST API, streaming is automatically enabled for the /chat endpoint, which returns text/event-stream headers and emits JSON lines as the AI generates content.

What data structure carries streaming chunks in Fabric?

The domain.StreamUpdate struct (defined in internal/domain/stream.go) transports each chunk. It includes a Type field (content, usage, or error), the text Content, and a Usage pointer containing token counts. Vendors push these structs through Go channels that the chatter and HTTP handler consume.

Can I stream responses from local LLMs using Fabric?

Yes. The Ollama vendor implementation in internal/plugins/ai/ollama/ollama.go supports streaming from local models. Configure the chatter with vendor: "ollama" and any local model name (e.g., llama3.1), and the SendStream method will deliver incremental output from your local Ollama instance through the same SSE pipeline used for cloud providers.

Does Fabric handle errors during a streaming session?

Yes. Vendors can emit StreamUpdate messages with Type: StreamTypeError at any point during the stream. In internal/core/chatter.go, these errors are forwarded through the UpdateChan to the caller, allowing real-time error handling without terminating the entire session. The REST API similarly forwards error messages as SSE events with "type":"error".

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}