Streaming vs Non-Streaming RAG Response Modes: A Production Implementation Guide

Non-streaming RAG returns a complete JSON payload after the LLM finishes generating the full answer, while streaming RAG emits Server-Sent Events (SSE) with incremental text chunks for real-time user interfaces.

The jamwithai/production-agentic-rag-course repository implements both retrieval-augmented generation patterns through a FastAPI-based architecture. Understanding the difference between streaming and non-streaming RAG response modes is critical for production systems where synchronous batch processing and real-time chat interfaces have different latency requirements.

Core Architectural Differences

The implementation exposes two distinct HTTP endpoints that share the same retrieval pipeline but differ in how they return the LLM-generated content to the client.

Non-streaming mode (/api/v1/ask) waits for the complete generation before returning a structured AskResponse JSON object. This approach suits batch processing or synchronous UIs that require the full answer immediately.

Streaming mode (/api/v1/stream) returns a StreamingResponse that yields Server-Sent Events as the LLM produces tokens. This enables "typing" effects in chat interfaces and reduces perceived latency for end users.

Both modes query the same hybrid search backend and utilize the RAGTracer for observability, but they invoke different methods in the Ollama client and apply distinct prompt-building strategies.

Non-Streaming RAG Implementation

As implemented in src/routers/ask.py, the ask_question function handles the non-streaming flow. This endpoint constructs the full response before transmitting any data to the client.

The architectural flow follows these steps:

1. Cache verification – Checks cache_client.find_cached_response for an exact match of the query.
2. Hybrid retrieval – Executes opensearch_client.search_unified to fetch relevant document chunks.
3. Structured prompt building – Calls RAGPromptBuilder.create_structured_prompt defined in src/services/ollama/prompts.py (with fallback to create_rag_prompt) to format the context.
4. LLM generation – Invokes ollama_client.generate_rag_answer, which internally calls OllamaClient.generate with stream=False.
5. Response assembly – Returns a populated AskResponse model containing the complete answer, sources, and metadata.

In src/services/ollama/client.py (lines 93-112), the generate_rag_answer method performs a single HTTP POST to the Ollama API, waiting for the entire completion before returning. When the cache hits, the system returns the full cached AskResponse immediately without re-invoking the LLM.

Streaming RAG Implementation

The streaming implementation resides in the same router file but uses the ask_question_stream function to handle POST /api/v1/stream requests. This endpoint returns a StreamingResponse that maintains an open connection throughout the generation process.

The streaming architectural flow modifies the non-streaming pattern in key ways:

• Endpoint signature: async def ask_question_stream(...) -> StreamingResponse instead of AskResponse.
• Prompt strategy: Directly uses RAGPromptBuilder.create_rag_prompt (simpler format) rather than the structured variant, as streaming favors plain text generation.
• LLM invocation: Calls ollama_client.generate_rag_answer_stream, which invokes OllamaClient.generate_stream with stream=True and yields JSON chunks.
• SSE emission: Each chunk is forwarded as data: {"chunk": "..."}\n\n until the LLM sends "done": true, at which point the final answer block is emitted.

In src/services/ollama/client.py (lines 274-292), the generate_rag_answer_stream method handles the chunked HTTP response, parsing each JSON line and yielding text segments. The cache mechanism works identically, but when a cached answer exists, it streams the stored text chunk-by-chunk to maintain interface consistency.

Implementation Differences in Code

The code-level distinctions between the two modes center on async iteration patterns and response construction:

Function Signatures

• Non-streaming: async def ask_question(...) -> AskResponse returns the complete response object.
• Streaming: async def ask_question_stream(...) -> StreamingResponse yields events progressively.

LLM Invocation Pattern

• Non-streaming: Uses await ollama_client.generate_rag_answer(...) which blocks until the full text is generated.
• Streaming: Uses async for chunk in ollama_client.generate_rag_answer_stream(...): to iterate over token chunks.

Response Construction

• Non-streaming: Builds the AskResponse object once and returns it as JSON.
• Streaming: Yields metadata first, then each text chunk as SSE data, followed by a completion JSON payload containing "done": true.

Tracing Span Lifecycle

• Non-streaming: The RAGTracer span (defined in src/services/langfuse/tracer.py) ends immediately after the single LLM call completes.
• Streaming: The generation span remains open until the "done" chunk is received and processed.

Both implementations share the same dependency injection from src/dependencies.py for cache, embeddings, and OpenSearch clients, ensuring consistent retrieval behavior regardless of the response mode.

Practical Usage Examples

Non-Streaming API Call

Use this mode when your consumer can wait for the complete answer, such as in batch processing or synchronous API integrations:

curl -X POST https://my-api.example.com/api/v1/ask \
     -H "Content-Type: application/json" \
     -d '{"query":"What is retrieval-augmented generation?","model":"llama3.2:3b","use_hybrid":false}'

Response payload:

{
  "query": "What is retrieval-augmented generation?",
  "answer": "RAG combines retrieval systems with generative models...",
  "sources": ["https://arxiv.org/pdf/2104.12345.pdf"],
  "chunks_used": 7,
  "search_mode": "bm25"
}

Streaming API Call

Implement this pattern for chat interfaces requiring real-time text display:

const payload = {
  query: "Explain streaming RAG",
  model: "llama3.2:3b",
  use_hybrid: false
};

const response = await fetch('https://my-api.example.com/api/v1/stream', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Accept': 'text/plain' },
  body: JSON.stringify(payload)
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

let answer = '';
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const chunkText = decoder.decode(value, { stream: true });
  const lines = chunkText.split('\n').filter(l => l.startsWith('data:'));
  
  for (const line of lines) {
    const payload = JSON.parse(line.replace(/^data:\s*/, ''));
    if (payload.chunk) {
      answer += payload.chunk;
      console.log('Received:', payload.chunk);
    }
    if (payload.done) {
      console.log('Complete answer:', answer);
    }
  }
}

The streaming endpoint emits SSE-formatted data where each token arrives as it is generated, allowing immediate UI updates.

Summary

• Non-streaming RAG (/api/v1/ask) returns complete AskResponse JSON after full LLM generation, using OllamaClient.generate with stream=False and structured prompts in src/routers/ask.py.
• Streaming RAG (/api/v1/stream) yields Server-Sent Events through StreamingResponse, utilizing OllamaClient.generate_stream with stream=True and simpler prompts for real-time delivery.
• Both modes share identical retrieval, caching, and tracing infrastructure defined in src/services/ollama/client.py and src/services/langfuse/tracer.py, differing only in the LLM client invocation and response serialization patterns.
• Caching behavior is consistent across modes: hits return immediately in non-streaming, while streaming decomposes cached answers into chunks for progressive delivery.

Frequently Asked Questions

When should I use streaming versus non-streaming RAG?

Use streaming RAG when building conversational interfaces where users expect to see text appear progressively, reducing perceived latency. Use non-streaming RAG for batch processing, synchronous API integrations, or when downstream systems require the complete answer in a single atomic response from the jamwithai/production-agentic-rag-course API.

Does streaming affect the quality of the generated answer?

No, according to the source code implementation, both modes use the same underlying LLM and retrieval pipeline. The only difference is that streaming uses create_rag_prompt (simpler format) while non-streaming attempts create_structured_prompt first, but the model parameters and context retrieval remain identical.

How does caching work with streaming responses?

The cache check occurs before LLM invocation in both modes. When a cache hit occurs in streaming mode, the system decomposes the cached answer into chunks and streams them sequentially, ensuring the client receives the same progressive flow as a live generation would provide.

Can I switch between modes without changing my retrieval configuration?

Yes, both endpoints utilize the same dependency injection from src/dependencies.py for OpenSearch, embeddings, and cache clients. The retrieval phase (search_unified) and prompt building infrastructure are shared; you only need to change the endpoint URL from /api/v1/ask to /api/v1/stream and handle the SSE response format on the client side.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}