# Streaming vs Non-Streaming RAG Response Modes: A Production Implementation Guide

> Understand streaming vs non-streaming RAG response modes for production. Learn how to implement RAG effectively with incremental text or complete JSON payloads for better user experiences.

- Repository: [jamwithai/production-agentic-rag-course](https://github.com/jamwithai/production-agentic-rag-course)
- Tags: deep-dive
- Published: 2026-03-23

---

**Non-streaming RAG returns a complete JSON payload after the LLM finishes generating the full answer, while streaming RAG emits Server-Sent Events (SSE) with incremental text chunks for real-time user interfaces.**

The `jamwithai/production-agentic-rag-course` repository implements both retrieval-augmented generation patterns through a FastAPI-based architecture. Understanding the difference between streaming and non-streaming RAG response modes is critical for production systems where synchronous batch processing and real-time chat interfaces have different latency requirements.

## Core Architectural Differences

The implementation exposes two distinct HTTP endpoints that share the same retrieval pipeline but differ in how they return the LLM-generated content to the client.

**Non-streaming mode** (`/api/v1/ask`) waits for the complete generation before returning a structured `AskResponse` JSON object. This approach suits batch processing or synchronous UIs that require the full answer immediately.

**Streaming mode** (`/api/v1/stream`) returns a `StreamingResponse` that yields Server-Sent Events as the LLM produces tokens. This enables "typing" effects in chat interfaces and reduces perceived latency for end users.

Both modes query the same hybrid search backend and utilize the `RAGTracer` for observability, but they invoke different methods in the Ollama client and apply distinct prompt-building strategies.

## Non-Streaming RAG Implementation

As implemented in [`src/routers/ask.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/routers/ask.py), the `ask_question` function handles the non-streaming flow. This endpoint constructs the full response before transmitting any data to the client.

The architectural flow follows these steps:

1. **Cache verification** – Checks `cache_client.find_cached_response` for an exact match of the query.
2. **Hybrid retrieval** – Executes `opensearch_client.search_unified` to fetch relevant document chunks.
3. **Structured prompt building** – Calls `RAGPromptBuilder.create_structured_prompt` defined in [`src/services/ollama/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/ollama/prompts.py) (with fallback to `create_rag_prompt`) to format the context.
4. **LLM generation** – Invokes `ollama_client.generate_rag_answer`, which internally calls `OllamaClient.generate` with `stream=False`.
5. **Response assembly** – Returns a populated `AskResponse` model containing the complete answer, sources, and metadata.

In [`src/services/ollama/client.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/ollama/client.py) (lines 93-112), the `generate_rag_answer` method performs a single HTTP POST to the Ollama API, waiting for the entire completion before returning. When the cache hits, the system returns the full cached `AskResponse` immediately without re-invoking the LLM.

## Streaming RAG Implementation

The streaming implementation resides in the same router file but uses the `ask_question_stream` function to handle `POST /api/v1/stream` requests. This endpoint returns a `StreamingResponse` that maintains an open connection throughout the generation process.

The streaming architectural flow modifies the non-streaming pattern in key ways:

- **Endpoint signature**: `async def ask_question_stream(...) -> StreamingResponse` instead of `AskResponse`.
- **Prompt strategy**: Directly uses `RAGPromptBuilder.create_rag_prompt` (simpler format) rather than the structured variant, as streaming favors plain text generation.
- **LLM invocation**: Calls `ollama_client.generate_rag_answer_stream`, which invokes `OllamaClient.generate_stream` with `stream=True` and yields JSON chunks.
- **SSE emission**: Each chunk is forwarded as `data: {"chunk": "..."}\n\n` until the LLM sends `"done": true`, at which point the final answer block is emitted.

In [`src/services/ollama/client.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/ollama/client.py) (lines 274-292), the `generate_rag_answer_stream` method handles the chunked HTTP response, parsing each JSON line and yielding text segments. The cache mechanism works identically, but when a cached answer exists, it streams the stored text chunk-by-chunk to maintain interface consistency.

## Implementation Differences in Code

The code-level distinctions between the two modes center on async iteration patterns and response construction:

**Function Signatures**

- **Non-streaming**: `async def ask_question(...) -> AskResponse` returns the complete response object.
- **Streaming**: `async def ask_question_stream(...) -> StreamingResponse` yields events progressively.

**LLM Invocation Pattern**

- **Non-streaming**: Uses `await ollama_client.generate_rag_answer(...)` which blocks until the full text is generated.
- **Streaming**: Uses `async for chunk in ollama_client.generate_rag_answer_stream(...):` to iterate over token chunks.

**Response Construction**

- **Non-streaming**: Builds the `AskResponse` object once and returns it as JSON.
- **Streaming**: Yields metadata first, then each text chunk as SSE data, followed by a completion JSON payload containing `"done": true`.

**Tracing Span Lifecycle**

- **Non-streaming**: The `RAGTracer` span (defined in [`src/services/langfuse/tracer.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/langfuse/tracer.py)) ends immediately after the single LLM call completes.
- **Streaming**: The generation span remains open until the `"done"` chunk is received and processed.

Both implementations share the same dependency injection from [`src/dependencies.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/dependencies.py) for cache, embeddings, and OpenSearch clients, ensuring consistent retrieval behavior regardless of the response mode.

## Practical Usage Examples

### Non-Streaming API Call

Use this mode when your consumer can wait for the complete answer, such as in batch processing or synchronous API integrations:

```bash
curl -X POST https://my-api.example.com/api/v1/ask \
     -H "Content-Type: application/json" \
     -d '{"query":"What is retrieval-augmented generation?","model":"llama3.2:3b","use_hybrid":false}'

```

**Response payload:**

```json
{
  "query": "What is retrieval-augmented generation?",
  "answer": "RAG combines retrieval systems with generative models...",
  "sources": ["https://arxiv.org/pdf/2104.12345.pdf"],
  "chunks_used": 7,
  "search_mode": "bm25"
}

```

### Streaming API Call

Implement this pattern for chat interfaces requiring real-time text display:

```javascript
const payload = {
  query: "Explain streaming RAG",
  model: "llama3.2:3b",
  use_hybrid: false
};

const response = await fetch('https://my-api.example.com/api/v1/stream', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Accept': 'text/plain' },
  body: JSON.stringify(payload)
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

let answer = '';
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const chunkText = decoder.decode(value, { stream: true });
  const lines = chunkText.split('\n').filter(l => l.startsWith('data:'));
  
  for (const line of lines) {
    const payload = JSON.parse(line.replace(/^data:\s*/, ''));
    if (payload.chunk) {
      answer += payload.chunk;
      console.log('Received:', payload.chunk);
    }
    if (payload.done) {
      console.log('Complete answer:', answer);
    }
  }
}

```

The streaming endpoint emits SSE-formatted data where each token arrives as it is generated, allowing immediate UI updates.

## Summary

- **Non-streaming RAG** (`/api/v1/ask`) returns complete `AskResponse` JSON after full LLM generation, using `OllamaClient.generate` with `stream=False` and structured prompts in [`src/routers/ask.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/routers/ask.py).
- **Streaming RAG** (`/api/v1/stream`) yields Server-Sent Events through `StreamingResponse`, utilizing `OllamaClient.generate_stream` with `stream=True` and simpler prompts for real-time delivery.
- Both modes share identical retrieval, caching, and tracing infrastructure defined in [`src/services/ollama/client.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/ollama/client.py) and [`src/services/langfuse/tracer.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/langfuse/tracer.py), differing only in the LLM client invocation and response serialization patterns.
- **Caching behavior** is consistent across modes: hits return immediately in non-streaming, while streaming decomposes cached answers into chunks for progressive delivery.

## Frequently Asked Questions

### When should I use streaming versus non-streaming RAG?

Use **streaming RAG** when building conversational interfaces where users expect to see text appear progressively, reducing perceived latency. Use **non-streaming RAG** for batch processing, synchronous API integrations, or when downstream systems require the complete answer in a single atomic response from the `jamwithai/production-agentic-rag-course` API.

### Does streaming affect the quality of the generated answer?

No, according to the source code implementation, both modes use the same underlying LLM and retrieval pipeline. The only difference is that streaming uses `create_rag_prompt` (simpler format) while non-streaming attempts `create_structured_prompt` first, but the model parameters and context retrieval remain identical.

### How does caching work with streaming responses?

The cache check occurs before LLM invocation in both modes. When a cache hit occurs in streaming mode, the system decomposes the cached answer into chunks and streams them sequentially, ensuring the client receives the same progressive flow as a live generation would provide.

### Can I switch between modes without changing my retrieval configuration?

Yes, both endpoints utilize the same dependency injection from [`src/dependencies.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/dependencies.py) for OpenSearch, embeddings, and cache clients. The retrieval phase (`search_unified`) and prompt building infrastructure are shared; you only need to change the endpoint URL from `/api/v1/ask` to `/api/v1/stream` and handle the SSE response format on the client side.