# How nGPT Implements Streaming Responses and Real-Time Markdown Rendering

> Discover how nGPT streams responses and renders markdown in real-time using SSE and Live display panels. Explore the technical details in renderers.py.

- Repository: [nazDridoy/ngpt](https://github.com/nazdridoy/ngpt)
- Tags: internals
- Published: 2026-03-07

---

**nGPT achieves real-time markdown rendering by combining Server-Sent Events (SSE) streaming from OpenAI-compatible APIs with Rich's Live display panels, coordinated through thread-safe callbacks in [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py).**

The nGPT CLI tool provides a seamless streaming experience that displays AI responses as they arrive, formatted as markdown in real-time. This article examines the underlying mechanism for streaming responses and real-time markdown rendering in nGPT, breaking down the architecture into three coordinated components: HTTP-SSE streaming, live UI rendering, and thread-safe coordination.

## Core Architecture Overview

nGPT's streaming capability relies on three integrated pieces working together:

- **HTTP-SSE Streaming Layer** – Handles Server-Sent Events from the API and forwards chunks via callbacks
- **Live Markdown UI Layer** – Manages Rich's `Live` display for real-time markdown rendering
- **Thread-Safe Coordination** – Ensures the spinner, live panel, and terminal output never conflict

## HTTP-SSE Streaming Layer

The streaming process begins in [`ngpt/api/client.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/api/client.py), where the `NGPTClient.chat()` method handles the HTTP request to OpenAI-compatible endpoints.

When `stream=True` is passed, the client:

1. Sends the request with streaming enabled
2. Reads the response line-by-line as Server-Sent Events (SSE)
3. Parses each JSON chunk to extract the `content` field
4. Accumulates the text and forwards it to the `stream_callback` function

This design decouples the network layer from the UI layer. The client knows nothing about markdown rendering; it simply delivers text chunks as they arrive from the API.

## Live Markdown UI Layer

The visual magic happens in [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py), specifically in the `prettify_streaming_markdown()` function. This function sets up a Rich `Live` display that updates in real-time as new content arrives.

The implementation creates three objects:

- `live_display` – A `rich.live.Live` context manager wrapping a `Panel(Markdown(...))`
- `update_content` – A callback function that rewrites the Markdown renderable and calls `live.refresh()`
- `setup_spinner` – A factory for creating the initial loading spinner

When `update_content()` receives new text, it:

1. Truncates content to visible height if necessary
2. Updates the Markdown renderable with the accumulated text
3. Calls `live.refresh()` to force immediate terminal redraw

This creates the illusion of real-time markdown rendering, with headers, bold text, and code fences formatting correctly as they stream in.

## Thread-Safe Coordination

Terminal rendering requires careful synchronization to prevent the spinner, live panel, and final output from colliding. nGPT solves this through a global `TERMINAL_RENDER_LOCK` in [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py).

The `create_spinner_handling_callback()` function wraps the original callback to manage the spinner lifecycle:

1. Before the API request starts, a spinner thread begins running
2. On the first content chunk, the wrapper stops the spinner and clears its line
3. Subsequent chunks flow directly to the UI callback

This ensures the user sees a loading indicator during network latency, which seamlessly transitions to the streaming markdown display once content arrives.

## Implementation Walkthrough

Here is the complete flow when a user runs `ngpt text "Explain quantum computing"`:

1. **Command Entry** – `text_mode()` in [`ngpt/cli/modes/text.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/cli/modes/text.py) parses arguments and determines whether to use plain text or stream-prettify mode.

2. **UI Setup** – For prettify mode, it calls `prettify_streaming_markdown()` to create the `Live` panel, update callback, and spinner factory.

3. **Spinner Start** – A daemon thread starts the spinner via `setup_spinner()`, displaying a loading indicator while waiting for the API.

4. **API Request** – `NGPTClient.chat()` sends the HTTP request with `stream=True` and the wrapped callback.

5. **SSE Processing** – As the API streams SSE events, each chunk is parsed and the accumulated text is passed to the spinner-handling wrapper.

6. **First Chunk Handling** – The wrapper stops the spinner, clears its line, and calls `update_content()` to start the `Live` display.

7. **Continuous Updates** – Each subsequent chunk updates the Markdown panel via `live.refresh()`, rendering headers, lists, and code blocks in real-time.

8. **Completion** – When the `[DONE]` marker arrives, `update_content(..., complete=True)` calls `live.stop()`, finalizing the display and releasing the terminal.

## Code Examples

### Minimal Streaming Client (No UI)

```python
from ngpt.api.client import NGPTClient

def my_callback(accumulated):
    # Called after each chunk arrives

    print(accumulated, end='', flush=True)

client = NGPTClient(api_key="sk-...")
client.chat(
    prompt="Explain quantum tunnelling.",
    stream=True,
    stream_callback=my_callback,
)

```

This example demonstrates the raw streaming mechanism where the client handles SSE parsing and delivers accumulated text to your callback.

### Real-Time Markdown UI (Full Implementation)

```python
from ngpt.ui.renderers import prettify_streaming_markdown, create_spinner_handling_callback
from ngpt.api.client import NGPTClient
import threading

# Build the live markdown panel

live, update_md, spinner_factory = prettify_streaming_markdown()

# Start spinner before API call

stop_event = threading.Event()
stop_spinner = spinner_factory(stop_event, color="cyan")

# Wrap callback to handle spinner cleanup

wrapped_cb = create_spinner_handling_callback(
    original_callback=update_md,
    stop_spinner_func=stop_spinner,
    first_content_received_ref=[False],
)

# Execute streaming request

client = NGPTClient()
client.chat(
    prompt="Write a short poem about sunrise in markdown.",
    stream=True,
    stream_callback=wrapped_cb,
)

# Live panel stops automatically when stream completes

```

Running this snippet produces a Rich panel that updates line-by-line, interpreting Markdown syntax as it streams in.

## Key Files

- **[`ngpt/api/client.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/api/client.py)** – Implements HTTP requests, SSE parsing, and the `stream_callback` forwarding mechanism.
- **[`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py)** – Contains `prettify_streaming_markdown()`, `create_spinner_handling_callback()`, and the `TERMINAL_RENDER_LOCK` for thread-safe terminal access.
- **[`ngpt/cli/modes/text.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/cli/modes/text.py)** – Orchestrates the text mode command, deciding between plain output and stream-prettify mode.
- **[`ngpt/ui/tui.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/tui.py)** – Provides spinner utilities and terminal helpers used during the streaming flow.
- **[`ngpt/ui/colors.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/colors.py)** – Defines the color scheme used for spinners and panel borders.

## Summary

- nGPT uses **Server-Sent Events (SSE)** to stream API responses chunk-by-chunk through `NGPTClient.chat()`.
- The **Rich library's `Live` display** renders Markdown in real-time via `prettify_streaming_markdown()` in [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py).
- A **global terminal lock** and spinner-handling wrapper ensure the loading indicator, live panel, and final output never conflict.
- The architecture cleanly separates network logic ([`ngpt/api/client.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/api/client.py)) from presentation logic ([`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py)), enabling both minimal callbacks and full interactive UIs.

## Frequently Asked Questions

### How does nGPT handle the transition from loading spinner to streaming text?

nGPT uses `create_spinner_handling_callback()` in [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py) to wrap the markdown update function. This wrapper tracks whether content has arrived via a `first_content_received_ref` list. On the first chunk, it calls the `stop_spinner_func` to clear the loading indicator, then forwards subsequent chunks directly to the live markdown panel.

### What prevents the spinner and live markdown panel from writing over each other?

A global `TERMINAL_RENDER_LOCK` defined in [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py) ensures thread-safe access to the terminal. The spinner runs in a daemon thread while the main thread handles the HTTP stream and Rich's `Live` display. The lock guarantees that only one component writes to stdout at any moment, preventing visual corruption.

### Can I use nGPT's streaming mechanism without the markdown rendering?

Yes. The `NGPTClient.chat()` method in [`ngpt/api/client.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/api/client.py) accepts any callable as `stream_callback`. You can provide a simple function that prints raw text or processes it programmatically without importing [`ngpt/ui/renderers.py`](https://github.com/nazdridoy/ngpt/blob/main/ngpt/ui/renderers.py) or using Rich's Live display.

### Why does nGPT truncate content to the visible height during streaming?

The `update_content()` function in `prettify_streaming_markdown()` truncates accumulated text to the terminal's visible height to prevent the Rich Live panel from growing indefinitely. This ensures the display remains performant and readable during long streaming sessions, showing only the most recent content that fits the screen.