How the Streaming Token Buffer Handles Markdown and Code Blocks in ML Intern

The streaming token buffer in ML Intern accumulates raw tokens paragraph-by-paragraph, ensuring fenced code blocks render atomically only after the closing backticks arrive.

ML Intern, the open-source agent framework from Hugging Face, streams LLM output token-by-token to maintain a responsive terminal interface. To prevent fragmented code fences and broken Markdown structures during real-time display, the library implements a specialized streaming token buffer called _StreamBuffer. This stateful helper defined in agent/main.py manages the delicate balance between low-latency output and structural integrity.

Architecture of the Streaming Token Buffer

The _StreamBuffer class operates through a three-stage pipeline that processes incoming text incrementally. It balances immediate user feedback with the technical requirement to keep Markdown elements—especially fenced code blocks—intact until they are fully received.

Accumulating Raw Chunks

Every token arriving from the LLM appends to an internal string buffer via the add_chunk method. This method simply concatenates new content to self._buffer, allowing the system to handle arbitrarily fragmented streaming responses without loss.

def add_chunk(self, text: str):
    self._buffer += text          # ← adds streamed tokens

(source: [agent/main.py lines 22-23](https://github.com/huggingface/ml-intern/blob/main/agent/main.py#L22-L23))

Detecting Complete Markdown Blocks

Before emitting text, the buffer must determine whether a logical block is complete. A block is defined as text up to a paragraph break ("\n\n"), but with a critical safety constraint: fenced code sections must be closed.

The _pop_block method checks if the count of backtick triplets (```) is even. An odd count indicates an open code fence, so the method returns None and defers rendering until the closing delimiter arrives. Only when the fence is closed and a double newline appears does the buffer slice out the completed block for display.

def _pop_block(self) -> str | None:
    if self._buffer.count("```") % 2 == 1:
        return None                # inside an open code fence → wait

    idx = self._buffer.find("\n\n")
    if idx == -1:
        return None                # no paragraph break yet

    block = self._buffer[:idx]
    self._buffer = self._buffer[idx + 2:]
    return block

(source: [agent/main.py lines 25-33](https://github.com/huggingface/ml-intern/blob/main/agent/main.py#L25-L33))

Flushing Ready Blocks Progressively

Once _pop_block identifies a complete block, flush_ready handles the actual rendering. This async method repeatedly extracts ready blocks and passes them to print_markdown (from agent/utils/terminal_display.py) for styled terminal output with a typewriter effect. Empty blocks are skipped to avoid unnecessary renders.

When the stream terminates, the finish method ensures any remaining content—including text that might not end with a double newline—is flushed to the terminal.

async def flush_ready(...):
    while True:
        block = self._pop_block()
        if block is None: break
        if block.strip():
            await print_markdown(block, cancel_event=..., instant=...)

(source: [agent/main.py lines 40-48](https://github.com/huggingface/ml-intern/blob/main/agent/main.py#L40-L48))

Integration with the Event Loop

The streaming token buffer integrates directly into ML Intern's background event_listener inside agent/main.py. For each assistant_chunk event, incoming content appends to the buffer and immediately triggers flush_ready, while assistant_stream_end forces a final finish call to guarantee no text remains stranded in the buffer.

elif event.event_type == "assistant_chunk":
    content = event.data.get("content", "")
    if content:
        stream_buf.add_chunk(content)
        shimmer.stop()
        await stream_buf.flush_ready(cancel_event=_cancel_event())
elif event.event_type == "assistant_stream_end":
    shimmer.stop()
    await stream_buf.finish(cancel_event=_cancel_event())

(source: [agent/main.py lines 300-311](https://github.com/huggingface/ml-intern/blob/main/agent/main.py#L300-L311))

Practical Usage Example

The following example demonstrates how the buffer behaves when simulating segmented LLM output. Notice that the code block only appears after the closing fence arrives, while regular paragraphs render immediately upon detecting the double newline.

from agent.main import _StreamBuffer
from agent.utils.terminal_display import print_markdown
import asyncio

async def demo():
    console = ...                     # obtain a Rich console via get_console()

    buf = _StreamBuffer(console)

    # Simulate streamed tokens arriving in pieces

    buf.add_chunk("Here is some intro text.\n\n")
    await buf.flush_ready()           # prints the paragraph immediately

    # Start a fenced code block – nothing is shown yet

    buf.add_chunk("```python\nprint('Hello")
    await buf.flush_ready()           # no output because fence is open

    # Close the fence – now the whole block appears atomically

    buf.add_chunk(" world')\n```\n\n")
    await buf.flush_ready()           # prints the complete code block

    # End of stream – render any tail that didn't end with a double newline

    await buf.finish()

asyncio.run(demo())

Summary

• Paragraph-wise rendering – Users see each paragraph as soon as it is complete, keeping the interface responsive during long generations.
• Atomic code-block display – Fenced code emits only after the closing ``` appears, preventing stray "half-code" fragments from cluttering the terminal.
• Graceful termination – The finish method ensures any leftover text is printed once the stream ends, regardless of whether it terminated with a paragraph break.
• Event-driven integration – The buffer plugs directly into ML Intern's event system, processing assistant_chunk events incrementally and finalizing on assistant_stream_end.

Frequently Asked Questions

Where is the streaming token buffer implemented in ML Intern?

The streaming token buffer is implemented as the _StreamBuffer class in agent/main.py according to the Hugging Face ML Intern source code. This private helper is instantiated once per interactive session inside the background event_listener.

How does the buffer prevent broken code blocks from appearing?

The buffer counts occurrences of the backtick triplet () using `self._buffer.count("") % 2 == 1. An odd count indicates an open code fence, causing _pop_blockto returnNone` and defer rendering until the closing fence arrives, ensuring the entire block displays atomically.

What triggers the final render of leftover text?

The finish method is invoked when the assistant_stream_end event fires. This method flushes any remaining content in the internal buffer that may not have ended with a double newline, ensuring the user sees the complete LLM output.

Why does the buffer wait for paragraph breaks before rendering?

Waiting for "\n\n" paragraph breaks enables paragraph-wise streaming that maintains Markdown structural coherence. This approach allows the terminal to display content incrementally without breaking headers, lists, or code fences across multiple premature flushes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}