How Agent Zero Handles User Input and Responses: A Deep Dive into the Message Pipeline

Agent Zero normalizes all inbound requests into a UserMessage object, stores them in conversation history, streams LLM responses through extensible callbacks, and executes any embedded tool calls before returning the final output.

Agent Zero is an open-source agentic framework that orchestrates large language model interactions with tool execution capabilities. Understanding how Agent Zero handles user input and responses reveals a carefully architected pipeline that ensures consistent message normalization, extensible stream processing, and secure command execution. The framework transforms every request into a canonical data structure and processes it through four distinct stages before returning a rendered response.

Reception and Normalization with UserMessage

All external input—whether from HTTP POST requests, WebSocket connections, or internal API calls—converges on a single abstraction. In python/api/message.py, the endpoint constructs a UserMessage instance and passes it to the agent's communication loop.

# From python/api/message.py (line 71)

return context.communicate(UserMessage(message, attachment_paths)), context

The communicate method on AgentContext receives this object and initiates the processing pipeline. This design ensures that file attachments, system messages, and plain text are handled uniformly regardless of transport protocol.

The UserMessage Data Class

The canonical representation of user input is defined in agent.py as a lightweight dataclass:

# From agent.py (lines 319-324)

@dataclass
class UserMessage:
    message: str
    attachments: list[str] = field(default_factory=list[str])
    system_message: list[str] = field(default_factory=list[str])

This structure accommodates optional file paths alongside the primary message text, enabling multimodal workflows where users attach documents or images for the agent to analyze.

History Management and Prompt Construction

Once normalized, the UserMessage is persisted to the agent's conversation history via hist_add_user_message. During the next iteration of Agent.monologue, the framework assembles a prompt composed of system instructions, recent conversation turns, and any context injected by registered extensions.

This history-aware architecture allows the LLM to reference earlier parts of the conversation when formulating responses. The assembled prompt is then passed to call_chat_model to initiate the inference request.

LLM Invocation and Streaming Response Processing

Agent Zero implements a dual-channel streaming architecture where the LLM emits both reasoning traces and the final user-visible response simultaneously. The monologue method manages this through an async callback system:

# From agent.py (lines 35-43)

async def stream_callback(chunk: str, full: str):
    await self.handle_intervention()
    if chunk == full:
        printer.print("Response: ")
    stream_data = {"chunk": chunk, "full": full}
    await self.call_extensions("response_stream_chunk", stream_data=stream_data)
    if stream_data.get("chunk"):
        printer.stream(stream_data["chunk"])
    await self.handle_response_stream(stream_data["full"])

The callback invokes call_extensions("response_stream_chunk", ...) at every chunk, allowing middleware to filter, redact, or enrich content before it reaches the UI. This extensibility point is critical for implementing security policies, logging, or real-time formatting without modifying core engine code.

Tool Extraction and Execution

When the LLM finishes generating, Agent Zero inspects the response text for embedded JSON tool requests. The process_tools method in agent.py handles this extraction and dispatch:

# From agent.py (lines 55-72)

tool_request = extract_tools.json_parse_dirty(msg)
if tool_request is not None:
    raw_tool_name = tool_request.get("tool_name", tool_request.get("tool",""))
    tool_args = tool_request.get("tool_args", tool_request.get("args", {}))
    # ... tool resolution logic ...

    tool = self.get_tool(name=tool_name, method=tool_method, args=tool_args, ...)
    if tool:
        response = await tool.execute(**tool_args)

After execution, results are added back to the conversation history via hist_add_tool_result, allowing the LLM to incorporate tool outputs into subsequent reasoning or final answers.

The Terminal Input Tool Example

When the LLM requests user terminal access via the "input" tool, the request flows through python/tools/input.py:

# From python/tools/input.py (lines 8-20)

async def execute(self, keyboard="", **kwargs):
    keyboard = keyboard.rstrip()
    session = int(self.args.get("session", 0))
    args = {"runtime": "terminal", "code": keyboard, "session": session, "allow_running": True}
    cet = CodeExecution(self.agent, "code_execution_tool", "", args, self.message, self.loop_data)
    return await cet.execute(**args)

This tool forwards the keyboard input to the CodeExecution class, which runs commands in a sandboxed terminal session. The output is captured and returned to the agent, creating a secure loop where the LLM can execute shell commands without exposing the host system to uncontrolled access.

Summary

• Unified Input Abstraction: All requests become UserMessage objects in agent.py, standardizing handling across HTTP, WebSocket, and internal APIs.
• Persistent History: The hist_add_user_message and hist_add_tool_result functions maintain conversation context for multi-turn reasoning.
• Extensible Streaming: The stream_callback in Agent.monologue enables real-time modifications through the response_stream_chunk extension point.
• Secure Tool Execution: process_tools parses JSON payloads and dispatches to sandboxed tools like CodeExecution, with results reintegrated into the conversation history.

Frequently Asked Questions

What is the UserMessage class in Agent Zero?

The UserMessage class is a dataclass defined in agent.py (lines 319-324) that serves as the canonical container for all user input. It stores the message string, a list of attachment paths, and optional system messages, ensuring consistent data structures throughout the pipeline regardless of whether input arrives via HTTP API or WebSocket.

How does Agent Zero stream LLM responses to the user interface?

Agent Zero uses an async stream_callback function within the monologue method that receives text chunks as the LLM generates them. Each chunk is wrapped in a stream_data dictionary and passed through the response_stream_chunk extension hook, allowing middleware to modify content before printer.stream() renders it to the UI.

What happens when the LLM requests a tool execution?

When the LLM emits a JSON tool request, the process_tools method (lines 55-72 in agent.py) parses the payload using extract_tools.json_parse_dirty, resolves the appropriate tool instance via get_tool, and invokes await tool.execute(). The result is stored in history via hist_add_tool_result, and the conversation continues with the tool output available as context.

How does Agent Zero handle file attachments with user messages?

File paths are passed as the attachments parameter when constructing a UserMessage in python/api/message.py. These paths are stored alongside the text content in the dataclass and can be accessed by tools or extensions during prompt construction, enabling the agent to read and analyze uploaded documents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}