How the Agent Tool-Calling Architecture Works in Screenshot-to-Code

The Screenshot-to-Code agent uses a four-layer architecture where an AgentEngine orchestrates LLM streaming through vendor-specific ProviderSession implementations, detects tool calls, and executes them via an AgentToolRuntime that mutates file state and returns structured results back to the model context.

The agent tool-calling architecture in the abi/screenshot-to-code repository enables large language models (LLMs) from OpenAI, Anthropic, or Gemini to autonomously generate, edit, and enhance HTML code by invoking specialized tools. Understanding this architecture reveals how the backend transforms a static screenshot into interactive code through an iterative conversation loop that persists state across multiple turns.

The Four-Layer Architecture

The implementation separates concerns into four distinct layers, each with a specific responsibility in the tool-calling lifecycle:

Layer	Responsibility	Key Implementation
Agent Engine	Orchestrates the turn-based loop, streams deltas, and merges tool results back into context	backend/agent/engine.py
Provider Session	Wraps vendor-specific LLM APIs and normalizes streaming events into generic StreamEvent objects	backend/agent/providers/openai.py, anthropic.py, gemini.py
Tool Runtime	Maintains mutable file state (AgentFileState) and implements concrete tool logic	backend/agent/tools/runtime.py
Canonical Tool Definitions	Declares JSON schemas for available tools and serializes them to vendor-specific formats	backend/agent/tools/definitions.py

Agent Engine: Orchestrating the Main Loop

The AgentEngine class in backend/agent/engine.py serves as the central orchestrator. Its run() method (lines 28-49) initializes the conversation and enters the iterative processing loop.

# Simplified entry point (engine.py)

engine = AgentEngine(send_message, variant_index, file_state, …)
await engine.run(model, prompt_messages)

The engine executes the following sequence in _run_with_session() (lines 58-84):

1. Streams the turn by calling session.stream_turn(on_event), which yields assistant_delta, thinking_delta, and tool_call_delta events
2. Handles streamed tool deltas via _handle_streamed_tool_delta() to send live previews to the UI
3. Executes completed tool calls through AgentToolRuntime.execute() after the turn ends
4. Appends results back to the provider session using session.append_tool_results() so the LLM observes the outcomes
5. Terminates when a turn returns no tool calls, triggering _finalize_response() to emit the final HTML

The engine maintains conversation continuity by seeding file state from existing assistant messages at startup (seed_file_state_from_messages in backend/agent/state.py).

Provider Session: Vendor-Agnostic LLM Streaming

Each LLM vendor implements the ProviderSession protocol defined in backend/agent/providers/base.py. This abstraction allows the engine to interact with OpenAI, Anthropic, or Gemini through a unified interface.

For OpenAI, the stream_turn() method in backend/agent/providers/openai.py (lines 99-106) constructs the request:

params = {
    "model": get_openai_api_name(self._model),
    "input": self._input_items,
    "tools": self._tools,
    "tool_choice": "auto",
    "stream": True,
    "max_output_tokens": 50000,
}
stream = await self._client.responses.create(**params)
async for event in stream:
    await parse_event(event, state, on_event)
return _build_provider_turn(state)

The parse_event() function (lines 31-332) translates vendor-specific events into generic StreamEvent types:

• assistant_delta → Streams generated text to the UI
• thinking_delta → Displays model reasoning steps
• tool_call_delta → Aggregates partial JSON arguments for live previews

When the turn completes, _build_provider_turn() extracts fully-formed ToolCall objects, parses their JSON arguments, and returns a ProviderTurn containing both the assistant's text and any pending tool calls.

Anthropic and Gemini providers follow identical patterns in backend/agent/providers/anthropic.py and backend/agent/providers/gemini.py, handling their respective streaming formats while conforming to the same protocol.

Tool Runtime: Executing Concrete Actions

The AgentToolRuntime class in backend/agent/tools/runtime.py (lines 30-52) contains the imperative logic for each tool. The execute() method dispatches based on tool_call.name to specific handlers:

Tool	Handler	Function
create_file	_create_file()	Writes new HTML to AgentFileState and returns the full content
edit_file	_edit_file()	Performs exact string replacements via _apply_single_edit()
generate_images	_generate_images()	Async generation using Replicate (Flux) or OpenAI DALL-E 3
remove_background	_remove_background()	Batch processing (20 images per call) via Replicate
retrieve_option	_retrieve_option()	Fetches stored HTML for variant selection UI

All handlers return a ToolExecutionResult dataclass (defined in backend/agent/tools/types.py) containing:

• ok: Boolean success flag
• result: Full payload returned to the LLM
• summary: UI-friendly description
• updated_content: New HTML code streamed to the frontend via setCode

Canonical Tool Definitions: Schema Management

Tool availability is declared in backend/agent/tools/definitions.py through the canonical_tool_definitions() factory (lines 4-57). This function returns a list of CanonicalToolDefinition objects describing the JSON schema for each tool.

The provider factory in backend/agent/providers/factory.py serializes these definitions to vendor-specific formats:

canonical_tools = canonical_tool_definitions(image_generation_enabled=should_generate_images)
if model in OPENAI_MODELS:
    return OpenAIProviderSession(..., tools=serialize_openai_tools(canonical_tools))

For OpenAI, serialize_openai_tools() (lines 105-120) enforces strict schema compliance by adding "strict": True and requiring all object properties, ensuring the LLM generates valid JSON arguments. Anthropic and Gemini serializers apply analogous transformations for their respective formats.

End-to-End Execution Flow

The complete execution path from screenshot upload to generated code proceeds as follows:

1. The user uploads a screenshot, triggering AgentEngine.run() via the /generate_code endpoint
2. The factory instantiates the appropriate ProviderSession (OpenAI/Anthropic/Gemini)
3. stream_turn() begins emitting events: assistant tokens, thinking traces, and partial tool arguments
4. The engine streams these to the UI as they arrive, showing live previews when tools are invoked
5. Upon turn completion, the engine executes all pending ToolCall objects through the runtime
6. Tool results (HTML updates, image URLs) are sent to the UI and appended to the LLM context
7. The loop repeats until the LLM returns a turn without tool calls, at which point the final HTML is returned

Adding a New Tool: Implementation Guide

To extend the agent tool-calling architecture with custom functionality, modify two files:

First, declare the schema in backend/agent/tools/definitions.py:

def _my_tool_schema() -> Dict[str, Any]:
    return {
        "type": "object",
        "properties": {
            "message": {"type": "string", "description": "Content to process"}
        },
        "required": ["message"],
    }

# Add to canonical_tool_definitions()

tools.append(
    CanonicalToolDefinition(
        name="my_tool",
        description="Process a message and return metadata",
        parameters=_my_tool_schema(),
    )
)

Second, implement the logic in backend/agent/tools/runtime.py:

def execute(self, tool_call: ToolCall) -> ToolExecutionResult:
    if tool_call.name == "my_tool":
        return self._my_tool(tool_call.arguments)
    # ... existing tools

def _my_tool(self, args: Dict[str, Any]) -> ToolExecutionResult:
    message = ensure_str(args.get("message"))
    return ToolExecutionResult(
        ok=True,
        result={"processed": message.upper()},
        summary={"message": f"Processed: {message[:50]}"},
    )

The factory automatically exposes the new tool to all providers, and the engine handles streaming toolStart and toolResult events without additional wiring.

Summary

• The agent tool-calling architecture separates orchestration, vendor abstraction, execution, and schema definition into four distinct layers
• AgentEngine in backend/agent/engine.py manages the conversation loop, streaming deltas and merging tool results back into context
• ProviderSession implementations in backend/agent/providers/ normalize OpenAI, Anthropic, and Gemini streaming formats into generic events
• AgentToolRuntime in backend/agent/tools/runtime.py executes five canonical tools: create_file, edit_file, generate_images, remove_background, and retrieve_option
• Tool schemas are defined centrally in backend/agent/tools/definitions.py and serialized to vendor-specific formats by the factory
• The loop terminates when the LLM returns a turn without tool calls, yielding the final generated HTML

Frequently Asked Questions

How does the agent handle streaming tool calls in real-time?

The AgentEngine processes tool_call_delta events as they arrive from the ProviderSession, immediately forwarding them to _handle_streamed_tool_delta(). This sends toolStart messages and argument previews to the UI before execution completes, allowing users to see which tools the LLM intends to invoke while generation is still in progress.

What happens if a tool execution fails?

The AgentToolRuntime.execute() method catches exceptions and returns a ToolExecutionResult with ok=False and an error description in the result field. The engine appends this failure message to the LLM context via session.append_tool_results(), allowing the model to observe the error and potentially retry with corrected arguments in the next turn.

Can the architecture support tools that modify multiple files?

Currently, the AgentFileState class in backend/agent/state.py maintains a single mutable HTML string per conversation. While the existing tools (create_file, edit_file) operate on this singleton state, the ToolExecutionResult structure supports returning updated_content that the engine streams via setCode. Extending the runtime to handle multiple file buffers would require updating AgentFileState to manage a dictionary of paths-to-contents, but the existing dispatch mechanism in runtime.py can accommodate such extensions.

Why does OpenAI use "strict": True in tool definitions?

The serialize_openai_tools() function in backend/agent/providers/openai.py adds "strict": True to enforce that the LLM must generate JSON arguments that exactly match the provided schema, with all required fields present. This reduces parsing errors and eliminates the need for the runtime to handle missing keys or type mismatches, ensuring reliable execution of tools like edit_file which require precise string arguments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}