# How the Agent Tool-Calling Architecture Works in Screenshot-to-Code

> Explore the agent tool-calling architecture in screenshot-to-code. Understand how an AgentEngine orchestrates LLM streaming, detects, and executes tool calls for file state mutation and model feedback.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**The Screenshot-to-Code agent uses a four-layer architecture where an `AgentEngine` orchestrates LLM streaming through vendor-specific `ProviderSession` implementations, detects tool calls, and executes them via an `AgentToolRuntime` that mutates file state and returns structured results back to the model context.**

The **agent tool-calling architecture** in the [`abi/screenshot-to-code`](https://github.com/abi/screenshot-to-code) repository enables large language models (LLMs) from OpenAI, Anthropic, or Gemini to autonomously generate, edit, and enhance HTML code by invoking specialized tools. Understanding this architecture reveals how the backend transforms a static screenshot into interactive code through an iterative conversation loop that persists state across multiple turns.

## The Four-Layer Architecture

The implementation separates concerns into four distinct layers, each with a specific responsibility in the tool-calling lifecycle:

| Layer | Responsibility | Key Implementation |
|------|----------------|--------------------|
| **Agent Engine** | Orchestrates the turn-based loop, streams deltas, and merges tool results back into context | [`backend/agent/engine.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/engine.py) |
| **Provider Session** | Wraps vendor-specific LLM APIs and normalizes streaming events into generic `StreamEvent` objects | [`backend/agent/providers/openai.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/openai.py), [`anthropic.py`](https://github.com/abi/screenshot-to-code/blob/main/anthropic.py), [`gemini.py`](https://github.com/abi/screenshot-to-code/blob/main/gemini.py) |
| **Tool Runtime** | Maintains mutable file state (`AgentFileState`) and implements concrete tool logic | [`backend/agent/tools/runtime.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/runtime.py) |
| **Canonical Tool Definitions** | Declares JSON schemas for available tools and serializes them to vendor-specific formats | [`backend/agent/tools/definitions.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/definitions.py) |

## Agent Engine: Orchestrating the Main Loop

The `AgentEngine` class in [`backend/agent/engine.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/engine.py) serves as the central orchestrator. Its `run()` method (lines 28-49) initializes the conversation and enters the iterative processing loop.

```python

# Simplified entry point (engine.py)

engine = AgentEngine(send_message, variant_index, file_state, …)
await engine.run(model, prompt_messages)

```

The engine executes the following sequence in `_run_with_session()` (lines 58-84):

1. **Streams the turn** by calling `session.stream_turn(on_event)`, which yields `assistant_delta`, `thinking_delta`, and `tool_call_delta` events
2. **Handles streamed tool deltas** via `_handle_streamed_tool_delta()` to send live previews to the UI
3. **Executes completed tool calls** through `AgentToolRuntime.execute()` after the turn ends
4. **Appends results** back to the provider session using `session.append_tool_results()` so the LLM observes the outcomes
5. **Terminates** when a turn returns no tool calls, triggering `_finalize_response()` to emit the final HTML

The engine maintains conversation continuity by seeding file state from existing assistant messages at startup (`seed_file_state_from_messages` in [`backend/agent/state.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/state.py)).

## Provider Session: Vendor-Agnostic LLM Streaming

Each LLM vendor implements the `ProviderSession` protocol defined in [`backend/agent/providers/base.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/base.py). This abstraction allows the engine to interact with OpenAI, Anthropic, or Gemini through a unified interface.

For OpenAI, the `stream_turn()` method in [`backend/agent/providers/openai.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/openai.py) (lines 99-106) constructs the request:

```python
params = {
    "model": get_openai_api_name(self._model),
    "input": self._input_items,
    "tools": self._tools,
    "tool_choice": "auto",
    "stream": True,
    "max_output_tokens": 50000,
}
stream = await self._client.responses.create(**params)
async for event in stream:
    await parse_event(event, state, on_event)
return _build_provider_turn(state)

```

The `parse_event()` function (lines 31-332) translates vendor-specific events into generic `StreamEvent` types:
- `assistant_delta` → Streams generated text to the UI
- `thinking_delta` → Displays model reasoning steps
- `tool_call_delta` → Aggregates partial JSON arguments for live previews

When the turn completes, `_build_provider_turn()` extracts fully-formed `ToolCall` objects, parses their JSON arguments, and returns a `ProviderTurn` containing both the assistant's text and any pending tool calls.

Anthropic and Gemini providers follow identical patterns in [`backend/agent/providers/anthropic.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/anthropic.py) and [`backend/agent/providers/gemini.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/gemini.py), handling their respective streaming formats while conforming to the same protocol.

## Tool Runtime: Executing Concrete Actions

The `AgentToolRuntime` class in [`backend/agent/tools/runtime.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/runtime.py) (lines 30-52) contains the imperative logic for each tool. The `execute()` method dispatches based on `tool_call.name` to specific handlers:

| Tool | Handler | Function |
|------|---------|----------|
| `create_file` | `_create_file()` | Writes new HTML to `AgentFileState` and returns the full content |
| `edit_file` | `_edit_file()` | Performs exact string replacements via `_apply_single_edit()` |
| `generate_images` | `_generate_images()` | Async generation using Replicate (Flux) or OpenAI DALL-E 3 |
| `remove_background` | `_remove_background()` | Batch processing (20 images per call) via Replicate |
| `retrieve_option` | `_retrieve_option()` | Fetches stored HTML for variant selection UI |

All handlers return a `ToolExecutionResult` dataclass (defined in [`backend/agent/tools/types.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/types.py)) containing:
- `ok`: Boolean success flag
- `result`: Full payload returned to the LLM
- `summary`: UI-friendly description
- `updated_content`: New HTML code streamed to the frontend via `setCode`

## Canonical Tool Definitions: Schema Management

Tool availability is declared in [`backend/agent/tools/definitions.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/definitions.py) through the `canonical_tool_definitions()` factory (lines 4-57). This function returns a list of `CanonicalToolDefinition` objects describing the JSON schema for each tool.

The provider factory in [`backend/agent/providers/factory.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/factory.py) serializes these definitions to vendor-specific formats:

```python
canonical_tools = canonical_tool_definitions(image_generation_enabled=should_generate_images)
if model in OPENAI_MODELS:
    return OpenAIProviderSession(..., tools=serialize_openai_tools(canonical_tools))

```

For OpenAI, `serialize_openai_tools()` (lines 105-120) enforces strict schema compliance by adding `"strict": True` and requiring all object properties, ensuring the LLM generates valid JSON arguments. Anthropic and Gemini serializers apply analogous transformations for their respective formats.

## End-to-End Execution Flow

The complete execution path from screenshot upload to generated code proceeds as follows:

1. The user uploads a screenshot, triggering `AgentEngine.run()` via the `/generate_code` endpoint
2. The factory instantiates the appropriate `ProviderSession` (OpenAI/Anthropic/Gemini)
3. `stream_turn()` begins emitting events: assistant tokens, thinking traces, and partial tool arguments
4. The engine streams these to the UI as they arrive, showing live previews when tools are invoked
5. Upon turn completion, the engine executes all pending `ToolCall` objects through the runtime
6. Tool results (HTML updates, image URLs) are sent to the UI and appended to the LLM context
7. The loop repeats until the LLM returns a turn without tool calls, at which point the final HTML is returned

## Adding a New Tool: Implementation Guide

To extend the **agent tool-calling architecture** with custom functionality, modify two files:

First, declare the schema in [`backend/agent/tools/definitions.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/definitions.py):

```python
def _my_tool_schema() -> Dict[str, Any]:
    return {
        "type": "object",
        "properties": {
            "message": {"type": "string", "description": "Content to process"}
        },
        "required": ["message"],
    }

# Add to canonical_tool_definitions()

tools.append(
    CanonicalToolDefinition(
        name="my_tool",
        description="Process a message and return metadata",
        parameters=_my_tool_schema(),
    )
)

```

Second, implement the logic in [`backend/agent/tools/runtime.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/runtime.py):

```python
def execute(self, tool_call: ToolCall) -> ToolExecutionResult:
    if tool_call.name == "my_tool":
        return self._my_tool(tool_call.arguments)
    # ... existing tools

def _my_tool(self, args: Dict[str, Any]) -> ToolExecutionResult:
    message = ensure_str(args.get("message"))
    return ToolExecutionResult(
        ok=True,
        result={"processed": message.upper()},
        summary={"message": f"Processed: {message[:50]}"},
    )

```

The factory automatically exposes the new tool to all providers, and the engine handles streaming `toolStart` and `toolResult` events without additional wiring.

## Summary

- The **agent tool-calling architecture** separates orchestration, vendor abstraction, execution, and schema definition into four distinct layers
- `AgentEngine` in [`backend/agent/engine.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/engine.py) manages the conversation loop, streaming deltas and merging tool results back into context
- `ProviderSession` implementations in `backend/agent/providers/` normalize OpenAI, Anthropic, and Gemini streaming formats into generic events
- `AgentToolRuntime` in [`backend/agent/tools/runtime.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/runtime.py) executes five canonical tools: `create_file`, `edit_file`, `generate_images`, `remove_background`, and `retrieve_option`
- Tool schemas are defined centrally in [`backend/agent/tools/definitions.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/tools/definitions.py) and serialized to vendor-specific formats by the factory
- The loop terminates when the LLM returns a turn without tool calls, yielding the final generated HTML

## Frequently Asked Questions

### How does the agent handle streaming tool calls in real-time?

The `AgentEngine` processes `tool_call_delta` events as they arrive from the `ProviderSession`, immediately forwarding them to `_handle_streamed_tool_delta()`. This sends `toolStart` messages and argument previews to the UI before execution completes, allowing users to see which tools the LLM intends to invoke while generation is still in progress.

### What happens if a tool execution fails?

The `AgentToolRuntime.execute()` method catches exceptions and returns a `ToolExecutionResult` with `ok=False` and an error description in the `result` field. The engine appends this failure message to the LLM context via `session.append_tool_results()`, allowing the model to observe the error and potentially retry with corrected arguments in the next turn.

### Can the architecture support tools that modify multiple files?

Currently, the `AgentFileState` class in [`backend/agent/state.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/state.py) maintains a single mutable HTML string per conversation. While the existing tools (`create_file`, `edit_file`) operate on this singleton state, the `ToolExecutionResult` structure supports returning `updated_content` that the engine streams via `setCode`. Extending the runtime to handle multiple file buffers would require updating `AgentFileState` to manage a dictionary of paths-to-contents, but the existing dispatch mechanism in [`runtime.py`](https://github.com/abi/screenshot-to-code/blob/main/runtime.py) can accommodate such extensions.

### Why does OpenAI use `"strict": True` in tool definitions?

The `serialize_openai_tools()` function in [`backend/agent/providers/openai.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/openai.py) adds `"strict": True` to enforce that the LLM must generate JSON arguments that exactly match the provided schema, with all required fields present. This reduces parsing errors and eliminates the need for the runtime to handle missing keys or type mismatches, ensuring reliable execution of tools like `edit_file` which require precise string arguments.