# How the Browser Tool in Lemon AI Automates Web Interaction: Architecture and Code Walkthrough

> Explore Lemon AI's browser tool architecture and code. Learn how natural language prompts automate web interaction via Playwright and LangChain.

- Repository: [hexdocom/lemonai](https://github.com/hexdocom/lemonai)
- Tags: architecture
- Published: 2026-03-03

---

**The browser tool in Lemon AI forwards natural language prompts from a frontend runtime to a local FastAPI backend, which orchestrates a LangChain agent controlling a headless Playwright session to execute navigation, extraction, and screenshot tasks.**

The `hexdocom/lemonai` repository implements a sophisticated browser automation system that allows LLM agents to interact with live web pages without manual scripting. According to the lemonai source code, the architecture separates concerns across three distinct layers: a frontend tool definition, a runtime HTTP bridge, and a backend agent orchestration service powered by Playwright.

## Frontend Tool Definition

The browser tool surface is defined in [`src/tools/browser.js`](https://github.com/hexdocom/lemonai/blob/main/src/tools/browser.js), where the tool schema specifies the contract between the LLM and the automation engine.

```javascript
// src/tools/browser.js
const browser = {
  name: "browser",
  description: "Interact with the browser. Use it ONLY when you need to interact with a webpage.",
  params: {
    type: "object",
    properties: {
      question: {
        description: "What you want to do with a browser",
        type: "string"
      }
    },
    required: ["question"]
  },
  getActionDescription({ question }) {
    return question;
  }
};

module.exports = browser;

```

The **tool definition** exposes a single required parameter `question`, which contains the natural language instruction describing the desired browsing task. This schema allows the LLM to generate structured tool calls that capture user intent in plain text.

## Runtime Bridge to Backend

When an LLM invokes the browser tool, the frontend runtime serializes the request and transmits it via HTTP to the backend service. The `browser()` function in [`src/runtime/browser.js`](https://github.com/hexdocom/lemonai/blob/main/src/runtime/browser.js) handles this transmission.

```javascript
// src/runtime/browser.js
async function browser(action, uuid) {
  const host = 'localhost';
  const host_port = 9000;

  const request = {
    method: 'POST',
    url: `http://${host}:${host_port}/api/browser/task`,
    data: { prompt: action.params.question, llm_config: action.params.llm_config },
  };
  const response = await axios(request);
  const result_content = response.data.data.history.task;
  return {
    uuid,
    status: 'success',
    content: result_content,
    meta: {
      action_type: 'browser',
      json: {
        browser_history: response.data.data.history.browser_history,
        browser_history_screenshot: response.data.data.history.browser_history_screenshot
      }
    }
  };
}

```

The **runtime bridge** constructs a POST request to `http://localhost:9000/api/browser/task`, forwarding the user's `question` as the `prompt` field along with the selected LLM configuration. This decouples the frontend interface from the heavy browser automation logic running in a separate Python process.

## Backend API and Task Parsing

The FastAPI server receives the request at the `/api/browser/task` endpoint defined in [`browser_server/browser_use/server.py`](https://github.com/hexdocom/lemonai/blob/main/browser_server/browser_use/server.py).

```python

# browser_server/browser_use/server.py

@app.post("/api/browser/task")
async def browser_task(request: Request):
    start_time = datetime.datetime.now()
    data = await request.json()
    task = await parse_task_json(data)
    llm_config = task.llm_config
    history = await browser_agent_manager.run_task_only(
        task.prompt,
        model=llm_config["model_name"],
        api_key=llm_config["api_key"],
        base_url=llm_config["api_url"],
        conversation_id=task.conversation_id,
    )
    response = create_response(
        200,
        "Task Finished",
        {
            "time": datetime.datetime.now().isoformat(),
            "time_cost": (end_time - start_time).total_seconds(),
            "history": history
        },
    )
    return response

```

The **backend API** deserializes the incoming JSON into a `TaskRequest` object using `parse_task_json()` from the utilities module. It extracts the LLM credentials (`model_name`, `api_key`, `api_url`) and delegates execution to the `BrowserAgentManager`. This manager maintains persistent browser sessions across requests to optimize performance.

## Agent Orchestration with LangChain

The `BrowserAgentManager` constructs a LangChain agent capable of planning and executing browser actions. The agent factory resides in [`browser_server/browser_use/agent/agent.py`](https://github.com/hexdocom/lemonai/blob/main/browser_server/browser_use/agent/agent.py).

```python

# browser_server/browser_use/agent/agent.py

def get_agent(self, task: str, model: str, api_key: str, base_url, extend_prompt_id: int = -1,
              browser_session=None, conversation_id: str = None):
    llm = self._get_llm(model, api_key, base_url, conversation_id)
    extend_prompt = self.prompts_extend[extend_prompt_id]
    tool_calling_method = 'auto' if 'doubao' not in model else 'raw'
    return Agent(
        task=task,
        llm=llm,
        override_system_message=None,
        extend_system_message=extend_prompt,
        browser_session=browser_session,
        use_vision=False,
        tool_calling_method=tool_calling_method,
    )

```

The **agent initialization** binds the natural language `task` to a LangChain LLM instance, attaches an extended system prompt for behavioral guidance, and injects a shared **Playwright browser session**. The `tool_calling_method` adapts to different model providers, switching to `'raw'` for Doubao models while using `'auto'` for others.

## Task Execution and Response Formatting

The actual browsing session executes within [`browser_server/browser_use/service/browser_agent.py`](https://github.com/hexdocom/lemonai/blob/main/browser_server/browser_use/service/browser_agent.py), where the manager runs the agent and formats the results.

```python

# browser_server/browser_use/service/browser_agent.py

async def run_task_only(self, task: str, model: str, api_key: str,
                       base_url: str, conversation_id: Optional[str] = None) -> str:
    uid = str(uuid.uuid4())
    agent = browser_agent.get_agent(
        task=task, model=model, api_key=api_key,
        base_url=base_url, browser_session=self.browser_session,
        conversation_id=conversation_id
    )
    history = await agent.run(max_steps=config['agent']['max_steps'])
    result = self._format_history(history) or [self._get_null_response_result(model=model)]
    return {
        "uid": uid,
        "task": task,
        "status": "finished",
        "time": datetime.now().strftime("%Y%m%d%H%M%S"),
        "final_browser_result": self._get_final_result(history),
        "browser_history": result,
        "browser_history_screenshot": history.screenshots(),
    }

```

The **execution engine** runs the agent with a configurable step limit defined in [`config.yaml`](https://github.com/hexdocom/lemonai/blob/main/config.yaml), captures the structured interaction history, and extracts base64-encoded screenshots from the Playwright session. The response includes a unique task ID, timing metadata, the final extracted content, and visual evidence of the browsing session.

## Practical Usage Examples

### Invoking the Browser Tool via LLM Tool Call

When configuring an LLM with the browser tool, the interaction follows this pattern:

```json
{
  "messages": [
    {"role": "user", "content": "Find the latest weather forecast for Berlin and give me the temperature."}
  ],
  "tools": [
    {"type": "function", "function": {"name": "browser", "description": "Interact with the browser.", "parameters": {"type": "object", "properties": {"question": {"type": "string", "description": "What you want to do with a browser"}}, "required": ["question"]}}}
  ]
}

```

The LLM generates a tool call:

```json
{
  "name": "browser",
  "arguments": {"question": "Search for “Berlin weather today” and extract the temperature from the first result."}
}

```

The system returns a structured response containing the extracted text and screenshot history:

```json
{
  "status": "success",
  "content": "The current temperature in Berlin is 13 °C.",
  "meta": {
    "action_type": "browser",
    "json": {
      "browser_history": [...],
      "browser_history_screenshot": ["data:image/png;base64,..."]
    }
  }
}

```

### Direct API Access for Debugging

Developers can bypass the frontend and call the backend directly:

```bash
curl -X POST http://localhost:9000/api/browser/task \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "Search for “Python 3.12 release notes” and give me the first three bullet points.",
        "llm_config": {
          "model_name": "gpt-4o-mini",
          "api_key": "<YOUR_OPENAI_KEY>",
          "api_url": "https://api.openai.com/v1"
        }
      }'

```

## Summary

- **Three-tier architecture**: The browser tool separates frontend tool definition ([`src/tools/browser.js`](https://github.com/hexdocom/lemonai/blob/main/src/tools/browser.js)), runtime HTTP bridging ([`src/runtime/browser.js`](https://github.com/hexdocom/lemonai/blob/main/src/runtime/browser.js)), and backend agent execution (`browser_server/browser_use/`).
- **Natural language interface**: The tool accepts a single `question` parameter that the LangChain agent translates into concrete Playwright actions.
- **Persistent sessions**: The `BrowserAgentManager` maintains reusable Playwright browser sessions across API calls to reduce initialization overhead.
- **Comprehensive telemetry**: Every task returns extracted content, structured history, and base64-encoded screenshots for verification and debugging.
- **Model flexibility**: The agent factory supports multiple LLM providers through configurable `tool_calling_method` and custom base URLs.

## Frequently Asked Questions

### How does the frontend communicate with the browser automation backend?

The frontend runtime in [`src/runtime/browser.js`](https://github.com/hexdocom/lemonai/blob/main/src/runtime/browser.js) sends an HTTP POST request to `http://localhost:9000/api/browser/task` with the user's prompt and LLM configuration. This REST API acts as the sole communication channel between the JavaScript frontend and the Python-based browser service.

### What browser engine powers the automation?

As implemented in `hexdocom/lemonai`, the backend uses **Playwright** to control a headless browser session. The `BrowserAgentManager` initializes and maintains this session, passing it to the LangChain agent to perform navigation, clicking, and extraction operations.

### How are screenshots captured during automated browsing?

The agent execution method `run_task_only()` in [`browser_server/browser_use/service/browser_agent.py`](https://github.com/hexdocom/lemonai/blob/main/browser_server/browser_use/service/browser_agent.py) calls `history.screenshots()` on the agent's result object, which returns an array of base64-encoded PNG images representing the browser state at key interaction points.

### Can I configure the maximum number of browsing steps?

Yes. The agent respects the `max_steps` parameter defined in [`browser_server/browser_use/config/config.yaml`](https://github.com/hexdocom/lemonai/blob/main/browser_server/browser_use/config/config.yaml). This configuration limits how many actions the LangChain agent will attempt before terminating the task, preventing infinite loops on complex or ambiguous browsing instructions.