How the Browser Tool in Lemon AI Automates Web Interaction: Architecture and Code Walkthrough

The browser tool in Lemon AI forwards natural language prompts from a frontend runtime to a local FastAPI backend, which orchestrates a LangChain agent controlling a headless Playwright session to execute navigation, extraction, and screenshot tasks.

The hexdocom/lemonai repository implements a sophisticated browser automation system that allows LLM agents to interact with live web pages without manual scripting. According to the lemonai source code, the architecture separates concerns across three distinct layers: a frontend tool definition, a runtime HTTP bridge, and a backend agent orchestration service powered by Playwright.

Frontend Tool Definition

The browser tool surface is defined in src/tools/browser.js, where the tool schema specifies the contract between the LLM and the automation engine.

// src/tools/browser.js
const browser = {
  name: "browser",
  description: "Interact with the browser. Use it ONLY when you need to interact with a webpage.",
  params: {
    type: "object",
    properties: {
      question: {
        description: "What you want to do with a browser",
        type: "string"
      }
    },
    required: ["question"]
  },
  getActionDescription({ question }) {
    return question;
  }
};

module.exports = browser;

The tool definition exposes a single required parameter question, which contains the natural language instruction describing the desired browsing task. This schema allows the LLM to generate structured tool calls that capture user intent in plain text.

Runtime Bridge to Backend

When an LLM invokes the browser tool, the frontend runtime serializes the request and transmits it via HTTP to the backend service. The browser() function in src/runtime/browser.js handles this transmission.

// src/runtime/browser.js
async function browser(action, uuid) {
  const host = 'localhost';
  const host_port = 9000;

  const request = {
    method: 'POST',
    url: `http://${host}:${host_port}/api/browser/task`,
    data: { prompt: action.params.question, llm_config: action.params.llm_config },
  };
  const response = await axios(request);
  const result_content = response.data.data.history.task;
  return {
    uuid,
    status: 'success',
    content: result_content,
    meta: {
      action_type: 'browser',
      json: {
        browser_history: response.data.data.history.browser_history,
        browser_history_screenshot: response.data.data.history.browser_history_screenshot
      }
    }
  };
}

The runtime bridge constructs a POST request to http://localhost:9000/api/browser/task, forwarding the user's question as the prompt field along with the selected LLM configuration. This decouples the frontend interface from the heavy browser automation logic running in a separate Python process.

Backend API and Task Parsing

The FastAPI server receives the request at the /api/browser/task endpoint defined in browser_server/browser_use/server.py.

# browser_server/browser_use/server.py

@app.post("/api/browser/task")
async def browser_task(request: Request):
    start_time = datetime.datetime.now()
    data = await request.json()
    task = await parse_task_json(data)
    llm_config = task.llm_config
    history = await browser_agent_manager.run_task_only(
        task.prompt,
        model=llm_config["model_name"],
        api_key=llm_config["api_key"],
        base_url=llm_config["api_url"],
        conversation_id=task.conversation_id,
    )
    response = create_response(
        200,
        "Task Finished",
        {
            "time": datetime.datetime.now().isoformat(),
            "time_cost": (end_time - start_time).total_seconds(),
            "history": history
        },
    )
    return response

The backend API deserializes the incoming JSON into a TaskRequest object using parse_task_json() from the utilities module. It extracts the LLM credentials (model_name, api_key, api_url) and delegates execution to the BrowserAgentManager. This manager maintains persistent browser sessions across requests to optimize performance.

Agent Orchestration with LangChain

The BrowserAgentManager constructs a LangChain agent capable of planning and executing browser actions. The agent factory resides in browser_server/browser_use/agent/agent.py.

# browser_server/browser_use/agent/agent.py

def get_agent(self, task: str, model: str, api_key: str, base_url, extend_prompt_id: int = -1,
              browser_session=None, conversation_id: str = None):
    llm = self._get_llm(model, api_key, base_url, conversation_id)
    extend_prompt = self.prompts_extend[extend_prompt_id]
    tool_calling_method = 'auto' if 'doubao' not in model else 'raw'
    return Agent(
        task=task,
        llm=llm,
        override_system_message=None,
        extend_system_message=extend_prompt,
        browser_session=browser_session,
        use_vision=False,
        tool_calling_method=tool_calling_method,
    )

The agent initialization binds the natural language task to a LangChain LLM instance, attaches an extended system prompt for behavioral guidance, and injects a shared Playwright browser session. The tool_calling_method adapts to different model providers, switching to 'raw' for Doubao models while using 'auto' for others.

Task Execution and Response Formatting

The actual browsing session executes within browser_server/browser_use/service/browser_agent.py, where the manager runs the agent and formats the results.

# browser_server/browser_use/service/browser_agent.py

async def run_task_only(self, task: str, model: str, api_key: str,
                       base_url: str, conversation_id: Optional[str] = None) -> str:
    uid = str(uuid.uuid4())
    agent = browser_agent.get_agent(
        task=task, model=model, api_key=api_key,
        base_url=base_url, browser_session=self.browser_session,
        conversation_id=conversation_id
    )
    history = await agent.run(max_steps=config['agent']['max_steps'])
    result = self._format_history(history) or [self._get_null_response_result(model=model)]
    return {
        "uid": uid,
        "task": task,
        "status": "finished",
        "time": datetime.now().strftime("%Y%m%d%H%M%S"),
        "final_browser_result": self._get_final_result(history),
        "browser_history": result,
        "browser_history_screenshot": history.screenshots(),
    }

The execution engine runs the agent with a configurable step limit defined in config.yaml, captures the structured interaction history, and extracts base64-encoded screenshots from the Playwright session. The response includes a unique task ID, timing metadata, the final extracted content, and visual evidence of the browsing session.

Practical Usage Examples

Invoking the Browser Tool via LLM Tool Call

When configuring an LLM with the browser tool, the interaction follows this pattern:

{
  "messages": [
    {"role": "user", "content": "Find the latest weather forecast for Berlin and give me the temperature."}
  ],
  "tools": [
    {"type": "function", "function": {"name": "browser", "description": "Interact with the browser.", "parameters": {"type": "object", "properties": {"question": {"type": "string", "description": "What you want to do with a browser"}}, "required": ["question"]}}}
  ]
}

The LLM generates a tool call:

{
  "name": "browser",
  "arguments": {"question": "Search for “Berlin weather today” and extract the temperature from the first result."}
}

The system returns a structured response containing the extracted text and screenshot history:

{
  "status": "success",
  "content": "The current temperature in Berlin is 13 °C.",
  "meta": {
    "action_type": "browser",
    "json": {
      "browser_history": [...],
      "browser_history_screenshot": ["data:image/png;base64,..."]
    }
  }
}

Direct API Access for Debugging

Developers can bypass the frontend and call the backend directly:

curl -X POST http://localhost:9000/api/browser/task \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "Search for “Python 3.12 release notes” and give me the first three bullet points.",
        "llm_config": {
          "model_name": "gpt-4o-mini",
          "api_key": "<YOUR_OPENAI_KEY>",
          "api_url": "https://api.openai.com/v1"
        }
      }'

Summary

• Three-tier architecture: The browser tool separates frontend tool definition (src/tools/browser.js), runtime HTTP bridging (src/runtime/browser.js), and backend agent execution (browser_server/browser_use/).
• Natural language interface: The tool accepts a single question parameter that the LangChain agent translates into concrete Playwright actions.
• Persistent sessions: The BrowserAgentManager maintains reusable Playwright browser sessions across API calls to reduce initialization overhead.
• Comprehensive telemetry: Every task returns extracted content, structured history, and base64-encoded screenshots for verification and debugging.
• Model flexibility: The agent factory supports multiple LLM providers through configurable tool_calling_method and custom base URLs.

Frequently Asked Questions

How does the frontend communicate with the browser automation backend?

The frontend runtime in src/runtime/browser.js sends an HTTP POST request to http://localhost:9000/api/browser/task with the user's prompt and LLM configuration. This REST API acts as the sole communication channel between the JavaScript frontend and the Python-based browser service.

What browser engine powers the automation?

As implemented in hexdocom/lemonai, the backend uses Playwright to control a headless browser session. The BrowserAgentManager initializes and maintains this session, passing it to the LangChain agent to perform navigation, clicking, and extraction operations.

How are screenshots captured during automated browsing?

The agent execution method run_task_only() in browser_server/browser_use/service/browser_agent.py calls history.screenshots() on the agent's result object, which returns an array of base64-encoded PNG images representing the browser state at key interaction points.

Can I configure the maximum number of browsing steps?

Yes. The agent respects the max_steps parameter defined in browser_server/browser_use/config/config.yaml. This configuration limits how many actions the LangChain agent will attempt before terminating the task, preventing infinite loops on complex or ambiguous browsing instructions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}