How to Implement Browser Automation with AI Agents: A Complete Guide to Learning and Replay

You can implement browser automation with AI agents by using a two-phase architecture where an LLM-powered agent first learns a task by capturing stable XPath selectors, then replays the stored workflow via Playwright without further LLM calls, achieving approximately 75% faster execution.

The bojieli/ai-agent-book repository demonstrates a production-ready pattern for giving large language models direct control over web browsers. Built on top of the browser-use library (which wraps Microsoft Playwright), this architecture separates the cognitive task of understanding user intent from the mechanical execution of browser actions, enabling both adaptive learning and high-speed replay.

Architecture Overview

The implementation follows a strict separation of concerns across four distinct layers. According to the source code in chapter8/browser-use-rpa/IMPLEMENTATION_DETAILS.md, each layer handles a specific responsibility:

• Agent Layer (learning_agent/agent.py): Orchestrates LLM calls and intercepts the step method to capture actions during the learning phase.
• Knowledge Base (learning_agent/knowledge_base.py): Persists learned workflows and provides semantic intent matching to determine if an existing workflow can handle a new request.
• Replay Engine (learning_agent/replay.py): Executes stored workflows directly via Playwright, handling XPath resolution and fallback selectors.
• Browser Wrapper: Supplies the high-level Browser API from the browser-use library, built on Playwright.

This design allows the system to operate in two distinct modes: the learning phase, where the LLM explores and records actions, and the replay phase, where the system executes pre-recorded steps at native speed.

The Learning Phase: Capturing Workflows

During the learning phase, the agent receives a natural-language task description and interacts with the browser while extracting stable selectors for every action.

Installing the Browser-Use Stack

Begin by installing the core dependencies and browser drivers:

# Install the browser-use package (Python ≥ 3.11)

uv pip install browser-use

# Install the Chromium driver via Playwright

uvx playwright install chromium --with-deps --no-shell

Running Your First AI Agent

Create a basic agent that can navigate and interact with web pages:

from browser_use import Agent, ChatGoogle
from dotenv import load_dotenv
load_dotenv()               # loads GEMINI_API_KEY from .env

agent = Agent(
    task="Find the number of stars of the browser-use repo",
    llm=ChatGoogle(model="gemini-2.5-flash"),
    # Uncomment to use the Browser-Use cloud service:

    # browser=Browser(use_cloud=True),

)

agent.run_sync()            # blocks until the task finishes

Capturing Stable Selectors

The critical innovation in learning_agent/agent.py is the _get_element_info method, which extracts robust XPaths from EnhancedDOMTreeNode objects provided by browser-use. After each successful interaction, the agent captures:

• The primary XPath for deterministic element targeting
• Fallback CSS selectors for resilience
• Action parameters (click coordinates, input text, etc.)

As implemented in the source code, this information is serialized into a structured workflow object. The original step method is monkey-patched to preserve compatibility while adding capture logic, ensuring that the learning process remains transparent to the underlying browser operations.

The Replay Phase: Executing Without LLM Calls

Once a workflow is learned, the system can replay it without invoking the language model, eliminating latency and API costs.

Executing Stored Workflows

Use the replay engine to execute a previously saved workflow:

from learning_agent.replay import replay_workflow

replay_workflow(
    workflow_path="./email_knowledge/email_task.json",
    params={"recipient": "alice@example.com", "subject": "Hello"},
)

The replay engine (learning_agent/replay.py) loads the JSON workflow, resolves the stored XPath selectors, and drives Playwright directly. Each step waits for the target element to become visible using wait_for(state='visible') before interaction, ensuring synchronization with dynamic web applications.

Fallback Strategies for Robust Automation

The replay implementation includes a cascading fallback mechanism for element location:

1. Primary XPath: The originally captured XPath from the learning phase
2. CSS Selector: Fallback to CSS if XPath resolution fails
3. Attribute Matching: Final fallback using element attributes

This multi-layer approach ensures that minor DOM changes do not break automation scripts, maintaining reliability across sessions.

Building the Knowledge Base

The knowledge base (learning_agent/knowledge_base.py) serves as the persistent storage and retrieval layer for learned workflows.

Semantic Intent Matching

When a new request arrives, the system calculates a confidence score using _calculate_match_confidence to determine whether an existing workflow can satisfy the intent. This allows the agent to recognize that "send a follow-up email" can reuse the workflow learned for "send an email" without requiring a new learning phase.

You can interact with the knowledge base programmatically:

from learning_agent.knowledge_base import KnowledgeBase

kb = KnowledgeBase("./knowledge")
kb.save_workflow("email_send", workflow_object)
matches = kb.search_intent("send a follow-up email")

Performance Benefits

According to the implementation details in the repository, separating learning from replay yields significant performance improvements. The replay phase executes approximately 75% faster than the learning phase because it eliminates LLM inference calls entirely. Once a workflow is captured, the system operates at native Playwright speeds, making it suitable for high-frequency automation tasks.

Summary

• Browser automation with AI agents requires wrapping LLM cognition with robust selector extraction and persistent storage mechanisms.
• The learning phase captures stable XPaths via _get_element_info in learning_agent/agent.py, serializing actions to JSON workflows.
• The replay phase executes stored workflows through learning_agent/replay.py using Playwright directly, without LLM overhead.
• Semantic matching in learning_agent/knowledge_base.py enables workflow reuse across similar intents.
• This architecture achieves ~75% time reduction in replay mode compared to live LLM-driven automation.

Frequently Asked Questions

What is the difference between the learning phase and the replay phase in browser automation?

The learning phase involves the AI agent using an LLM to understand a task, interact with the browser, and capture stable selectors (XPath/CSS) for each action. The replay phase executes the pre-recorded workflow directly via Playwright without LLM calls, resulting in faster, deterministic execution. According to the bojieli/ai-agent-book source code, this separation reduces execution time by approximately 75%.

How does the system handle website changes that might break stored selectors?

The replay engine in learning_agent/replay.py implements a cascading fallback strategy. It first attempts to locate elements using the stored XPath, then falls back to CSS selectors, and finally uses attribute matching. Additionally, each step waits for elements to become visible (wait_for(state='visible')) to handle dynamic loading and DOM mutations gracefully.

Can I use this architecture with different LLM providers?

Yes. The browser-use library supports multiple LLM backends through its chat model interfaces. The examples in the repository use ChatGoogle with Gemini models, but you can substitute any compatible LLM client that conforms to the library's expected interface. The learning and replay mechanisms remain agnostic to the specific LLM used during the initial capture phase.

What file formats are used to store learned workflows?

Workflows are serialized as JSON files containing the sequence of actions, XPath selectors, fallback CSS selectors, and parameter placeholders. The KnowledgeBase class in learning_agent/knowledge_base.py handles the storage and retrieval, organizing workflows by semantic intent to enable reuse across similar tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}