How to Programmatically Control the Mouse, Keyboard, and Browser Using the Computer API

The Computer API exposes a pre-instantiated computer object that aggregates mouse, keyboard, and browser automation tools, allowing direct programmatic control via methods like computer.mouse.move(), computer.keyboard.write(), and computer.browser.search() without requiring explicit imports.

The openinterpreter/open-interpreter repository includes a built-in automation layer called the Computer API. This interface groups low-level system controls behind a single object that is automatically available in every Open Interpreter session, enabling you to programmatically control the mouse, keyboard, and browser using simple Python method calls.

Architecture of the Computer API

The Computer API follows a hub-and-spoke design where a central class instantiates specialized sub-tools and exposes them as attributes.

The Central Computer Hub

In interpreter/core/computer/computer.py, the Computer class serves as the main entry point. It receives the top-level interpreter object during initialization and creates instances of Mouse, Keyboard, and Browser. The class also generates a system message (self.system_message) that catalogs all public methods of the sub-tools and injects this documentation into the LLM prompt, ensuring the model knows it can invoke automation commands.

Mouse, Keyboard, and Browser Sub-tools

Each sub-tool is implemented in its own module and registered as an attribute of the central computer object:

• computer.mouse – Defined in interpreter/core/computer/mouse/mouse.py, this tool wraps pyautogui to handle cursor movement, clicking, and scrolling. It includes safety helpers that locate UI elements by visible text or icon using OCR rather than requiring raw coordinates.
• computer.keyboard – Located in interpreter/core/computer/keyboard/keyboard.py, this module uses pyautogui (or the system clipboard on macOS) to type strings, press individual keys, execute hotkey combinations, and manage key-down/up events.
• computer.browser – Implemented in interpreter/core/computer/browser/browser.py, this tool manages a Selenium WebDriver (Chrome) instance and provides high-level helpers such as search, fast_search, go_to_url, and analyze_page. It can call the Open Interpreter hosted API at /browser/search for quick results or drive an interactive headless Chrome session.

How the Computer API Initialization Works

The initialization flow in interpreter/core/computer/computer.py follows three steps:

1. Computer.__init__ receives the top-level interpreter object.
2. It instantiates Mouse, Keyboard, and Browser, passing a reference back to the Computer instance (self) so sub-tools can access shared state like verbosity settings.
3. The Computer aggregates method signatures from all sub-tools to build the system message that describes the full public API to the LLM.

Because the computer module is pre-imported in every Open Interpreter session, you can directly call automation commands. Attempting to explicitly import computer would break the system, as noted in the Computer.system_message documentation.

Controlling the Mouse with the Computer API

The computer.mouse tool provides pixel-precise control while prioritizing safety through visual recognition.

Moving and Clicking by Text or Coordinates

The computer.mouse.move() method accepts coordinates, text strings, or icon references. When you pass a text string, the method captures a screenshot, runs OCR via computer.display.find(), and resolves the first matching location. If multiple matches exist, it raises a ValueError accompanied by an annotated debug image.

# Move the cursor to a button that says "Submit" and click it

computer.mouse.move("Submit")          # Finds the text on screen and moves there

computer.mouse.click()                # Clicks the current mouse location

Internally, mouse.move calls smooth_move_to which wraps pyautogui.moveTo to animate the cursor path. When self.computer.verbose is enabled, the tool displays visual debugging aids such as screenshots with colored circles around click targets.

Automating Keyboard Input

The computer.keyboard tool optimizes text entry by choosing between keystroke simulation and clipboard pasting based on input length and platform.

Writing Text and Sending Hotkeys

The computer.keyboard.write() method detects the operating system and text length to determine the fastest input strategy. On macOS, it often uses the system clipboard for long strings, while short strings use pyautogui.write. When the input ends with a newline, the method automatically presses "enter" afterward.

# Type a multi-line email body

email_body = """Hi Alice,

Could you review the attached report?

Thanks,
Bob
"""
computer.keyboard.write(email_body)   # Uses clipboard for speed; ends with Enter automatically

For individual keys or combinations, use computer.keyboard.press() for single keys or computer.keyboard.hotkey() for chords like command+c.

Browser Automation Methods

The computer.browser tool bridges local Selenium automation with remote API endpoints for hybrid web control.

Navigating URLs and Searching the Web

The computer.browser.go_to_url() method lazily initializes a Selenium webdriver.Chrome instance and navigates to the specified address. The search_google() method automates the browser UI by sending Keys.COMMAND + "k" to focus the search bar.

# Open a website, search for a term, and click the first result

computer.browser.go_to_url("https://news.ycombinator.com")
computer.browser.search_google("OpenAI GPT‑4", delays=False)  # Fast, no artificial wait

# Assume the first result contains the text "OpenAI"

computer.mouse.move("OpenAI")         # Locate the link by its visible text

computer.mouse.click()

Using the Browser Search API Endpoint

For programmatic access without UI interaction, computer.browser.search() sends a GET request to the Open Interpreter backend endpoint at {api_base}/browser/search and returns structured JSON results.

# Get a concise list of top results from the Open Interpreter backend

results = computer.browser.search("latest Python 3.12 release notes")
for i, r in enumerate(results[:5], 1):
    print(f"{i}. {r['title']}\n   {r['url']}\n")

The computer.browser.fast_search() method runs this HTTP request in a background thread while simultaneously opening a Perplexity AI tab and performing a live Google search, returning whichever result arrives first.

Complete Workflow Examples

Combine all three sub-tools to automate complex multi-step tasks:

# Automate a search-and-type scenario:

computer.browser.go_to_url("https://google.com")
computer.keyboard.write("OpenAI ChatGPT\n")  # Types into the Google search box and hits Enter

computer.mouse.move("ChatGPT")               # Find the first result by its link text

computer.mouse.click()

This workflow executes without any explicit import statements, as the pre-instantiated computer object is already present in the Open Interpreter namespace.

Key Source Files and Implementation Details

The following files define the public surface of the Computer API according to the openinterpreter/open-interpreter source code:

• interpreter/core/computer/computer.py – Main Computer class, tool discovery, and system message generation.
• interpreter/core/computer/mouse/mouse.py – Mouse actions including move, click, scroll, visual debugging, and OCR-based targeting.
• interpreter/core/computer/keyboard/keyboard.py – Keyboard actions including write, press, hotkey, and clipboard optimization.
• interpreter/core/computer/browser/browser.py – Browser wrapper managing Selenium WebDriver, search, fast_search, go_to_url, and analyze_page.

Summary

• The Computer API exposes mouse, keyboard, and browser controls through a pre-instantiated computer object available in every Open Interpreter session.
• Mouse automation supports coordinate-based or OCR-based targeting via computer.mouse.move(), with visual debugging helpers in interpreter/core/computer/mouse/mouse.py.
• Keyboard automation optimizes input speed using clipboard operations or pyautogui via computer.keyboard.write(), implemented in interpreter/core/computer/keyboard/keyboard.py.
• Browser automation combines local Selenium control with remote API endpoints via computer.browser.search() and computer.browser.go_to_url(), defined in interpreter/core/computer/browser/browser.py.
• No explicit import computer is required; the object is automatically injected into the session namespace along with a system message documenting all available methods.

Frequently Asked Questions

Do I need to import the Computer API manually?

No. The computer object is pre-imported and pre-instantiated in every Open Interpreter session. According to the source code in interpreter/core/computer/computer.py, attempting to explicitly import computer will break the system. The object is automatically injected into the namespace with its sub-tools (mouse, keyboard, browser) already attached.

How does the Computer API find UI elements without exact coordinates?

The computer.mouse.move() method in interpreter/core/computer/mouse/mouse.py uses OCR to locate elements by visible text or icon. It captures a screenshot and calls computer.display.find() to resolve coordinates dynamically. If multiple matches are found, it raises a ValueError with an annotated image to help you disambiguate the target.

Can I use the Computer API outside of Open Interpreter sessions?

The Computer API is designed specifically for the Open Interpreter runtime environment. While the underlying code uses standard libraries like pyautogui and Selenium, the computer object initialization and system message generation are tightly coupled to the Open Interpreter context. To use these tools standalone, you would need to manually instantiate the Computer class from interpreter/core/computer/computer.py and provide the required interpreter context.

What is the difference between browser.search and browser.fast_search?

According to interpreter/core/computer/browser/browser.py, computer.browser.search() performs a synchronous HTTP GET request to the Open Interpreter API endpoint /browser/search and returns JSON results. In contrast, computer.browser.fast_search() runs this API request in a background thread while simultaneously opening a Perplexity AI tab and performing a live Google search, returning whichever result completes first for maximum speed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}