# How to Programmatically Control the Mouse, Keyboard, and Browser Using the Computer API

> Programmatically control your mouse, keyboard, and browser with the Computer API in Open Interpreter. Easily automate tasks using simple methods like computer.mouse.move() and computer.keyboard.write().

- Repository: [Open Interpreter/open-interpreter](https://github.com/openinterpreter/open-interpreter)
- Tags: how-to-guide
- Published: 2026-03-05

---

**The Computer API exposes a pre-instantiated `computer` object that aggregates mouse, keyboard, and browser automation tools, allowing direct programmatic control via methods like `computer.mouse.move()`, `computer.keyboard.write()`, and `computer.browser.search()` without requiring explicit imports.**

The openinterpreter/open-interpreter repository includes a built-in automation layer called the **Computer API**. This interface groups low-level system controls behind a single object that is automatically available in every Open Interpreter session, enabling you to programmatically control the mouse, keyboard, and browser using simple Python method calls.

## Architecture of the Computer API

The Computer API follows a hub-and-spoke design where a central class instantiates specialized sub-tools and exposes them as attributes.

### The Central Computer Hub

In [`interpreter/core/computer/computer.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/computer.py), the `Computer` class serves as the main entry point. It receives the top-level interpreter object during initialization and creates instances of `Mouse`, `Keyboard`, and `Browser`. The class also generates a **system message** (`self.system_message`) that catalogs all public methods of the sub-tools and injects this documentation into the LLM prompt, ensuring the model knows it can invoke automation commands.

### Mouse, Keyboard, and Browser Sub-tools

Each sub-tool is implemented in its own module and registered as an attribute of the central `computer` object:

- **`computer.mouse`** – Defined in [`interpreter/core/computer/mouse/mouse.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/mouse/mouse.py), this tool wraps **pyautogui** to handle cursor movement, clicking, and scrolling. It includes safety helpers that locate UI elements by visible **text** or **icon** using OCR rather than requiring raw coordinates.
- **`computer.keyboard`** – Located in [`interpreter/core/computer/keyboard/keyboard.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/keyboard/keyboard.py), this module uses pyautogui (or the system clipboard on macOS) to type strings, press individual keys, execute hotkey combinations, and manage key-down/up events.
- **`computer.browser`** – Implemented in [`interpreter/core/computer/browser/browser.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/browser/browser.py), this tool manages a Selenium WebDriver (Chrome) instance and provides high-level helpers such as `search`, `fast_search`, `go_to_url`, and `analyze_page`. It can call the Open Interpreter hosted API at `/browser/search` for quick results or drive an interactive headless Chrome session.

## How the Computer API Initialization Works

The initialization flow in [`interpreter/core/computer/computer.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/computer.py) follows three steps:

1. `Computer.__init__` receives the top-level interpreter object.
2. It instantiates `Mouse`, `Keyboard`, and `Browser`, passing a reference back to the `Computer` instance (`self`) so sub-tools can access shared state like verbosity settings.
3. The `Computer` aggregates method signatures from all sub-tools to build the system message that describes the full public API to the LLM.

Because the `computer` module is **pre-imported** in every Open Interpreter session, you can directly call automation commands. Attempting to explicitly `import computer` would break the system, as noted in the `Computer.system_message` documentation.

## Controlling the Mouse with the Computer API

The `computer.mouse` tool provides pixel-precise control while prioritizing safety through visual recognition.

### Moving and Clicking by Text or Coordinates

The `computer.mouse.move()` method accepts **coordinates**, **text strings**, or **icon references**. When you pass a text string, the method captures a screenshot, runs OCR via `computer.display.find()`, and resolves the first matching location. If multiple matches exist, it raises a `ValueError` accompanied by an annotated debug image.

```python

# Move the cursor to a button that says "Submit" and click it

computer.mouse.move("Submit")          # Finds the text on screen and moves there

computer.mouse.click()                # Clicks the current mouse location

```

Internally, `mouse.move` calls `smooth_move_to` which wraps `pyautogui.moveTo` to animate the cursor path. When `self.computer.verbose` is enabled, the tool displays visual debugging aids such as screenshots with colored circles around click targets.

## Automating Keyboard Input

The `computer.keyboard` tool optimizes text entry by choosing between keystroke simulation and clipboard pasting based on input length and platform.

### Writing Text and Sending Hotkeys

The `computer.keyboard.write()` method detects the operating system and text length to determine the fastest input strategy. On macOS, it often uses the system clipboard for long strings, while short strings use `pyautogui.write`. When the input ends with a newline, the method automatically presses `"enter"` afterward.

```python

# Type a multi-line email body

email_body = """Hi Alice,

Could you review the attached report?

Thanks,
Bob
"""
computer.keyboard.write(email_body)   # Uses clipboard for speed; ends with Enter automatically

```

For individual keys or combinations, use `computer.keyboard.press()` for single keys or `computer.keyboard.hotkey()` for chords like `command+c`.

## Browser Automation Methods

The `computer.browser` tool bridges local Selenium automation with remote API endpoints for hybrid web control.

### Navigating URLs and Searching the Web

The `computer.browser.go_to_url()` method lazily initializes a Selenium `webdriver.Chrome` instance and navigates to the specified address. The `search_google()` method automates the browser UI by sending `Keys.COMMAND + "k"` to focus the search bar.

```python

# Open a website, search for a term, and click the first result

computer.browser.go_to_url("https://news.ycombinator.com")
computer.browser.search_google("OpenAI GPT‑4", delays=False)  # Fast, no artificial wait

# Assume the first result contains the text "OpenAI"

computer.mouse.move("OpenAI")         # Locate the link by its visible text

computer.mouse.click()

```

### Using the Browser Search API Endpoint

For programmatic access without UI interaction, `computer.browser.search()` sends a GET request to the Open Interpreter backend endpoint at `{api_base}/browser/search` and returns structured JSON results.

```python

# Get a concise list of top results from the Open Interpreter backend

results = computer.browser.search("latest Python 3.12 release notes")
for i, r in enumerate(results[:5], 1):
    print(f"{i}. {r['title']}\n   {r['url']}\n")

```

The `computer.browser.fast_search()` method runs this HTTP request in a background thread while simultaneously opening a Perplexity AI tab and performing a live Google search, returning whichever result arrives first.

## Complete Workflow Examples

Combine all three sub-tools to automate complex multi-step tasks:

```python

# Automate a search-and-type scenario:

computer.browser.go_to_url("https://google.com")
computer.keyboard.write("OpenAI ChatGPT\n")  # Types into the Google search box and hits Enter

computer.mouse.move("ChatGPT")               # Find the first result by its link text

computer.mouse.click()

```

This workflow executes without any explicit import statements, as the pre-instantiated `computer` object is already present in the Open Interpreter namespace.

## Key Source Files and Implementation Details

The following files define the public surface of the Computer API according to the openinterpreter/open-interpreter source code:

- [`interpreter/core/computer/computer.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/computer.py) – Main `Computer` class, tool discovery, and system message generation.
- [`interpreter/core/computer/mouse/mouse.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/mouse/mouse.py) – Mouse actions including `move`, `click`, `scroll`, visual debugging, and OCR-based targeting.
- [`interpreter/core/computer/keyboard/keyboard.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/keyboard/keyboard.py) – Keyboard actions including `write`, `press`, `hotkey`, and clipboard optimization.
- [`interpreter/core/computer/browser/browser.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/browser/browser.py) – Browser wrapper managing Selenium WebDriver, `search`, `fast_search`, `go_to_url`, and `analyze_page`.

## Summary

- The **Computer API** exposes mouse, keyboard, and browser controls through a pre-instantiated `computer` object available in every Open Interpreter session.
- **Mouse automation** supports coordinate-based or OCR-based targeting via `computer.mouse.move()`, with visual debugging helpers in [`interpreter/core/computer/mouse/mouse.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/mouse/mouse.py).
- **Keyboard automation** optimizes input speed using clipboard operations or pyautogui via `computer.keyboard.write()`, implemented in [`interpreter/core/computer/keyboard/keyboard.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/keyboard/keyboard.py).
- **Browser automation** combines local Selenium control with remote API endpoints via `computer.browser.search()` and `computer.browser.go_to_url()`, defined in [`interpreter/core/computer/browser/browser.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/browser/browser.py).
- No explicit `import computer` is required; the object is automatically injected into the session namespace along with a system message documenting all available methods.

## Frequently Asked Questions

### Do I need to import the Computer API manually?

No. The `computer` object is pre-imported and pre-instantiated in every Open Interpreter session. According to the source code in [`interpreter/core/computer/computer.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/computer.py), attempting to explicitly `import computer` will break the system. The object is automatically injected into the namespace with its sub-tools (mouse, keyboard, browser) already attached.

### How does the Computer API find UI elements without exact coordinates?

The `computer.mouse.move()` method in [`interpreter/core/computer/mouse/mouse.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/mouse/mouse.py) uses OCR to locate elements by visible text or icon. It captures a screenshot and calls `computer.display.find()` to resolve coordinates dynamically. If multiple matches are found, it raises a `ValueError` with an annotated image to help you disambiguate the target.

### Can I use the Computer API outside of Open Interpreter sessions?

The Computer API is designed specifically for the Open Interpreter runtime environment. While the underlying code uses standard libraries like pyautogui and Selenium, the `computer` object initialization and system message generation are tightly coupled to the Open Interpreter context. To use these tools standalone, you would need to manually instantiate the `Computer` class from [`interpreter/core/computer/computer.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/computer.py) and provide the required interpreter context.

### What is the difference between browser.search and browser.fast_search?

According to [`interpreter/core/computer/browser/browser.py`](https://github.com/openinterpreter/open-interpreter/blob/main/interpreter/core/computer/browser/browser.py), `computer.browser.search()` performs a synchronous HTTP GET request to the Open Interpreter API endpoint `/browser/search` and returns JSON results. In contrast, `computer.browser.fast_search()` runs this API request in a background thread while simultaneously opening a Perplexity AI tab and performing a live Google search, returning whichever result completes first for maximum speed.