MCP Servers for OS-Level UI Automation: 7 Tools for Desktop Control

Seven MCP servers in the awesome-mcp-servers repository enable genuine OS-level UI automation across Linux, Windows, and macOS, exposing tools for screenshots, mouse/keyboard injection, window management, and accessibility tree inspection.

The punkpeye/awesome-mcp-servers repository maintains a curated OS Automation category that catalogs Model Context Protocol (MCP) servers capable of controlling desktop operating systems without application-specific integrations. According to the source code analysis of README.md (lines 2863–2877), these servers translate MCP tool calls into native OS actions, allowing AI agents to observe and manipulate graphical user interfaces directly.

Linux UI Automation MCP Servers

GNOME Desktop Control (sbuysse/gnome-desktop-mcp)

The sbuysse/gnome-desktop-mcp server provides comprehensive automation for GNOME desktop environments versions 45–49. Located in the OS Automation section at line 2867, this server exposes tools for screenshots, window management, mouse and keyboard injection, clipboard operations, workspace manipulation, and notification handling.

Hybrid Computer-Use Runtime (dimpagk92/cellar)

dimpagk92/cellar operates on macOS (Apple Silicon) and Linux through a hybrid "computer-use" architecture that merges accessibility tree data, Chrome DevTools Protocol, and computer vision. As documented at line 2868, this server exposes four high-level MCP tools: see, act, think, and perceive, enabling both UI observation and interaction at the OS level.

Windows UI Automation MCP Servers

Native Windows UI Automation (Harusame64/desktop-touch-mcp)

Harusame64/desktop-touch-mcp targets native Windows UI using the UI Automation (UIA) framework and Chrome DevTools Protocol (CDP). Listed at lines 2869–2870, this server provides entity-based clicking, keyboard control, mouse manipulation, clipboard access, terminal control, and UI notifications.

Remote Windows Control (glasswarp/mcp-server)

The glasswarp/mcp-server enables automation of any Windows PC—local or remote over LAN—without requiring source code changes to target applications. Documented at line 2870, its capabilities include UI observation via screenshots and UIA, clicking, typing, dragging, scrolling, and application launching.

WPF Application Automation (faze79/WPFVisualTreeMcp)

For .NET ecosystems, faze79/WPFVisualTreeMcp specializes in running WPF desktop applications. According to line 2874, this server can inspect the visual tree, read and write properties, capture screenshots with pop-up handling, perform clicks, select items, input text, and execute keyboard shortcuts.

macOS UI Automation MCP Servers

Window Management & OCR (ostapondo/Plonk)

ostapondo/Plonk provides macOS-specific window manager and menu-bar utilities. Detailed at line 2876, this server exposes tools to list monitors and windows, arrange layouts, snap windows, save workspaces, keep the system awake, measure screen dimensions, capture window-specific screenshots, annotate images, and perform on-device OCR—all callable through standard MCP tools.

Cross-Platform UI Automation

Universal GUI Automation (fixed-width/glass)

fixed-width/glass offers the broadest platform support, running on Linux, macOS, Windows, Android, and iOS without requiring modifications to target applications. As noted at line 2875, this server provides tools to launch applications, capture screenshots, read accessibility trees, inject mouse/keyboard gestures, tail logs, and perform frame diffs.

Implementing UI Automation with MCP Tools

MCP clients discover automation capabilities through the standard tools/list method, then invoke specific actions using the tools/call method. The exact tool names and schemas vary by server, but the JSON-RPC pattern remains consistent.

Capturing Screenshots

To capture the current screen on a GNOME desktop:

{
  "method": "tools/call",
  "params": {
    "name": "screenshot",
    "arguments": {}
  }
}

Result: Returns a base64-encoded PNG of the current display.

Injecting Mouse Events

To click a specific UI element on Windows using element selectors:

{
  "method": "tools/call",
  "params": {
    "name": "click",
    "arguments": {
      "selector": "window.title='Calculator' && element.id='Button5'"
    }
  }
}

Result: Returns a success flag and any UI state changes.

Listing Active Windows

To enumerate visible windows using the cross-platform glass server:

{
  "method": "tools/call",
  "params": {
    "name": "list_windows",
    "arguments": {}
  }
}

Result: Returns a JSON array of objects containing id, title, and bounds for each window.

Performing OCR on UI Elements

To extract text from a specific screen region on macOS:

{
  "method": "tools/call",
  "params": {
    "name": "ocr_region",
    "arguments": {
      "window_name": "Notes",
      "region": {"x": 10, "y": 20, "width": 300, "height": 120}
    }
  }
}

Result: Returns extracted text strings with confidence scores.

Server Configuration and Manifests

Each OS automation server defines its concrete tool signatures in an mcp_manifest.json file located in the repository root. These manifests specify input_schema and output_schema definitions that MCP clients use to validate requests. The README.md file (lines 2863–2877) serves as the authoritative index, mapping platform capabilities to specific GitHub repositories.

Summary

• Seven specialized MCP servers provide OS-level UI automation according to the awesome-mcp-servers repository.
• Platform coverage includes Linux (GNOME), Windows (native UIA, WPF, remote), macOS (window management, OCR), and cross-platform implementations.
• Core capabilities span screenshots, mouse/keyboard injection, window management, accessibility tree inspection, and clipboard operations.
• Implementation pattern follows standard MCP protocol: clients call tools/list to discover capabilities, then tools/call with JSON arguments to execute actions.
• Source reference lines 2863–2877 in README.md contain the complete OS Automation category listing.

Frequently Asked Questions

Which MCP servers support cross-platform UI automation?

fixed-width/glass is the only server in the OS Automation category that supports Linux, macOS, Windows, Android, and iOS simultaneously. According to line 2875, it requires no source code changes to target applications and provides universal tools for screenshot capture, accessibility tree reading, and input injection across all supported platforms.

How do MCP servers inject keyboard and mouse events on Windows?

Windows-specific servers use different native APIs. Harusame64/desktop-touch-mcp utilizes the UI Automation (UIA) framework and Chrome DevTools Protocol for entity-based interaction (line 2869), while glasswarp/mcp-server implements direct input injection for clicking, typing, dragging, and scrolling operations (line 2870). Both expose these capabilities as standard MCP tools callable by any compatible client.

Can these servers automate applications without accessible APIs?

Yes. dimpagk92/cellar (line 2868) specifically addresses this by merging accessibility tree data with computer vision capabilities, allowing it to interact with applications lacking proper accessibility implementations. Similarly, fixed-width/glass performs frame differencing and visual analysis to automate arbitrary GUI apps regardless of their accessibility support.

What is the difference between gnome-desktop-mcp and Plonk?

While both provide OS-level automation, sbuysse/gnome-desktop-mcp targets Linux GNOME environments (versions 45–49) with deep desktop integration including workspace and notification management (line 2867). ostapondo/Plonk focuses exclusively on macOS, offering specialized window tiling, workspace saving, and on-device OCR capabilities not available in the GNOME implementation (line 2876).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}