OpenWork Performance Benchmarks: A Complete Technical Guide to Database, Workspace, and E2E Testing

The different-ai/openwork repository provides three core performance benchmarks: an admin-scale MySQL database benchmark, a Windows workspace-session performance flow, and a hands-free computer-use e2e benchmark, all runnable via pnpm commands defined in the root package.json.

OpenWork is an open-source AI-powered workspace platform that ships with comprehensive performance validation to ensure responsiveness across its database layer, Electron desktop client, and agent-driven workflows. This guide explains how each benchmark works architecturally, where the source code lives, and how to run them yourself.

Admin-Scale MySQL Benchmark

The most heavily instrumented benchmark measures end-to-end latency of the admin-panel database layer using a disposable MySQL instance.

What It Measures

This benchmark in [ee/packages/den-db/scripts/admin-scale-benchmark.ts](https://github.com/different-ai/openwork/blob/dev/ee/packages/den-db/scripts/admin-scale-benchmark.ts) captures:

• Initial connection latency to MySQL
• User search execution time
• Organization search execution time

The script automatically provisions a temporary database at mysql://root:password@127.0.0.1:3306/openwork_admin_scale_benchmark and destroys it after completion.

Architecture and Safety Guards

The benchmark enforces clean-state execution. From the source:

• It checks for git status --porcelain and throws if uncommitted changes exist
• Uses tsx for TypeScript execution without separate compilation
• Writes structured JSON results to evals/results/admin-scale-performance-benchmark/latest.json

How to Run

# Install dependencies first

pnpm install

# Execute the MySQL admin-scale benchmark

pnpm benchmark:admin-scale:mysql

Sample output includes timing breakdowns:

Resetting disposable benchmark database admin_scale_benchmark
... 500 ms initial, 300 ms searches

Windows Workspace-Session Performance Flow

This benchmark validates Electron desktop performance under heavy workspace and session load.

What It Measures

Located in evals/flows/windows-workspace-session-performance.flow.mjs, this benchmark tests:

• Workspace startup time on Windows
• Bulk session creation (default: 20 workspaces, 200 total sessions)
• UI navigation latency between sessions
• Responsive threshold compliance

The accompanying narrative document at [evals/voiceovers/windows-workspace-session-performance.md](https://github.com/different-ai/openwork/blob/dev/evals/voiceovers/windows-workspace-session-performance.md) describes the testing intent and pass criteria.

Test-Kit Integration

The flow uses the @openwork/testkit framework with ctx.prove and ctx.expectText APIs. It captures:

• Screenshots as evidence (tagged "admin-scale-budget-evidence")
• Timing assertions with benchmarkSessionCount and benchmarkSessionTotal variables
• Automatic profile isolation via OPENWORK_DEV_PROFILE=auto

How to Run

# Start the dev build

pnpm dev

# Run the workspace-session performance flow

pnpm test:e2e windows-workspace-session-performance

Runtime logs show progress:

benchmark session ready <sessionId>
benchmarkSessionCount: 20
benchmarkSessionTotal: 200

Hands-Free Computer-Use E2E Benchmark

The third benchmark measures end-to-end quality and latency of the autonomous "hands-free" agent flow.

Platform-Specific Behavior

Found in packages/handsfree/test/e2e/run.mjs, this benchmark:

• Skips entirely on non-macOS platforms with message: "computer-use e2e benchmark is macOS-only; skipping"
• Executes full agent loop on macOS, measuring perception-to-action latency
• Validates hands-free quality metrics including task completion rate and response time

How to Run

# Run as part of standard e2e suite

pnpm test:e2e handsfree

Or execute all e2e benchmarks together:

pnpm test:e2e

Benchmark Infrastructure and CI Integration

All benchmarks share common infrastructure patterns according to the OpenWork source code:

Pattern	Implementation
Disposable resources	Temporary MySQL containers, fresh Electron profiles
CLI entry points	package.json scripts like benchmark:admin-scale:mysql
Result persistence	JSON files in evals/results/ directories
Evidence capture	Screenshots and timing logs for regression analysis

The root [package.json](https://github.com/different-ai/openwork/blob/dev/package.json) defines the benchmark orchestration layer, using node --import tsx for TypeScript execution without build steps.

Summary

• Admin-scale MySQL benchmark validates database layer performance with disposable instances and structured JSON reporting.

• Windows workspace-session flow tests Electron desktop scalability under 200+ session loads using the testkit framework.

• Hands-free e2e benchmark measures autonomous agent quality, macOS-only, integrated into standard test suites.

• All benchmarks enforce clean Git state, isolate resources, and output machine-readable results for CI tracking.

Frequently Asked Questions

How do I add a new performance benchmark to OpenWork?

Create a .flow.mjs file in evals/flows/ following the testkit pattern used in windows-workspace-session-performance.flow.mjs. Add a narrative description in evals/voiceovers/ and register a pnpm script in root package.json for discoverability.

Why does the hands-free benchmark skip on my machine?

The intentional platform restriction in packages/handsfree/test/e2e/run.mjs limits execution to macOS because the underlying computer-use APIs depend on macOS accessibility frameworks. This is logged explicitly when running on Linux or Windows.

Where are benchmark results stored?

JSON artifacts persist to evals/results/<benchmark-name>/latest.json at runtime. The admin-scale benchmark generates this path automatically; other flows may configure output locations via environment variables in their testkit configuration.

Can I run benchmarks without installing MySQL locally?

Yes. The admin-scale benchmark in [ee/packages/den-db/scripts/admin-scale-benchmark.ts](https://github.com/different-ai/openwork/blob/dev/ee/packages/den-db/scripts/admin-scale-benchmark.ts) expects a running MySQL instance at the documented connection string, but you can modify the connection parameters or use containerized MySQL via Docker for isolated testing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}