What Kind of Tests Are Included in OpenWork? A Complete Guide to the Testing Architecture

OpenWork ships a comprehensive automated-testing suite that combines fast Vitest unit tests with full-stack end-to-end specifications using the custom @openwork/testkit framework, which records observable evidence like screenshots and logs against real desktop and headless UI instances.

The different-ai/openwork repository implements a multi-layered testing strategy designed to validate everything from isolated utility functions to complex user workflows involving live LLM providers. This architecture ensures that both the Electron desktop application and the underlying server logic meet strict correctness standards before reaching production.

Unit and Integration Tests with Vitest

Test Location and Structure

The fastest tier of the OpenWork test suite consists of unit and integration tests located adjacent to production code, following the apps/server/src/*.test.ts naming pattern. These tests run with Vitest and validate pure functions, API routes, and internal utilities without requiring the UI to launch or external services to be provisioned.

Key Unit Test Examples

Real-world examples demonstrate the scope of unit testing across the codebase:

• apps/server/src/workspaces.test.ts validates workspace creation logic and runtime path normalization
• ee/packages/utils/src/observability.test.ts verifies telemetry helpers and observability utilities used by the testkit itself

import { expect, test } from "vitest";
import { normalizeWorkspaceRelativePath } from "./server/src/runtime-provider-merge";

test("normalizes a relative path", () => {
  const result = normalizeWorkspaceRelativePath("../foo/bar.txt");
  expect(result).toBe("foo/bar.txt");
});

Slow Specification Tests and @openwork/testkit

Full-Stack E2E Architecture

OpenWork's most distinctive testing layer consists of "slow" specification tests (*.slow.test.ts) that exercise the complete application stack. These tests launch the Electron desktop app (or the headless web UI) and interact with a live Den control plane, using the @openwork/testkit package to orchestrate complex user scenarios.

Each specification functions as a voice-over spec that describes user behavior in natural language while programmatically driving the application through helpers like app(), server(), control(), and sendComposerMessage().

Evidence-Based Validation

The testkit implements an "evidence" layer that automatically captures tapes (video-like recordings) for each test run. This evidence includes:

• Screenshots captured via screenshot(desktopApp) after significant UI actions
• Validation checks using validate(screenshot, [...]) that perform vision-plus-DOM verification to ensure the UI displays expected states

This approach catches visual regressions that pure JavaScript logic tests might miss, ensuring that hidden UI bugs are detected even when the underlying code passes unit tests.

Critical Slow Test Specifications

The evals/specs/ directory contains several high-value test scenarios that verify real-world behavior:

• evals/specs/model-lands-mid-run.slow.test.ts: Verifies that granting a new model mid-task does not interrupt the user's flow, ensuring the model only becomes selectable after task completion
• evals/specs/org-team-lifecycle-mega.slow.test.ts: Exercises the complete organization and team lifecycle—including creation, invitation, and role changes—in a single long-running specification
• evals/specs/cloud-provider-sync-contract.slow.test.ts: Validates synchronization between the local OpenWork engine and remote LLM providers, ensuring graceful handling of provider updates
• evals/specs/headless-web-source-server.test.ts: Confirms the headless-web UI (Vite + openwork-server) mirrors desktop behavior without requiring Electron
• evals/specs/app-smoke.slow.test.ts: Provides a quick sanity check that the desktop application starts, signs in, and executes basic chat requests

Test Execution Strategy

Fast Path for Rapid Feedback

The Vitest runner executes all *.test.ts files in under a minute, providing immediate feedback on logic correctness during development. These tests require no external services, UI initialization, or real model credentials.

Slow Path with Daytona Sandboxes

End-to-end specifications execute via pnpm test:e2e or the built-in run-tests skill. This process spins up a Daytona sandbox, provisions a temporary Den instance with real model credentials, launches the desktop or headless UI, and executes the *.slow.test.ts suite. The CI pipeline marks a run as passed only when every tape is recorded and all validation steps succeed.

These slow specs are deliberately separated from fast unit tests so the CI can run them in parallel on dedicated sandboxes with fresh Den instances.

import { app, server, test, needs } from "@openwork/testkit";

test("basic app smoke", { timeout: 180_000 }, async ({ evidence, place }) => {
  // Spin up a temporary Den and desktop instance
  await using den = await server({ place });
  await using desktop = await app({ den, as: "admin", place });

  // Verify the app can reach its Den and display a welcome banner
  const banner = await desktop.page.textContent("[data-test=welcome-banner]");
  expect(banner).toContain("Welcome");

  // Capture evidence for CI
  const shot = await screenshot(desktop);
  const ok   = await validate(shot, ["Welcome banner is visible"]);
  evidence.fact("welcome banner appears", ok.ok);
});

Summary

• OpenWork combines Vitest unit tests with custom @openwork/testkit E2E specifications to achieve full-stack coverage according to the different-ai/openwork source code
• Unit tests (*.test.ts) reside next to source code in paths like apps/server/src/ and validate pure functions without UI initialization
• Slow specifications (*.slow.test.ts) in evals/specs/ test real user scenarios against live Den control planes and Electron instances
• The evidence layer captures screenshots and validation results via screenshot() and validate() to prevent visual regressions
• Daytona sandboxes provide isolated, production-like environments for CI execution of slow tests with real LLM credentials

Frequently Asked Questions

What testing framework does OpenWork use for unit tests?

OpenWork uses Vitest for all unit and integration testing. Tests are co-located with source code using the *.test.ts naming convention and validate logic in files like apps/server/src/workspaces.test.ts without launching the desktop application or external services.

How does OpenWork test the Electron desktop application?

The repository uses @openwork/testkit to launch real Electron instances (or headless web UIs) in temporary Daytona sandboxes. These "slow" specifications interact with live Den control planes and real LLM providers to validate complete user workflows rather than mocking dependencies.

What is the difference between *.test.ts and *.slow.test.ts files?

Files ending in *.test.ts are fast Vitest unit tests that run in under a minute without UI dependencies. Files ending in *.slow.test.ts are full-stack specifications that boot the actual application, execute against real services, and record evidence tapes for CI validation.

How does OpenWork prevent UI regressions in its test suite?

The @openwork/testkit framework implements an evidence layer that automatically captures screenshots after significant actions and runs vision-plus-DOM validation checks. This ensures the UI displays correct states even when underlying JavaScript logic passes traditional unit tests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}