How to Run Playwright E2E Tests for OpenMAIC: Complete Setup Guide

To run Playwright E2E tests for OpenMAIC, install dependencies with pnpm, ensure the application server runs on port 3002 with the Pro editor enabled, and execute pnpm exec playwright test from the repository root.

The OpenMAIC repository includes a comprehensive Playwright test suite that validates the web UI, agent workbench, and rendering pipeline. This guide explains how to configure and execute Playwright E2E tests for OpenMAIC by referencing the actual implementation in playwright.config.ts and the test specifications located in e2e/tests/.

Prerequisites for OpenMAIC E2E Testing

Before executing the test suite, ensure your environment meets the following requirements:

• Node.js version 22 or higher
• pnpm package manager installed globally

Verify your Node.js version with node --version before proceeding.

Running Playwright E2E Tests Locally

Executing the OpenMAIC Playwright suite locally involves three distinct phases: dependency installation, server configuration, and test execution.

Install Development Dependencies

Clone the repository and install all required packages using pnpm. The Playwright binaries and browser engines are downloaded automatically during this step.

git clone https://github.com/THU-MAIC/OpenMAIC.git
cd OpenMAIC
pnpm install

Configure the Application Server

According to playwright.config.ts, the test runner expects the application to be accessible on port 3002 with the Pro editor feature explicitly enabled. The configuration automatically handles server startup via the webServer block, but you must ensure the environment variables are correctly passed.

The critical environment variables are:

• PORT=3002
• NEXT_PUBLIC_MAIC_EDITOR_ENABLED=true

Locally, Playwright uses pnpm dev to start the server, while CI environments use pnpm start for production builds.

Execute the Test Suite

Run the full Playwright test suite using either the direct command or the package script shortcut defined in package.json:

# Direct execution

pnpm exec playwright test

# Or use the shortcut (if defined in package.json)

pnpm test:e2e

Playwright will automatically launch Chromium, navigate to http://localhost:3002, and execute all specifications in e2e/tests/ in parallel.

Understanding the Playwright Configuration

The playwright.config.ts file at the repository root defines the test behavior and environment setup. Key configuration properties include:

• testDir: './e2e/tests' – Specifies the directory containing all .spec.ts test files
• baseURL: 'http://localhost:3002' – Sets the base URL for all page navigations
• webServer.command – Dynamically selects between pnpm dev (local) and pnpm start (CI) based on the environment
• env – Injects required environment variables including PORT and NEXT_PUBLIC_MAIC_EDITOR_ENABLED

This configuration ensures consistent test execution across different environments without manual server management.

Running E2E Tests in CI Pipelines

When executing in continuous integration environments, set the CI environment variable to enable production-specific settings:

CI=1 pnpm exec playwright test

The CI-specific adaptations in playwright.config.ts include:

• forbidOnly: true – Prevents tests marked with .only from slipping into production
• retries: 2 – Automatically retries failed tests up to two times to handle flakiness
• workers: 2 – Limits parallelism to two workers, optimized for standard 4-core CI runners

These settings ensure reliable test execution in resource-constrained CI environments while maintaining strict quality gates.

Writing Playwright Tests for OpenMAIC

The OpenMAIC test suite follows the Page Object Model pattern for maintainable test code. Tests are located in e2e/tests/ while reusable page logic resides in e2e/pages/.

Example Test Specification

The following spec demonstrates navigating from the home page to course generation:

// e2e/tests/home-to-generation.spec.ts
import { test, expect, type Page } from '@playwright/test';
import { HomePage } from '../pages/home.page';

test('generate a course from the home page', async ({ page }) => {
  const home = new HomePage(page);
  await home.goto();                         // navigates to baseURL
  await home.fillPrompt('Quantum physics');  // type a topic
  await home.submit();                       // start generation
  await expect(page).toHaveURL(/\/course\/.+/); // redirects to the new course
});

Page Object Implementation

The corresponding page object encapsulates locator strategies and interaction methods:

// e2e/pages/home.page.ts
import { type Page, type Locator } from '@playwright/test';

export class HomePage {
  readonly page: Page;
  readonly promptInput: Locator;
  readonly generateBtn: Locator;

  constructor(page: Page) {
    this.page = page;
    this.promptInput = page.locator('#prompt');
    this.generateBtn = page.locator('button:has-text("Generate")');
  }

  async goto() {
    await this.page.goto('/');
  }

  async fillPrompt(text: string) {
    await this.promptInput.fill(text);
  }

  async submit() {
    await this.generateBtn.click();
  }
}

This architecture separates test logic from implementation details, making the suite resilient to UI changes.

Summary

Running Playwright E2E tests for OpenMAIC requires proper environment setup and understanding of the configuration in playwright.config.ts:

• Install dependencies using pnpm install after cloning the repository
• Ensure the application server listens on port 3002 with NEXT_PUBLIC_MAIC_EDITOR_ENABLED=true
• Execute tests via pnpm exec playwright test or the pnpm test:e2e shortcut
• Use CI=1 prefix in continuous integration environments to enable production build testing and retry logic
• Organize tests using the Page Object Model pattern with files in e2e/tests/ and helpers in e2e/pages/

Frequently Asked Questions

What Node.js version is required to run Playwright tests in OpenMAIC?

OpenMAIC requires Node.js version 22 or higher to execute the Playwright test suite. Earlier versions may fail during dependency installation or test execution due to incompatible features in the test runner or application code.

Why does the Playwright configuration use port 3002 instead of the default Next.js port 3000?

The playwright.config.ts explicitly sets PORT=3002 in the webServer environment block to avoid conflicts with other local development servers and to ensure consistent testing conditions. This configuration prevents interference from other Next.js instances that might be running on the standard port 3000 during development.

How can I run a specific test file instead of the entire OpenMAIC E2E suite?

To execute a single test file, append the relative path to the Playwright command:

pnpm exec playwright test e2e/tests/home-to-generation.spec.ts

You can also run tests in headed mode for debugging by adding the --headed flag, or use pnpm exec playwright test --ui to launch the interactive test runner interface.

Does OpenMAIC support Dockerized Playwright testing?

Yes, the repository includes optional Docker configuration files (Dockerfile and docker-compose.yml) that can containerize the application and test environment. This setup launches the application on port 3002 before executing Playwright tests, ensuring consistent environments across different machines and CI pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}