# How Desktop Commander MCP's UI A/B Testing System Tracks User Engagement Metrics

> Discover how Desktop Commander MCP's UI A/B testing system tracks user engagement metrics using feature flags and deterministic variant assignment. Understand your users better.

- Repository: [Eduard Ruzga/DesktopCommanderMCP](https://github.com/wonderwhy-er/DesktopCommanderMCP)
- Tags: performance
- Published: 2026-07-29

---

**Desktop Commander MCP implements a lightweight, deterministic A/B testing framework built on feature-flag infrastructure that hashes persistent client IDs to assign variants, gates UI components via `hasFeature()`, and reports engagement metrics through a telemetry pipeline to `telemetry.desktopcommander.app`.**

Desktop Commander MCP is an open-source Model Context Protocol (MCP) server for desktop automation. Its UI A/B testing system enables maintainers to experiment with different onboarding flows and interface elements while accurately measuring which variants drive higher user engagement, all through a privacy-respecting pipeline that anonymizes participants via persistent client identifiers.

## Deterministic Variant Assignment via Client ID Hashing

The core assignment logic resides in [`src/utils/ab-test.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/ab-test.ts). When the client initializes, the system generates or retrieves a persistent identifier through `configManager.getOrCreateClientId()`. This client ID serves as the stable bucketing key for deterministic variant selection.

The `getVariant()` function combines the client ID with the experiment name and computes a hash:

```typescript
const clientId = await configManager.getOrCreateClientId();
const hash = hashCode(clientId + experimentName);
const roll = hash % totalWeight; // weighted roll
// iterate over variants → first where roll < cumulative weight

```

This approach ensures that the same user always sees the same variant across sessions, eliminating user-splitting issues that invalidate experiment results. The selected variant is cached in memory (`variantCache`) and persisted to configuration storage under the key `abTest_<experimentName>`.

## Feature-Flag Integration and Remote Configuration

The A/B system builds upon the `featureFlagManager` defined in [`src/utils/feature-flags.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/feature-flags.ts). At startup, the client fetches remote experiment definitions—including variant weights and names—and caches them locally.

If the cache is stale or unavailable, the system invokes `waitForFreshFlags()` to block until fresh definitions arrive. This guarantees that UI decisions use the latest experiment parameters. The feature-flag infrastructure also supports exclusion lists (such as `welcome_page_excluded_clients`) that can filter out specific clients from experiments regardless of their hash-based assignment.

## UI Gating and Engagement Measurement

UI components query the A/B system through the `hasFeature()` function exported from [`src/utils/ab-test.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/ab-test.ts). The welcome onboarding flow in [`src/utils/welcome-onboarding.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/welcome-onboarding.ts) demonstrates this pattern:

1. **Eligibility check**: Verify `welcomeOnboardingEligible` status
2. **Pending state validation**: Confirm the onboarding flag is still pending  
3. **Feature-flag readiness**: Ensure flags are loaded
4. **Exclusion list check**: Verify the client is not in `welcome_page_excluded_clients`
5. **A/B decision**: Execute `await hasFeature('showOnboardingPage')`

When `hasFeature()` returns `true`, the user enters the **treatment** variant and sees the welcome page. If `false`, the user follows the **control** path. This binary decision point creates a clean separation for measuring engagement differentials between variants.

## Telemetry Pipeline and Event Tracking

Every A/B decision and outcome flows through [`src/utils/capture.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/capture.ts). The `capture()` function constructs standardized JSON payloads and transmits them to `https://telemetry.desktopcommander.app/mp/collect`, but only when `configManager.getValue('telemetryEnabled')` returns true.

The system emits two critical event types:

- **Decision events**: `capture('server_welcome_page_ab_decision', { variant: 'treatment', loaded_from_cache: loadedFromCache })`
- **Outcome events**: `capture('server_welcome_page_opened', { success: true })` or with `{ success: false, error }` on failures

To correlate subsequent user actions with their original assignments, [`src/utils/ab-test.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/ab-test.ts) provides `getABTestAssignments()`. This function iterates over all experiments, reads persisted `abTest_...` values from the config, and returns a mapping like `{ ab_OnboardingPreTool: 'showOnboardingPage' }`. The telemetry pipeline merges this map into every event, enabling backend analytics to attribute engagement metrics (page-open rates, feature usage) to specific variants.

## Implementation Examples

Check whether the current user belongs to a specific variant:

```typescript
import { hasFeature } from './ab-test.js';

if (await hasFeature('showOnboardingPage')) {
  // Show the welcome page (treatment variant)
}

```

Manually report an A/B-related telemetry event:

```typescript
import { capture } from './capture.js';

capture('server_welcome_page_ab_decision', {
  variant: 'treatment',  // or 'control'
  loaded_from_cache: false
});

```

Retrieve all stored A/B assignments for batch analytics:

```typescript
import { getABTestAssignments } from './ab-test.js';

const assignments = await getABTestAssignments();
// Example output: { ab_OnboardingPreTool: 'showOnboardingPage' }

```

## Summary

- **Deterministic assignment**: Uses `hashCode(clientId + experimentName)` in [`src/utils/ab-test.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/ab-test.ts) to ensure consistent variant allocation across sessions.
- **Feature-flag foundation**: Remote configuration in [`src/utils/feature-flags.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/feature-flags.ts) drives experiment weights and exclusion lists like `welcome_page_excluded_clients`.
- **UI gating**: Components call `hasFeature()` to branch between treatment and control paths, as implemented in [`src/utils/welcome-onboarding.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/welcome-onboarding.ts).
- **Privacy-respecting telemetry**: Events flow through `capture()` in [`src/utils/capture.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/capture.ts) to `telemetry.desktopcommander.app`, respecting the `telemetryEnabled` config flag.
- **Correlation tracking**: `getABTestAssignments()` aggregates persisted variant data to attribute engagement metrics to specific experiments.

## Frequently Asked Questions

### How does Desktop Commander MCP ensure users consistently see the same A/B variant?

The system hashes the persistent `clientId` (retrieved via `configManager.getOrCreateClientId()`) combined with the experiment name to generate a deterministic roll. This assignment is cached in memory and persisted to config under `abTest_<experimentName>`, ensuring identical variant selection across application restarts according to the source code in [`src/utils/ab-test.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/ab-test.ts).

### What telemetry events does the A/B testing system generate?

The pipeline emits `server_welcome_page_ab_decision` events when variant assignments occur, and `server_welcome_page_opened` events tracking success or failure of the UI action. Each payload includes the variant name, cache status, and client ID, enabling precise engagement analysis as implemented in [`src/utils/capture.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/capture.ts).

### Can specific users be excluded from A/B tests?

Yes. The `featureFlagManager` supports exclusion lists such as `welcome_page_excluded_clients` that bypass hash-based assignment. Additionally, users can opt out entirely by disabling `telemetryEnabled` in the config manager, which prevents `capture()` from transmitting any data to the telemetry proxy.

### How are A/B assignments correlated with long-term user engagement?

The `getABTestAssignments()` function in [`src/utils/ab-test.ts`](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/utils/ab-test.ts) collects all persisted variant assignments from config storage. These mappings are attached to every telemetry event via the capture pipeline, allowing the backend analytics system to correlate subsequent feature usage and session duration with the original A/B variant.