How Desktop Commander MCP's UI A/B Testing System Tracks User Engagement Metrics

Desktop Commander MCP implements a lightweight, deterministic A/B testing framework built on feature-flag infrastructure that hashes persistent client IDs to assign variants, gates UI components via hasFeature(), and reports engagement metrics through a telemetry pipeline to telemetry.desktopcommander.app.

Desktop Commander MCP is an open-source Model Context Protocol (MCP) server for desktop automation. Its UI A/B testing system enables maintainers to experiment with different onboarding flows and interface elements while accurately measuring which variants drive higher user engagement, all through a privacy-respecting pipeline that anonymizes participants via persistent client identifiers.

Deterministic Variant Assignment via Client ID Hashing

The core assignment logic resides in src/utils/ab-test.ts. When the client initializes, the system generates or retrieves a persistent identifier through configManager.getOrCreateClientId(). This client ID serves as the stable bucketing key for deterministic variant selection.

The getVariant() function combines the client ID with the experiment name and computes a hash:

const clientId = await configManager.getOrCreateClientId();
const hash = hashCode(clientId + experimentName);
const roll = hash % totalWeight; // weighted roll
// iterate over variants → first where roll < cumulative weight

This approach ensures that the same user always sees the same variant across sessions, eliminating user-splitting issues that invalidate experiment results. The selected variant is cached in memory (variantCache) and persisted to configuration storage under the key abTest_<experimentName>.

Feature-Flag Integration and Remote Configuration

The A/B system builds upon the featureFlagManager defined in src/utils/feature-flags.ts. At startup, the client fetches remote experiment definitions—including variant weights and names—and caches them locally.

If the cache is stale or unavailable, the system invokes waitForFreshFlags() to block until fresh definitions arrive. This guarantees that UI decisions use the latest experiment parameters. The feature-flag infrastructure also supports exclusion lists (such as welcome_page_excluded_clients) that can filter out specific clients from experiments regardless of their hash-based assignment.

UI Gating and Engagement Measurement

UI components query the A/B system through the hasFeature() function exported from src/utils/ab-test.ts. The welcome onboarding flow in src/utils/welcome-onboarding.ts demonstrates this pattern:

1. Eligibility check: Verify welcomeOnboardingEligible status
2. Pending state validation: Confirm the onboarding flag is still pending
3. Feature-flag readiness: Ensure flags are loaded
4. Exclusion list check: Verify the client is not in welcome_page_excluded_clients
5. A/B decision: Execute await hasFeature('showOnboardingPage')

When hasFeature() returns true, the user enters the treatment variant and sees the welcome page. If false, the user follows the control path. This binary decision point creates a clean separation for measuring engagement differentials between variants.

Telemetry Pipeline and Event Tracking

Every A/B decision and outcome flows through src/utils/capture.ts. The capture() function constructs standardized JSON payloads and transmits them to https://telemetry.desktopcommander.app/mp/collect, but only when configManager.getValue('telemetryEnabled') returns true.

The system emits two critical event types:

• Decision events: capture('server_welcome_page_ab_decision', { variant: 'treatment', loaded_from_cache: loadedFromCache })
• Outcome events: capture('server_welcome_page_opened', { success: true }) or with { success: false, error } on failures

To correlate subsequent user actions with their original assignments, src/utils/ab-test.ts provides getABTestAssignments(). This function iterates over all experiments, reads persisted abTest_... values from the config, and returns a mapping like { ab_OnboardingPreTool: 'showOnboardingPage' }. The telemetry pipeline merges this map into every event, enabling backend analytics to attribute engagement metrics (page-open rates, feature usage) to specific variants.

Implementation Examples

Check whether the current user belongs to a specific variant:

import { hasFeature } from './ab-test.js';

if (await hasFeature('showOnboardingPage')) {
  // Show the welcome page (treatment variant)
}

Manually report an A/B-related telemetry event:

import { capture } from './capture.js';

capture('server_welcome_page_ab_decision', {
  variant: 'treatment',  // or 'control'
  loaded_from_cache: false
});

Retrieve all stored A/B assignments for batch analytics:

import { getABTestAssignments } from './ab-test.js';

const assignments = await getABTestAssignments();
// Example output: { ab_OnboardingPreTool: 'showOnboardingPage' }

Summary

• Deterministic assignment: Uses hashCode(clientId + experimentName) in src/utils/ab-test.ts to ensure consistent variant allocation across sessions.
• Feature-flag foundation: Remote configuration in src/utils/feature-flags.ts drives experiment weights and exclusion lists like welcome_page_excluded_clients.
• UI gating: Components call hasFeature() to branch between treatment and control paths, as implemented in src/utils/welcome-onboarding.ts.
• Privacy-respecting telemetry: Events flow through capture() in src/utils/capture.ts to telemetry.desktopcommander.app, respecting the telemetryEnabled config flag.
• Correlation tracking: getABTestAssignments() aggregates persisted variant data to attribute engagement metrics to specific experiments.

Frequently Asked Questions

How does Desktop Commander MCP ensure users consistently see the same A/B variant?

The system hashes the persistent clientId (retrieved via configManager.getOrCreateClientId()) combined with the experiment name to generate a deterministic roll. This assignment is cached in memory and persisted to config under abTest_<experimentName>, ensuring identical variant selection across application restarts according to the source code in src/utils/ab-test.ts.

What telemetry events does the A/B testing system generate?

The pipeline emits server_welcome_page_ab_decision events when variant assignments occur, and server_welcome_page_opened events tracking success or failure of the UI action. Each payload includes the variant name, cache status, and client ID, enabling precise engagement analysis as implemented in src/utils/capture.ts.

Can specific users be excluded from A/B tests?

Yes. The featureFlagManager supports exclusion lists such as welcome_page_excluded_clients that bypass hash-based assignment. Additionally, users can opt out entirely by disabling telemetryEnabled in the config manager, which prevents capture() from transmitting any data to the telemetry proxy.

How are A/B assignments correlated with long-term user engagement?

The getABTestAssignments() function in src/utils/ab-test.ts collects all persisted variant assignments from config storage. These mappings are attached to every telemetry event via the capture pipeline, allowing the backend analytics system to correlate subsequent feature usage and session duration with the original A/B variant.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}