MCP-UI A/B Testing System in Desktop Commander MCP: How It Controls UI Previews

The MCP-UI A/B testing system is a lightweight framework that uses deterministic hashing and remote feature flags to decide whether users see the new MCP-UI preview or the legacy interface.

The Desktop Commander MCP repository implements a sophisticated gating mechanism to gradually roll out its new interface. The MCP-UI A/B testing system determines which users receive the updated preview experience based on weighted variants and persistent assignment logic defined in the source code.

How Experiment Definitions Drive the MCP-UI A/B Testing System

Experiments are defined remotely in a feature-flags JSON configuration managed by featureFlagManager. Each experiment lists weighted variants—such as showMCPUi versus notShowMCPUi—that control the probability of assignment.

In src/utils/ab-test.ts (lines 7-16), the system loads these configurations to determine available variants and their respective weights before making any assignment decisions.

Variant Assignment Logic in ab-test.ts

The core assignment engine lives in src/utils/ab-test.ts. When getABTestVariant() is called, the system executes a deterministic selection process:

1. Configuration Loading: Fetches the experiment definition from remote feature flags.
2. Persistence Check: Looks for an existing assignment via configManager.getValue('abTest_<experiment>').
3. Deterministic Hashing: If no assignment exists, hashes the client-ID together with the experiment name to select a variant respecting the defined weights (lines 48-92).
4. Caching: Stores the chosen variant both for the current session and in persistent configuration for future runs (lines 94-96).

This approach ensures that the same user always sees the same variant across sessions while maintaining statistical balance according to the configured weights.

Decision Wrapper for MCP-UI Previews

The file src/utils/mcp-ui-ab-test.ts provides a higher-level abstraction through resolveMcpUiPreviewDecision(). This function combines the raw A/B test result with additional contextual factors (lines 25-66):

• Existing Assignment: Checks configManager.getValue('abTest_McpUiPreviews') for previous decisions.
• First-Run Detection: Considers whether the application is launching for the first time.
• Data Freshness: Evaluates whether feature-flag data was loaded from cache or freshly fetched from featureFlagManager.
• Telemetry Capture: Records the decision via the capture utility for analytics purposes.

The function returns a boolean: true when the UI preview should be displayed and false to retain the legacy interface.

Public API for UI Components

shouldShowMcpUiPreviews() serves as the exported entry point used throughout the codebase. Located in src/utils/mcp-ui-ab-test.ts (lines 68-77), this function forwards dependency implementations to the decision resolver, providing a clean interface for UI components.

// Example: Using the decision in a UI component
import { shouldShowMcpUiPreviews } from '../utils/mcp-ui-ab-test.js';

async function maybeRenderPreview() {
  const showPreview = await shouldShowMcpUiPreviews();
  if (showPreview) {
    // Load the new MCP-UI preview
    import('./McpUiPreview.js').then(mod => mod.render());
  } else {
    // Fallback to the older UI
    import('./LegacyUi.js').then(mod => mod.render());
  }
}

Role in Controlled UI Preview Rollouts

Components rendering the new MCP-UI preview call shouldShowMcpUiPreviews() to determine which interface to load. When the function resolves to true, the preview UI renders; otherwise, the system maintains the legacy UI.

This gating mechanism allows the Desktop Commander MCP team to:

• Roll out the preview gradually to specific user segments
• Collect real-world usage data through the telemetry system
• Revert or modify exposure without deploying new code
• Maintain consistent user experiences through persistent assignment storage

// Directly querying a variant (useful for analytics)
import { getABTestVariant } from '../utils/ab-test.js';

async function logCurrentVariant() {
  const variant = await getABTestVariant('McpUiPreviews');
  console.log('MCP-UI variant:', variant);
}

Summary

• The MCP-UI A/B testing system uses remote feature-flag configurations defined in featureFlagManager to establish weighted experiment variants.
• Variant assignment occurs in src/utils/ab-test.ts through deterministic hashing of client-ID and experiment names, ensuring consistent user experiences.
• The resolveMcpUiPreviewDecision() function in src/utils/mcp-ui-ab-test.ts combines A/B test results with contextual data like first-run status and cache freshness.
• shouldShowMcpUiPreviews() provides the public API that returns boolean values to gate access to the new UI preview.
• Persistent storage via configManager maintains variant assignments across sessions, while telemetry capture enables data-driven rollout decisions.

Frequently Asked Questions

How does the MCP-UI A/B testing system ensure users see the same variant across sessions?

The system persists variant assignments using configManager.getValue('abTest_<experiment>') and configManager.setValue(). When a user first receives an assignment, the variant name is stored in the configuration. Subsequent sessions check this persistent storage before calculating new assignments, ensuring consistency across application restarts.

What determines which variant a user receives in the MCP-UI A/B testing system?

If no prior assignment exists, the system deterministically hashes the user's client-ID combined with the experiment name. This hash selects a variant based on the weights defined in the remote feature-flags JSON. The deterministic approach ensures that the same user always receives the same variant while maintaining the statistical distribution specified by the weights.

Can the MCP-UI A/B testing system work offline or with cached feature flags?

Yes. The resolveMcpUiPreviewDecision() function specifically checks whether feature-flag data was loaded from cache or freshly fetched. The system can make assignment decisions using cached configurations, though it records the data freshness status in telemetry to distinguish between online and offline assignment contexts.

How does the MCP-UI A/B testing system support gradual rollbacks?

Because the gating logic depends on remote feature-flag configurations rather than hardcoded values, the team can modify variant weights or disable experiments entirely by updating the remote JSON. The featureFlagManager loads these changes, and subsequent calls to shouldShowMcpUiPreviews() reflect the new configuration without requiring a code deployment or application update.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}