DesktopCommanderMCP A/B Testing Infrastructure: How shouldShowMcpUiPreviews Controls Feature Rollouts

DesktopCommanderMCP uses a lightweight client-side A/B testing system driven by remote feature-flag JSON files, where shouldShowMcpUiPreviews deterministically assigns users to variants using a persistent clientId while safely falling back to enabled UI previews on errors.

The DesktopCommanderMCP repository implements a deterministic, remotely configurable A/B testing infrastructure that controls feature exposure without requiring server-side state. This system relies on remote JSON configuration, consistent client-side hashing, and local persistence to manage experiments like the MCP UI Previews rollout. Understanding how shouldShowMcpUiPreviews evaluates user eligibility reveals a robust pattern for client-side feature flagging in TypeScript applications.

How the A/B Testing Infrastructure Works

Remote Feature Flag Management

The feature-flag manager in src/utils/feature-flags.ts drives the entire system by loading JSON configuration from https://desktopcommander.app/flags/v2/production.json. On startup, it immediately reads a cached copy stored locally as feature-flags.json to ensure responsiveness, then fetches a fresh copy in the background to synchronize with remote changes.

The manager exposes two critical methods for downstream consumers:

• wasLoadedFromCache() – indicates whether current flags are from local storage
• waitForFreshFlags() – returns a promise that resolves when the remote fetch completes

This dual-phase loading ensures the application remains functional during startup while eventually converging on the latest remote configuration.

Experiment Configuration Format

Experiments are defined in the remote JSON using a V2 format consumed by src/utils/ab-test.ts. Each experiment specifies weighted variants that determine user assignment probabilities:

{
  "flags": {
    "experiments": {
      "McpUiPreviews": {
        "variants": [
          { "name": "showMCPUi", "weight": 80 },
          { "name": "notShowMCPUi", "weight": 20 }
        ]
      }
    }
  }
}

The client reads these definitions through featureFlagManager.get('experiments') without hardcoding experiment logic, allowing dynamic configuration changes without redeployment.

Deterministic Variant Assignment

The A/B testing infrastructure guarantees consistent user experiences through deterministic variant selection. When a user first encounters an experiment, the system generates a hash from their persistent clientId (stored in config.json) combined with the experiment name.

The selection process in src/utils/ab-test.ts follows these steps:

1. Hash the concatenation of clientId + experimentName
2. Select a variant respecting the configured percentage weights (e.g., 80% vs 20%)
3. Persist the chosen variant under the key abTest_<experimentName> using configManager.setValue
4. Return the same variant on subsequent calls for that user

This approach ensures that refreshing the application or restarting the computer does not change the user's assigned experience, eliminating flickering UI states and maintaining experimental validity.

How shouldShowMcpUiPreviews Evaluates User Eligibility

The shouldShowMcpUiPreviews() function exported from src/utils/mcp-ui-ab-test.ts serves as the decision layer for the MCP UI Previews experiment. Rather than implementing logic directly, it delegates to resolveMcpUiPreviewDecision with injected dependencies for testability:

export async function shouldShowMcpUiPreviews(): Promise<boolean> {
  return resolveMcpUiPreviewDecision({
    getExistingAssignment: () => configManager.getValue(`abTest_${MCP_UI_EXPERIMENT_NAME}`),
    isFirstRun: () => configManager.isFirstRun(),
    wasLoadedFromCache: () => featureFlagManager.wasLoadedFromCache(),
    waitForFreshFlags: () => featureFlagManager.waitForFreshFlags(),
    getABTestVariant,
    capture,
  });
}

The resolution logic implements a three-phase decision tree:

Existing Assignment Lookup

First, the function checks for a previously persisted variant using configManager.getValue. If an assignment exists, it evaluates whether the cached decision came from stale flags. When wasLoadedFromCache() returns true, the system optionally waits for fresh flags via waitForFreshFlags() to determine if remote overrides should replace the local assignment before returning the boolean value (showMCPUi yields true, notShowMCPUi yields false).

First-Run Variant Resolution

For fresh installations where configManager.isFirstRun() returns true, the system fetches the current remote variant before making any UI promises. If the remote variant is recognized (either showMCPUi or notShowMCPUi), the function immediately captures the decision for telemetry using capture('server_mcp_ui_ab_decision', …) and returns the corresponding boolean. If the remote variant is missing or malformed, the UI defaults to enabled previews as a safe fallback.

Error Handling and Telemetry

Any unexpected exceptions during flag retrieval or variant resolution trigger a catch-all fallback returning true, ensuring the MCP UI previews remain visible even during infrastructure failures. This fail-open philosophy prioritizes user experience over experimental rigor when system health is compromised.

Server Integration

The A/B testing decision integrates directly into the MCP server lifecycle in src/server.ts (lines 297-318). When constructing UI metadata for tools requiring previews, the server awaits the boolean result:

const showMcpUiPreviews = await shouldShowMcpUiPreviews();
const uiMetadata = buildUiToolMeta(toolName, toolParams, showMcpUiPreviews);

This integration ensures that tool previews respect the experiment configuration without forcing individual tools to understand A/B testing mechanics.

Practical Implementation Examples

Retrieving a Variant for Any Experiment

import { getABTestVariant } from './utils/ab-test.js';

// Returns "showMCPUi", "notShowMCPUi", or null if unassigned
const variant = await getABTestVariant('McpUiPreviews');

Checking Feature Availability Directly

import { hasFeature } from './utils/ab-test.js';

if (await hasFeature('showMCPUi')) {
  // Execute preview-specific code path
}

Using the MCP UI Preview Helper

import { shouldShowMcpUiPreviews } from './utils/mcp-ui-ab-test.js';

const uiEnabled = await shouldShowMcpUiPreviews();
if (uiEnabled) {
  // Initialize MCP UI preview components
}

Refreshing Feature Flags Manually

import { featureFlagManager } from './utils/feature-flags.js';

// Force immediate remote refresh; resolves when complete
await featureFlagManager.refresh();

Summary

• Remote-first configuration: Feature flags load from https://desktopcommander.app/flags/v2/production.json with local caching in src/utils/feature-flags.ts to balance startup speed with configuration freshness.
• Deterministic assignment: Variants are selected via hash of clientId + experimentName in src/utils/ab-test.ts and persisted in config.json under keys matching abTest_<experimentName>.
• Fail-safe defaults: shouldShowMcpUiPreviews in src/utils/mcp-ui-ab-test.ts returns true on errors and when remote variants are unrecognized, ensuring UI previews remain available during outages.
• First-run telemetry: Fresh installations report their A/B assignment via the capture function, enabling accurate experiment participation tracking without exposing personal data.
• Server-side consumption: The decision integrates at the server level in src/server.ts, passing the boolean to buildUiToolMeta to control preview rendering.

Frequently Asked Questions

How does DesktopCommanderMCP ensure users always see the same A/B test variant?

The system combines a stable clientId from config.json with the experiment name, hashes the concatenation, and caches the resulting variant selection under a persistent key. This deterministic approach prevents variant switching when the application restarts or flags refresh.

What happens if the remote feature flag file cannot be loaded?

If the remote JSON at https://desktopcommander.app/flags/v2/production.json fails to load, the system relies on the locally cached feature-flags.json file. For the specific MCP UI previews experiment, any error or missing configuration causes shouldShowMcpUiPreviews to default to true, ensuring the feature remains visible rather than hidden.

How does the A/B testing infrastructure distinguish between new and returning users?

The configManager.isFirstRun() method identifies fresh installations. During first run, the system waits for fresh remote flags before assigning a variant and reports the decision via telemetry. Returning users immediately load their cached assignment from config.json, skipping the telemetry capture and remote wait unless the cache is stale.

Where are the core A/B testing files located in the repository?

The primary files are src/utils/feature-flags.ts for remote flag fetching, src/utils/ab-test.ts for core assignment logic, and src/utils/mcp-ui-ab-test.ts for the specific preview decision implementation. Integration occurs in src/server.ts, and the full system is validated by test/ab-test.test.js.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}