A/B Testing System for UI Previews in Desktop Commander MCP: Architecture and Implementation

Desktop Commander MCP implements a deterministic, client-side A/B testing framework that assigns users to weighted UI preview variants using consistent hashing of persistent client IDs, ensuring experiment consistency across sessions without requiring external dependencies.

The Desktop Commander MCP repository includes a lightweight A/B testing system specifically designed for UI preview experiments. Built atop the application's feature-flag infrastructure, this system enables developers to gradually roll out new preview interfaces while maintaining experiment consistency across sessions. Understanding this architecture reveals how the application balances flexibility with deterministic user experiences using only local computation and remote configuration.

Three-Layer Architecture

The A/B testing system for UI previews operates through three distinct layers that separate configuration fetching, variant assignment, and UI integration.

Layer 1: Feature-Flag Service

The foundation resides in src/utils/feature-flags.ts, which handles remote configuration management. This service fetches a remote JSON file named production.json containing experiment definitions and weighted variants. It caches these flags locally and refreshes them every 5 minutes to balance freshness with network efficiency. The service exposes three critical APIs: get for retrieving flag values, wasLoadedFromCache for checking data freshness, and waitForFreshFlags for ensuring updated configuration before critical operations.

Layer 2: A/B Test Utility

The core logic lives in src/utils/ab-test.ts. This utility reads the experiments section from the flag payload and assigns each client a deterministic variant using a hash of the persistent client-ID combined with the experiment name. The system supports unequal splits via weight fields in the configuration, calculating assignment by reducing the hash modulo the total weight. Once assigned, the variant persists under configuration keys formatted as abTest_<experimentName>, ensuring the same user always sees the same UI preview across application restarts.

Layer 3: UI Integration Points

Integration occurs throughout the application, particularly in UI components and onboarding logic. Code checks variant assignment via hasFeature('variantName') or getABTestVariant('experiment') to determine which UI preview components to render. The file src/utils/welcome-onboarding.ts demonstrates this pattern by controlling onboarding page flows through A/B test assignments. These integration points also support analytics logging through capture events, enabling data-driven decisions on permanent feature rollout.

How UI Preview Experiments Work

The system follows a five-step pipeline from configuration to rendering:

1. Remote definition – Experiments are defined in the remote flags JSON (v2 format). For example:

   {
     "flags": {
       "experiments": {
         "FilePreviewUI": {
           "variants": [
             { "name": "legacyPreview", "weight": 30 },
             { "name": "newPreview",    "weight": 70 }
           ]
         }
       }
     }
   }

2. Deterministic assignment – When the client first needs a preview, ab-test.ts hashes the client's UUID together with the experiment name. The hash is reduced modulo the total weight to pick a variant, guaranteeing the same user always sees the same UI preview.

3. Persistence – The chosen variant is stored in the config manager under abTest_FilePreviewUI. Subsequent runs read the value directly from src/config-manager.ts, avoiding extra network calls or recomputation.

4. Feature-gate check – UI preview code queries the test utility:

   if (await hasFeature('newPreview')) {
     // render the next‑gen preview UI
   } else {
     // fall back to the legacy preview
   }

5. Analytics exposure – All assignments are exposed via getABTestAssignments() and can be sent with telemetry events, enabling analysis of which UI preview performs better before permanent rollout.

Implementation Code Examples

Fetching the variant for a specific preview experiment:

import { getABTestVariant } from '@/utils/ab-test.js';

const variant = await getABTestVariant('FilePreviewUI');
if (variant === 'newPreview') {
  // Load the modern preview component
} else {
  // Load the legacy component
}

Using the helper hasFeature for conditional rendering:

import { hasFeature } from '@/utils/ab-test.js';

if (await hasFeature('newPreview')) {
  // Enable advanced markdown rendering
}

Retrieving all assignments for analytics tracking:

import { getABTestAssignments } from '@/utils/ab-test.js';

const assignments = await getABTestAssignments();
// → { ab_FilePreviewUI: 'newPreview', ab_OnboardingPreTool: 'showOnboardingPage', … }

Key Files and Responsibilities

File	Role
src/utils/ab-test.ts	Core A/B-test logic – variant selection, caching, and public helpers (getABTestVariant, hasFeature, getABTestAssignments).
src/utils/feature-flags.ts	Remote flag fetching, local caching with 5-minute refresh intervals, and fresh-fetch coordination.
src/config-manager.ts	Persistent storage for client-ID and A/B-test assignment keys (abTest_<experimentName>).
src/utils/welcome-onboarding.ts	Example implementation showing A/B-controlled UI flows (onboarding page variations).

Summary

• Desktop Commander MCP implements a three-layer A/B testing system for UI previews using feature-flags, deterministic hashing, and persistent storage.
• Variant assignment uses consistent hashing of client-ID and experiment name, ensuring users always see the same UI preview across sessions.
• The system supports weighted variants (unequal traffic splits) through configuration in production.json.
• Assignments persist under abTest_<experimentName> keys in the config manager, eliminating the need for repeated remote calls.
• Integration occurs via hasFeature() and getABTestVariant() calls in UI components, with src/utils/welcome-onboarding.ts serving as a reference implementation.

Frequently Asked Questions

How does the system ensure users see the same variant across sessions?

The system calculates a deterministic hash using the user's persistent client-ID (stored in src/config-manager.ts) combined with the experiment name. It reduces this hash modulo the total weight to select a variant, then persists the result under abTest_<experimentName>. Subsequent sessions read this cached value directly, ensuring complete consistency without requiring external state management.

What file controls the experiment weights and variant definitions?

The remote production.json file defines all experiments and their weighted variants. The src/utils/feature-flags.ts service fetches this JSON, which contains an experiments object where each experiment lists variants with name and weight properties. The system refreshes this configuration every 5 minutes while maintaining local cache for offline functionality.

How often does the system refresh experiment configurations from remote?

The feature-flag service in src/utils/feature-flags.ts refreshes the remote configuration every 5 minutes. This interval balances the need for timely experiment updates with network efficiency and offline capability. Developers can use waitForFreshFlags() to force an immediate refresh before critical operations.

Can the A/B testing system handle unequal traffic splits between variants?

Yes, the system explicitly supports unequal splits through the weight field in the JSON configuration. When getABTestVariant processes an experiment, it sums all variant weights and uses the hash modulo this total to determine assignment. This allows configurations like 30/70 splits or any other weighted distribution without requiring code changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}