How to Run A/B Testing of Different LLM Models in Production with LMForge

LMForge enables A/B testing of different LLM models in production by treating each model as a configurable resource attached to an App through the model_config field, allowing teams to create draft configurations with different providers, publish variants, and route traffic via token-based switching without deploying code changes.

Managing multiple large language models in production requires a systematic approach to compare performance and user satisfaction. The LMForge platform (haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents) provides a built-in mechanism for A/B testing different LLM models by leveraging its app versioning and configuration management system. This architecture allows engineers to test variants like DeepSeek Chat against OpenAI GPT-4 using isolated draft configurations and traffic splitting strategies.

Understanding LMForge's Model Configuration Architecture

LMForge abstracts LLM providers into configurable resources through the App model defined in api/internal/model/app.py. Each application stores its active model settings in a model_config field that specifies the provider, model identifier, and inference parameters.

When you publish a configuration, the platform stores it in the AppConfig table (see the AppConfig model in api/internal/model/app.py around line 183) and maintains a complete history through the AppConfigVersion entity. This versioning system ensures you can maintain parallel draft configurations for A/B testing while preserving the ability to roll back instantly.

The validation logic in api/internal/service/app_service.py within the _validate_draft_app_config method (around line 702) ensures that model identifiers and parameters are correct before any configuration reaches production.

Step-by-Step Workflow for A/B Testing Different LLM Models in Production

1. Define Draft Configurations for Each Variant

Begin by creating separate draft configurations for your control (Variant A) and treatment (Variant B) models. Each draft contains a distinct model_config object specifying different providers or model versions.

from uuid import UUID
from api.internal.service.app_service import AppService

# Assume `app_service` is injected already

app_id = UUID("...")          # the app you want to experiment on

# Draft for Variant A (e.g., DeepSeek Chat)

variant_a = {
    "model_config": {
        "provider": "deepseek",
        "model": "deepseek-chat",
        "parameters": {"temperature": 0.7}
    }
}
app_service.update_draft_app_config(app_id, variant_a, account)

# Draft for Variant B (e.g., OpenAI GPT‑4)

variant_b = {
    "model_config": {
        "provider": "openai",
        "model": "gpt-4",
        "parameters": {"temperature": 0.7}
    }
}
app_service.update_draft_app_config(app_id, variant_b, account)

The _validate_draft_app_config method in api/internal/service/app_service.py automatically validates these configurations against the supported provider registry in api/internal/core/language_model/__init__.py.

2. Publish the Control Variant

Deploy Variant A to production by publishing its draft configuration. This moves the configuration from draft state to the live AppConfig table.

app_service.publish_draft_app_config(app_id, account)

Once published, the app status changes to PUBLISHED and the configuration becomes the default for standard traffic.

3. Configure Traffic Splitting with Tokens

LMForge uses per-app tokens (App.token) stored in the App model to authenticate requests. For A/B testing, generate distinct tokens for each variant and implement routing logic in your load balancer or application layer.

# Generate a distinct token for Variant B (kept as a draft)

token_b = app_service.regenerate_web_app_token(app_id, account)

Route traffic based on token characteristics or headers:

from flask import request, jsonify
from api.internal.service.app_service import AppService

@app.route("/v1/chat")
def chat():
    token = request.headers.get("X-App-Token")
    # Simple 50/50 split for demo purposes

    if int(token[-1], 16) % 2 == 0:   # even → Variant A (published)

        app = app_service.get_published_config(app_id, account)
    else:                             # odd → Variant B (draft)

        app = app_service.get_draft_app_config(app_id, account)

    # Load the appropriate LLM based on the resolved config

    llm = app_service.language_model_service.load_language_model(app["model_config"])
    # ...continue with normal chat flow...

    return jsonify({"reply": llm.invoke(...)})

4. Collect Metrics and Compare Performance

Since all requests traverse the same handler stack—entering through LanguageModelHandler.get_language_models and processing via LanguageModelService—you can instrument logging or Prometheus counters inside language_model_service.load_language_model in api/internal/service/language_model_service.py.

Track latency, token usage, and response quality per variant by tagging metrics with the model identifier from the active model_config.

5. Promote the Winning Model

After statistical significance is reached, promote the winning variant by publishing its draft configuration:

# After the test, promote Variant B

app_service.publish_draft_app_config(app_id, account)

This atomic operation updates the live configuration without service restarts. You may then delete the obsolete draft or retain it for future experiments.

Key Components for LLM A/B Testing

The following files implement the core A/B testing functionality in LMForge:

• api/internal/model/app.py: Defines the App model with model_config, AppConfig storage, and AppConfigVersion history tracking.
• api/internal/service/app_service.py: Contains _validate_draft_app_config for validation, publish_draft_app_config for deployment, and regenerate_web_app_token for routing isolation.
• api/internal/service/language_model_service.py: Implements load_language_model to instantiate concrete BaseLanguageModel instances from configuration.
• api/internal/handler/language_model_handler.py: Exposes provider and model metadata for UI-driven configuration.
• api/internal/schema/app_schema.py: Defines the JSON schema for model_config payloads.
• api/internal/core/language_model/__init__.py: Registers available LLM providers (DeepSeek, OpenAI, etc.).

Summary

• LMForge enables A/B testing of different LLM models in production through draft configurations and token-based routing.
• The model_config field in api/internal/model/app.py abstracts providers, allowing you to switch between DeepSeek, OpenAI, and other models without code changes.
• Use AppService.publish_draft_app_config to atomically promote winning variants while AppConfigVersion maintains complete audit history.
• Traffic splitting leverages per-app tokens (App.token) and regenerate_web_app_token to isolate experimental cohorts.
• All model loading flows through language_model_service.load_language_model, providing a single instrumentation point for comparative metrics.

Frequently Asked Questions

How does token-based routing work for A/B testing in LMForge?

LMForge assigns a unique token (App.token) to each application for authentication. By generating separate tokens for different configuration variants using AppService.regenerate_web_app_token, you can route traffic based on token characteristics in your load balancer or application gateway. This allows you to direct specific user segments to Variant A (published config) or Variant B (draft config) without modifying the core application code.

Can I test more than two LLM models simultaneously?

Yes. LMForge supports multi-variant testing by maintaining multiple draft configurations through update_draft_app_config. Each draft can specify a different model_config with unique provider and model combinations. You would implement custom routing logic that selects the appropriate draft based on user segmentation or random distribution, though only one configuration can be in the published state at a time per application.

How does LMForge validate model configurations before they reach production?

The platform validates all configurations through _validate_draft_app_config in api/internal/service/app_service.py. This method checks that the provider exists in the registry (api/internal/core/language_model/__init__.py), the model identifier is supported, and parameters like temperature and max_tokens fall within acceptable ranges. Validation occurs during draft creation, preventing invalid configurations from being published.

What happens to historical versions during an A/B test?

LMForge maintains immutable version history through the AppConfigVersion model in api/internal/model/app.py. Every publish operation creates a new version record, allowing you to roll back to previous configurations instantly if an A/B test reveals performance degradation. This versioning system ensures that switching between Variant A and Variant B (or reverting to a pre-test baseline) requires only a configuration change, not a code deployment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}