# How to Run A/B Testing of Different LLM Models in Production with LMForge

> Discover how LMForge streamlines A/B testing of LLM models in production. Effortlessly compare model performance by routing traffic without code deployments. Learn more today.

- Repository: [Haohao/lmforge-end-to-end-llmops-platform-for-multi-model-agents](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents)
- Tags: how-to-guide
- Published: 2026-03-03

---

**LMForge enables A/B testing of different LLM models in production by treating each model as a configurable resource attached to an App through the `model_config` field, allowing teams to create draft configurations with different providers, publish variants, and route traffic via token-based switching without deploying code changes.**

Managing multiple large language models in production requires a systematic approach to compare performance and user satisfaction. The LMForge platform (haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents) provides a built-in mechanism for A/B testing different LLM models by leveraging its app versioning and configuration management system. This architecture allows engineers to test variants like DeepSeek Chat against OpenAI GPT-4 using isolated draft configurations and traffic splitting strategies.

## Understanding LMForge's Model Configuration Architecture

LMForge abstracts LLM providers into configurable resources through the `App` model defined in [`api/internal/model/app.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/model/app.py). Each application stores its active model settings in a `model_config` field that specifies the provider, model identifier, and inference parameters.

When you publish a configuration, the platform stores it in the `AppConfig` table (see the `AppConfig` model in [`api/internal/model/app.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/model/app.py) around line 183) and maintains a complete history through the `AppConfigVersion` entity. This versioning system ensures you can maintain parallel draft configurations for A/B testing while preserving the ability to roll back instantly.

The validation logic in [`api/internal/service/app_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/app_service.py) within the `_validate_draft_app_config` method (around line 702) ensures that model identifiers and parameters are correct before any configuration reaches production.

## Step-by-Step Workflow for A/B Testing Different LLM Models in Production

### 1. Define Draft Configurations for Each Variant

Begin by creating separate draft configurations for your control (Variant A) and treatment (Variant B) models. Each draft contains a distinct `model_config` object specifying different providers or model versions.

```python
from uuid import UUID
from api.internal.service.app_service import AppService

# Assume `app_service` is injected already

app_id = UUID("...")          # the app you want to experiment on

# Draft for Variant A (e.g., DeepSeek Chat)

variant_a = {
    "model_config": {
        "provider": "deepseek",
        "model": "deepseek-chat",
        "parameters": {"temperature": 0.7}
    }
}
app_service.update_draft_app_config(app_id, variant_a, account)

# Draft for Variant B (e.g., OpenAI GPT‑4)

variant_b = {
    "model_config": {
        "provider": "openai",
        "model": "gpt-4",
        "parameters": {"temperature": 0.7}
    }
}
app_service.update_draft_app_config(app_id, variant_b, account)

```

The `_validate_draft_app_config` method in [`api/internal/service/app_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/app_service.py) automatically validates these configurations against the supported provider registry in [`api/internal/core/language_model/__init__.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/core/language_model/__init__.py).

### 2. Publish the Control Variant

Deploy Variant A to production by publishing its draft configuration. This moves the configuration from draft state to the live `AppConfig` table.

```python
app_service.publish_draft_app_config(app_id, account)

```

Once published, the app status changes to `PUBLISHED` and the configuration becomes the default for standard traffic.

### 3. Configure Traffic Splitting with Tokens

LMForge uses per-app tokens (`App.token`) stored in the `App` model to authenticate requests. For A/B testing, generate distinct tokens for each variant and implement routing logic in your load balancer or application layer.

```python

# Generate a distinct token for Variant B (kept as a draft)

token_b = app_service.regenerate_web_app_token(app_id, account)

```

Route traffic based on token characteristics or headers:

```python
from flask import request, jsonify
from api.internal.service.app_service import AppService

@app.route("/v1/chat")
def chat():
    token = request.headers.get("X-App-Token")
    # Simple 50/50 split for demo purposes

    if int(token[-1], 16) % 2 == 0:   # even → Variant A (published)

        app = app_service.get_published_config(app_id, account)
    else:                             # odd → Variant B (draft)

        app = app_service.get_draft_app_config(app_id, account)

    # Load the appropriate LLM based on the resolved config

    llm = app_service.language_model_service.load_language_model(app["model_config"])
    # ...continue with normal chat flow...

    return jsonify({"reply": llm.invoke(...)})

```

### 4. Collect Metrics and Compare Performance

Since all requests traverse the same handler stack—entering through `LanguageModelHandler.get_language_models` and processing via `LanguageModelService`—you can instrument logging or Prometheus counters inside `language_model_service.load_language_model` in [`api/internal/service/language_model_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/language_model_service.py).

Track latency, token usage, and response quality per variant by tagging metrics with the model identifier from the active `model_config`.

### 5. Promote the Winning Model

After statistical significance is reached, promote the winning variant by publishing its draft configuration:

```python

# After the test, promote Variant B

app_service.publish_draft_app_config(app_id, account)

```

This atomic operation updates the live configuration without service restarts. You may then delete the obsolete draft or retain it for future experiments.

## Key Components for LLM A/B Testing

The following files implement the core A/B testing functionality in LMForge:

- **[`api/internal/model/app.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/model/app.py)**: Defines the `App` model with `model_config`, `AppConfig` storage, and `AppConfigVersion` history tracking.
- **[`api/internal/service/app_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/app_service.py)**: Contains `_validate_draft_app_config` for validation, `publish_draft_app_config` for deployment, and `regenerate_web_app_token` for routing isolation.
- **[`api/internal/service/language_model_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/language_model_service.py)**: Implements `load_language_model` to instantiate concrete `BaseLanguageModel` instances from configuration.
- **[`api/internal/handler/language_model_handler.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/handler/language_model_handler.py)**: Exposes provider and model metadata for UI-driven configuration.
- **[`api/internal/schema/app_schema.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/schema/app_schema.py)**: Defines the JSON schema for `model_config` payloads.
- **[`api/internal/core/language_model/__init__.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/core/language_model/__init__.py)**: Registers available LLM providers (DeepSeek, OpenAI, etc.).

## Summary

- LMForge enables **A/B testing of different LLM models in production** through draft configurations and token-based routing.
- The `model_config` field in [`api/internal/model/app.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/model/app.py) abstracts providers, allowing you to switch between DeepSeek, OpenAI, and other models without code changes.
- Use `AppService.publish_draft_app_config` to atomically promote winning variants while `AppConfigVersion` maintains complete audit history.
- Traffic splitting leverages per-app tokens (`App.token`) and `regenerate_web_app_token` to isolate experimental cohorts.
- All model loading flows through `language_model_service.load_language_model`, providing a single instrumentation point for comparative metrics.

## Frequently Asked Questions

### How does token-based routing work for A/B testing in LMForge?

LMForge assigns a unique token (`App.token`) to each application for authentication. By generating separate tokens for different configuration variants using `AppService.regenerate_web_app_token`, you can route traffic based on token characteristics in your load balancer or application gateway. This allows you to direct specific user segments to Variant A (published config) or Variant B (draft config) without modifying the core application code.

### Can I test more than two LLM models simultaneously?

Yes. LMForge supports multi-variant testing by maintaining multiple draft configurations through `update_draft_app_config`. Each draft can specify a different `model_config` with unique provider and model combinations. You would implement custom routing logic that selects the appropriate draft based on user segmentation or random distribution, though only one configuration can be in the published state at a time per application.

### How does LMForge validate model configurations before they reach production?

The platform validates all configurations through `_validate_draft_app_config` in [`api/internal/service/app_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/app_service.py). This method checks that the provider exists in the registry ([`api/internal/core/language_model/__init__.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/core/language_model/__init__.py)), the model identifier is supported, and parameters like temperature and max_tokens fall within acceptable ranges. Validation occurs during draft creation, preventing invalid configurations from being published.

### What happens to historical versions during an A/B test?

LMForge maintains immutable version history through the `AppConfigVersion` model in [`api/internal/model/app.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/model/app.py). Every publish operation creates a new version record, allowing you to roll back to previous configurations instantly if an A/B test reveals performance degradation. This versioning system ensures that switching between Variant A and Variant B (or reverting to a pre-test baseline) requires only a configuration change, not a code deployment.