# How to Run Multiple Airflow Clusters with Different API Versions Simultaneously

> Learn how to run multiple Airflow clusters with different API versions simultaneously by launching separate MCP server instances. Configure each with unique AIRFLOW_API_BASE_URL and AIRFLOW_API_VERSION.

- Repository: [JungJungIn/mcp-airflow-api](https://github.com/call518/mcp-airflow-api)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Launch separate MCP server instances for each Airflow cluster, configuring each with its own `AIRFLOW_API_BASE_URL` and `AIRFLOW_API_VERSION` environment variables.**

The **mcp-airflow-api** package is architected to connect to a single Airflow REST API instance per server process. To manage multiple clusters running different versions—such as Airflow 2.x using the v1 API and Airflow 3.x using the v2 API—you deploy isolated MCP server instances with version-specific configurations.

## Understanding the Version-Aware Architecture

The server dynamically selects the appropriate tool set based on the `AIRFLOW_API_VERSION` environment variable at startup.

### How the Server Selects API Tools

In [`src/mcp_airflow_api/mcp_main.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/mcp_main.py) (lines 241-250), the entry point reads the API version and registers the corresponding tool suite:

```python
api_version = get_api_version()  # Reads AIRFLOW_API_VERSION, defaults to "v1"

if api_version == "v1":
    from mcp_airflow_api.tools import v1_tools
    v1_tools.register_tools(mcp_instance)
elif api_version == "v2":
    from mcp_airflow_api.tools import v2_tools
    v2_tools.register_tools(mcp_instance)

```

The **v1 tool suite** ([`v1_tools.py`](https://github.com/call518/mcp-airflow-api/blob/main/v1_tools.py)) registers the v1-specific request function and common tools compatible with Airflow 2.x. The **v2 tool suite** ([`v2_tools.py`](https://github.com/call518/mcp-airflow-api/blob/main/v2_tools.py)) registers the v2 request function and adds asset-related tools for Airflow 3.x features.

### Environment-Based Configuration

The [`src/mcp_airflow_api/functions.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/functions.py) file handles version detection and URL construction:

- `get_api_version()` (lines 61-73) returns the value of `AIRFLOW_API_VERSION` or defaults to `"v1"`
- `construct_api_url()` (lines 75-80) builds versioned endpoints: `f"{base_url}/{version}{path}"`

This ensures each server instance constructs requests appropriate for its configured Airflow cluster.

## Configuring Multiple MCP Server Instances

To run multiple clusters simultaneously, launch separate processes with distinct environment configurations.

### Command Line Deployment

Run two separate instances from the terminal, each targeting a different cluster:

```bash

# Instance 1 – Airflow 2.x cluster (v1 API)

AIRFLOW_API_BASE_URL=http://airflow-v1.example.com/api \
AIRFLOW_API_VERSION=v1 \
python -m mcp_airflow_api

# Instance 2 – Airflow 3.x cluster (v2 API)

AIRFLOW_API_BASE_URL=http://airflow-v2.example.com/api \
AIRFLOW_API_VERSION=v2 \
python -m mcp_airflow_api

```

Each process logs the version it loaded, confirming isolation:

```

INFO:root:Loading Airflow API v1 tools (Airflow 2.x)
INFO:root:Loading Airflow API v2 tools (Airflow 3.0+)

```

### Docker Compose Setup

For production deployments, use Docker Compose to run isolated containers:

```yaml
version: "3.9"
services:
  mcp-v1:
    image: ghcr.io/call518/mcp-airflow-api:latest
    environment:
      - AIRFLOW_API_BASE_URL=http://airflow-v1/api
      - AIRFLOW_API_VERSION=v1
    ports:
      - "8001:8000"

  mcp-v2:
    image: ghcr.io/call518/mcp-airflow-api:latest
    environment:
      - AIRFLOW_API_BASE_URL=http://airflow-v2/api
      - AIRFLOW_API_VERSION=v2
    ports:
      - "8002:8000"

```

Running `docker compose up -d` starts both servers simultaneously, each bound to a distinct port and Airflow cluster.

### Programmatic Orchestration

To manage instances within a larger Python application:

```python
import os
from subprocess import Popen

def start_mcp(base_url: str, version: str, port: int):
    env = os.environ.copy()
    env.update({
        "AIRFLOW_API_BASE_URL": base_url,
        "AIRFLOW_API_VERSION": version,
        "FASTMCP_TYPE": "streamable-http",
        "FASTMCP_PORT": str(port)
    })
    return Popen(["python", "-m", "mcp_airflow_api"], env=env)

# Launch Airflow 2.x (v1) instance

proc_v1 = start_mcp("http://airflow-v1/api", "v1", 8001)

# Launch Airflow 3.x (v2) instance  

proc_v2 = start_mcp("http://airflow-v2/api", "v2", 8002)

```

Both processes run concurrently with isolated environments, enabling simultaneous management of heterogeneous Airflow deployments.

## Key Implementation Files

| File | Role | Critical Sections |
|------|------|-------------------|
| [`src/mcp_airflow_api/mcp_main.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/mcp_main.py) | Entry point and version selection | Lines 241-250: Version-based tool registration |
| [`src/mcp_airflow_api/functions.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/functions.py) | Environment configuration and URL construction | `get_api_version()` (lines 61-73), `construct_api_url()` (lines 75-80) |
| [`src/mcp_airflow_api/tools/v1_tools.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/tools/v1_tools.py) | v1 API tool registration | Assigns `airflow_request_v1` to common tools |
| [`src/mcp_airflow_api/tools/v2_tools.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/tools/v2_tools.py) | v2 API tool registration | Assigns `airflow_request_v2` and adds asset tools |
| [`src/mcp_airflow_api/tools/common_tools.py`](https://github.com/call518/mcp-airflow-api/blob/main/src/mcp_airflow_api/tools/common_tools.py) | Core tool implementations | 43 shared tools (e.g., `list_dags`, `get_health`) |

## Summary

- The **mcp-airflow-api** server operates against a single Airflow cluster per process, determined by the `AIRFLOW_API_VERSION` environment variable.
- To run multiple Airflow clusters with different API versions simultaneously, launch **separate server instances** with distinct `AIRFLOW_API_BASE_URL` and `AIRFLOW_API_VERSION` configurations.
- The version-aware registration logic in [`mcp_main.py`](https://github.com/call518/mcp-airflow-api/blob/main/mcp_main.py) automatically loads the correct tool suite (v1 or v2) and request functions based on the environment.
- Deployment options include command-line execution, Docker Compose for containerized isolation, or programmatic orchestration using Python subprocesses.

## Frequently Asked Questions

### Can a single MCP server instance connect to multiple Airflow clusters simultaneously?

No. The architecture is designed for one cluster per server process. The `get_api_version()` function in [`functions.py`](https://github.com/call518/mcp-airflow-api/blob/main/functions.py) reads `AIRFLOW_API_VERSION` once at startup, and [`mcp_main.py`](https://github.com/call518/mcp-airflow-api/blob/main/mcp_main.py) registers only the corresponding tool set. To manage multiple clusters, run separate instances with unique environment configurations.

### What happens if I specify an invalid API version?

The server defaults to `"v1"` if `AIRFLOW_API_VERSION` is unset or contains an unrecognized value. This fallback is implemented in `get_api_version()` within [`functions.py`](https://github.com/call518/mcp-airflow-api/blob/main/functions.py). However, tool registration in [`mcp_main.py`](https://github.com/call518/mcp-airflow-api/blob/main/mcp_main.py) explicitly checks for `"v1"` or `"v2"`, so invalid versions may result in no tools being registered or unexpected behavior.

### Do I need different Docker images for v1 and v2 API support?

No. The same `ghcr.io/call518/mcp-airflow-api:latest` image supports both API versions. The version-specific behavior is determined entirely by the `AIRFLOW_API_VERSION` environment variable at runtime. Use the same image with different environment configurations for each cluster.

### How do I handle authentication when running multiple instances?

Each instance manages its own authentication independently via environment variables. Set `AIRFLOW_API_AUTH_TYPE` (Basic or JWT), `AIRFLOW_API_USERNAME`, `AIRFLOW_API_PASSWORD`, or `AIRFLOW_API_TOKEN` uniquely for each instance. The [`functions.py`](https://github.com/call518/mcp-airflow-api/blob/main/functions.py) module reads these per-process, ensuring isolated credentials for each Airflow cluster connection.