How to Run Multiple Airflow Clusters with Different API Versions Simultaneously

Launch separate MCP server instances for each Airflow cluster, configuring each with its own AIRFLOW_API_BASE_URL and AIRFLOW_API_VERSION environment variables.

The mcp-airflow-api package is architected to connect to a single Airflow REST API instance per server process. To manage multiple clusters running different versions—such as Airflow 2.x using the v1 API and Airflow 3.x using the v2 API—you deploy isolated MCP server instances with version-specific configurations.

Understanding the Version-Aware Architecture

The server dynamically selects the appropriate tool set based on the AIRFLOW_API_VERSION environment variable at startup.

How the Server Selects API Tools

In src/mcp_airflow_api/mcp_main.py (lines 241-250), the entry point reads the API version and registers the corresponding tool suite:

api_version = get_api_version()  # Reads AIRFLOW_API_VERSION, defaults to "v1"

if api_version == "v1":
    from mcp_airflow_api.tools import v1_tools
    v1_tools.register_tools(mcp_instance)
elif api_version == "v2":
    from mcp_airflow_api.tools import v2_tools
    v2_tools.register_tools(mcp_instance)

The v1 tool suite (v1_tools.py) registers the v1-specific request function and common tools compatible with Airflow 2.x. The v2 tool suite (v2_tools.py) registers the v2 request function and adds asset-related tools for Airflow 3.x features.

Environment-Based Configuration

The src/mcp_airflow_api/functions.py file handles version detection and URL construction:

• get_api_version() (lines 61-73) returns the value of AIRFLOW_API_VERSION or defaults to "v1"
• construct_api_url() (lines 75-80) builds versioned endpoints: f"{base_url}/{version}{path}"

This ensures each server instance constructs requests appropriate for its configured Airflow cluster.

Configuring Multiple MCP Server Instances

To run multiple clusters simultaneously, launch separate processes with distinct environment configurations.

Command Line Deployment

Run two separate instances from the terminal, each targeting a different cluster:

# Instance 1 – Airflow 2.x cluster (v1 API)

AIRFLOW_API_BASE_URL=http://airflow-v1.example.com/api \
AIRFLOW_API_VERSION=v1 \
python -m mcp_airflow_api

# Instance 2 – Airflow 3.x cluster (v2 API)

AIRFLOW_API_BASE_URL=http://airflow-v2.example.com/api \
AIRFLOW_API_VERSION=v2 \
python -m mcp_airflow_api

Each process logs the version it loaded, confirming isolation:

INFO:root:Loading Airflow API v1 tools (Airflow 2.x)
INFO:root:Loading Airflow API v2 tools (Airflow 3.0+)

Docker Compose Setup

For production deployments, use Docker Compose to run isolated containers:

version: "3.9"
services:
  mcp-v1:
    image: ghcr.io/call518/mcp-airflow-api:latest
    environment:
      - AIRFLOW_API_BASE_URL=http://airflow-v1/api
      - AIRFLOW_API_VERSION=v1
    ports:
      - "8001:8000"

  mcp-v2:
    image: ghcr.io/call518/mcp-airflow-api:latest
    environment:
      - AIRFLOW_API_BASE_URL=http://airflow-v2/api
      - AIRFLOW_API_VERSION=v2
    ports:
      - "8002:8000"

Running docker compose up -d starts both servers simultaneously, each bound to a distinct port and Airflow cluster.

Programmatic Orchestration

To manage instances within a larger Python application:

import os
from subprocess import Popen

def start_mcp(base_url: str, version: str, port: int):
    env = os.environ.copy()
    env.update({
        "AIRFLOW_API_BASE_URL": base_url,
        "AIRFLOW_API_VERSION": version,
        "FASTMCP_TYPE": "streamable-http",
        "FASTMCP_PORT": str(port)
    })
    return Popen(["python", "-m", "mcp_airflow_api"], env=env)

# Launch Airflow 2.x (v1) instance

proc_v1 = start_mcp("http://airflow-v1/api", "v1", 8001)

# Launch Airflow 3.x (v2) instance  

proc_v2 = start_mcp("http://airflow-v2/api", "v2", 8002)

Both processes run concurrently with isolated environments, enabling simultaneous management of heterogeneous Airflow deployments.

Key Implementation Files

File	Role	Critical Sections
src/mcp_airflow_api/mcp_main.py	Entry point and version selection	Lines 241-250: Version-based tool registration
src/mcp_airflow_api/functions.py	Environment configuration and URL construction	get_api_version() (lines 61-73), construct_api_url() (lines 75-80)
src/mcp_airflow_api/tools/v1_tools.py	v1 API tool registration	Assigns airflow_request_v1 to common tools
src/mcp_airflow_api/tools/v2_tools.py	v2 API tool registration	Assigns airflow_request_v2 and adds asset tools
src/mcp_airflow_api/tools/common_tools.py	Core tool implementations	43 shared tools (e.g., list_dags, get_health)

Summary

• The mcp-airflow-api server operates against a single Airflow cluster per process, determined by the AIRFLOW_API_VERSION environment variable.
• To run multiple Airflow clusters with different API versions simultaneously, launch separate server instances with distinct AIRFLOW_API_BASE_URL and AIRFLOW_API_VERSION configurations.
• The version-aware registration logic in mcp_main.py automatically loads the correct tool suite (v1 or v2) and request functions based on the environment.
• Deployment options include command-line execution, Docker Compose for containerized isolation, or programmatic orchestration using Python subprocesses.

Frequently Asked Questions

Can a single MCP server instance connect to multiple Airflow clusters simultaneously?

No. The architecture is designed for one cluster per server process. The get_api_version() function in functions.py reads AIRFLOW_API_VERSION once at startup, and mcp_main.py registers only the corresponding tool set. To manage multiple clusters, run separate instances with unique environment configurations.

What happens if I specify an invalid API version?

The server defaults to "v1" if AIRFLOW_API_VERSION is unset or contains an unrecognized value. This fallback is implemented in get_api_version() within functions.py. However, tool registration in mcp_main.py explicitly checks for "v1" or "v2", so invalid versions may result in no tools being registered or unexpected behavior.

Do I need different Docker images for v1 and v2 API support?

No. The same ghcr.io/call518/mcp-airflow-api:latest image supports both API versions. The version-specific behavior is determined entirely by the AIRFLOW_API_VERSION environment variable at runtime. Use the same image with different environment configurations for each cluster.

How do I handle authentication when running multiple instances?

Each instance manages its own authentication independently via environment variables. Set AIRFLOW_API_AUTH_TYPE (Basic or JWT), AIRFLOW_API_USERNAME, AIRFLOW_API_PASSWORD, or AIRFLOW_API_TOKEN uniquely for each instance. The functions.py module reads these per-process, ensuring isolated credentials for each Airflow cluster connection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}