How the Easegress AI Gateway Routes Requests to OpenAI and Other LLM Providers

The Easegress AI Gateway unifies OpenAI, Azure, Anthropic, and other LLM providers behind a common Provider interface, using a pluggable registry pattern in Go to route requests, normalize responses, and track token usage across all supported backends.

The Easegress AI Gateway is a cloud-native traffic management layer designed to abstract the complexities of multiple large-language-model (LLM) APIs. By implementing a consistent provider framework in the megaease/easegress repository, the gateway enables operators to switch between OpenAI, Azure, Anthropic, and other backends without changing client code.

Provider Architecture and Registration

The gateway’s extensibility relies on a registry pattern that decouples provider-specific logic from the core request handling pipeline.

The Provider Interface Abstraction

All LLM backends implement a common Provider interface defined in the aigatewaycontroller package. This interface standardizes operations such as request preparation, health checking, and token accounting. Concrete implementations embed BaseProvider from pkg/object/aigatewaycontroller/providers/base.go, which supplies the majority of the shared HTTP handling logic.

Registration in the Global Registry

Each provider registers itself during package initialization using the global ProviderTypeRegistry. This registration maps a string identifier to a factory function that creates the provider instance.

In pkg/object/aigatewaycontroller/providers/openai.go, the OpenAI provider registers the type "openai":

func init() {
    providers.RegisterProviderType("openai", func() providers.Provider {
        return &OpenAIProvider{}
    })
}

Similarly, the Azure provider registers "azure" in pkg/object/aigatewaycontroller/providers/azure.go and customizes its base URL construction during initialization.

How the Easegress AI Gateway Processes OpenAI Requests

When a pipeline receives a request, the gateway orchestrates the flow through the provider’s Handle method, which is implemented by BaseProvider.

Request Preparation and Authentication

The prepareRequest function in pkg/object/aigatewaycontroller/providers/common.go constructs the outbound http.Request:

1. Copies headers from the incoming Easegress request.
2. Adds the Authorization: Bearer <APIKey> header using the key from the provider spec.
3. Appends provider-specific query parameters (e.g., Azure’s api-version).

The RequestMapper (defined in BaseProvider) determines the final request path and body format, defaulting to OpenAI-compatible endpoints.

Proxying and Response Handling

BaseProvider.ProxyRequest forwards the prepared request to the remote LLM service. It reads the response body and stores it in the aicontext.Response object. This method handles connection pooling, timeouts, and error propagation back to the client.

Token Accounting and Usage Tracking

After the remote call succeeds, BaseProvider.ParseTokens extracts usage statistics from the provider’s JSON response. It supports:

• Completions and chat completions
• Embeddings
• Image generations
• Streaming responses (extracting the final chunk via getLastChunkFromOpenAIStream)

This unified token accounting enables rate limiting and cost tracking across heterogeneous providers.

Supporting Multiple AI Providers (Azure, Anthropic, and Beyond)

The gateway’s architecture minimizes provider-specific code by leveraging the OpenAI-compatible API standard, while allowing overrides for proprietary formats.

Azure-Specific URL Construction

The Azure provider in pkg/object/aigatewaycontroller/providers/azure.go overrides the init method to synthesize a base URL when the spec lacks an explicit BaseURL:

func (p *AzureProvider) init(spec *ProviderSpec) error {
    if spec.BaseURL == "" {
        spec.BaseURL = fmt.Sprintf("https://%s.openai.azure.com/openai/deployments/%s", 
            spec.Endpoint, spec.DeploymentID)
    }
    return p.BaseProvider.init(spec)
}

This ensures requests route to the correct Azure OpenAI Service deployment without manual URL configuration.

Anthropic Adapter Pattern

The Anthropic provider implements custom request and response adapters to translate between Claude’s native API format and the OpenAI-compatible format used by BaseProvider. The conversion logic resides in pkg/object/aigatewaycontroller/aicontext/anthropic.go, mapping Claude’s messages and content structures to OpenAI’s chat.completions schema.

Configuration Examples

Pipeline YAML for OpenAI

Define an AI proxy pipeline in Easegress using the AIProxy kind:

kind: AIProxy
metadata:
  name: openai-proxy
spec:
  provider:
    name: openai
    providerType: openai
    baseURL: https://api.openai.com/v1
    apiKey: ${OPENAI_API_KEY}
    headers:
      "OpenAI-Organization": "my-org"
  routes:
    - path: /v1/chat/completions
      method: POST

The gateway parses this spec, validates it via ValidateSpec, and instantiates an OpenAIProvider to handle matching requests.

Programmatic Provider Creation

For custom controllers, create providers directly in Go:

import (
    "github.com/megaease/easegress/v2/pkg/object/aigatewaycontroller/aicontext"
    "github.com/megaease/easegress/v2/pkg/object/aigatewaycontroller/providers"
)

spec := &aicontext.ProviderSpec{
    Name:         "my-openai",
    ProviderType: providers.OpenAIProviderType,
    BaseURL:      "https://api.openai.com/v1",
    APIKey:       os.Getenv("OPENAI_API_KEY"),
    Headers: map[string]string{
        "OpenAI-Organization": "my-org",
    },
}

if err := providers.ValidateSpec(spec); err != nil {
    log.Fatalf("invalid spec: %v", err)
}

provider := providers.NewProvider(spec)

Summary

• The Easegress AI Gateway uses a provider registry pattern to abstract OpenAI, Azure, Anthropic, and other LLM backends behind a unified interface.
• Provider registration occurs at package initialization in files like openai.go and azure.go, mapping type strings to factory functions.
• Request handling is centralized in BaseProvider from pkg/object/aigatewaycontroller/providers/base.go, which manages authentication, proxying, and token accounting.
• Provider-specific customizations are minimal: Azure overrides URL construction, while Anthropic implements request/response adapters to maintain OpenAI compatibility.
• Token tracking works across all providers, including streaming responses, enabling unified rate limiting and cost monitoring.

Frequently Asked Questions

What providers does the Easegress AI Gateway support?

The gateway supports OpenAI, Azure OpenAI, Anthropic (Claude), AWS Bedrock, Cohere, DeepSeek, Google Gemini, Mistral, Ollama, Qwen, and any other provider implementing an OpenAI-compatible HTTP API. Each provider registers its type string (e.g., "openai", "azure", "anthropic") in the global ProviderTypeRegistry during initialization.

How does the gateway handle authentication with OpenAI?

Authentication is handled by the prepareRequest function in pkg/object/aigatewaycontroller/providers/common.go. It automatically injects the Authorization: Bearer <APIKey> header using the API key defined in the provider specification. Additional headers, such as OpenAI-Organization, can be configured via the headers map in the YAML spec or ProviderSpec struct.

Can I use the Easegress AI Gateway with self-hosted models like Ollama?

Yes. The gateway treats Ollama as a standard provider type that implements the OpenAI-compatible API. You configure it by setting the providerType to "ollama" and pointing the baseURL to your local Ollama instance (e.g., http://localhost:11434/v1). The BaseProvider logic handles the rest, including health checks against the /models endpoint.

How does token usage tracking work across different providers?

Token accounting is unified through BaseProvider.ParseTokens in pkg/object/aigatewaycontroller/providers/base.go. This method extracts prompt_tokens and completion_tokens from the JSON response body. For streaming responses, the gateway buffers the stream and extracts the final chunk using getLastChunkFromOpenAIStream to retrieve usage statistics. This normalized data enables consistent rate limiting and cost monitoring regardless of the backend provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}