# How the Easegress AI Gateway Routes Requests to OpenAI and Other LLM Providers

> Discover how the Easegress AI Gateway routes requests to OpenAI and other LLM providers. Learn about unifying APIs, normalizing responses, and tracking token usage with this powerful tool.

- Repository: [MegaEase/easegress](https://github.com/megaease/easegress)
- Tags: how-to-guide
- Published: 2026-03-07

---

**The Easegress AI Gateway unifies OpenAI, Azure, Anthropic, and other LLM providers behind a common Provider interface, using a pluggable registry pattern in Go to route requests, normalize responses, and track token usage across all supported backends.**

The **Easegress AI Gateway** is a cloud-native traffic management layer designed to abstract the complexities of multiple large-language-model (LLM) APIs. By implementing a consistent provider framework in the `megaease/easegress` repository, the gateway enables operators to switch between OpenAI, Azure, Anthropic, and other backends without changing client code.

## Provider Architecture and Registration

The gateway’s extensibility relies on a registry pattern that decouples provider-specific logic from the core request handling pipeline.

### The Provider Interface Abstraction

All LLM backends implement a common **Provider** interface defined in the `aigatewaycontroller` package. This interface standardizes operations such as request preparation, health checking, and token accounting. Concrete implementations embed `BaseProvider` from [`pkg/object/aigatewaycontroller/providers/base.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/base.go), which supplies the majority of the shared HTTP handling logic.

### Registration in the Global Registry

Each provider registers itself during package initialization using the global `ProviderTypeRegistry`. This registration maps a string identifier to a factory function that creates the provider instance.

In [`pkg/object/aigatewaycontroller/providers/openai.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/openai.go), the OpenAI provider registers the type `"openai"`:

```go
func init() {
    providers.RegisterProviderType("openai", func() providers.Provider {
        return &OpenAIProvider{}
    })
}

```

Similarly, the Azure provider registers `"azure"` in [`pkg/object/aigatewaycontroller/providers/azure.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/azure.go) and customizes its base URL construction during initialization.

## How the Easegress AI Gateway Processes OpenAI Requests

When a pipeline receives a request, the gateway orchestrates the flow through the provider’s `Handle` method, which is implemented by `BaseProvider`.

### Request Preparation and Authentication

The `prepareRequest` function in [`pkg/object/aigatewaycontroller/providers/common.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/common.go) constructs the outbound `http.Request`:

1. Copies headers from the incoming Easegress request.
2. Adds the `Authorization: Bearer <APIKey>` header using the key from the provider spec.
3. Appends provider-specific query parameters (e.g., Azure’s `api-version`).

The `RequestMapper` (defined in `BaseProvider`) determines the final request path and body format, defaulting to OpenAI-compatible endpoints.

### Proxying and Response Handling

`BaseProvider.ProxyRequest` forwards the prepared request to the remote LLM service. It reads the response body and stores it in the `aicontext.Response` object. This method handles connection pooling, timeouts, and error propagation back to the client.

### Token Accounting and Usage Tracking

After the remote call succeeds, `BaseProvider.ParseTokens` extracts usage statistics from the provider’s JSON response. It supports:

- Completions and chat completions
- Embeddings
- Image generations
- Streaming responses (extracting the final chunk via `getLastChunkFromOpenAIStream`)

This unified token accounting enables rate limiting and cost tracking across heterogeneous providers.

## Supporting Multiple AI Providers (Azure, Anthropic, and Beyond)

The gateway’s architecture minimizes provider-specific code by leveraging the OpenAI-compatible API standard, while allowing overrides for proprietary formats.

### Azure-Specific URL Construction

The Azure provider in [`pkg/object/aigatewaycontroller/providers/azure.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/azure.go) overrides the `init` method to synthesize a base URL when the spec lacks an explicit `BaseURL`:

```go
func (p *AzureProvider) init(spec *ProviderSpec) error {
    if spec.BaseURL == "" {
        spec.BaseURL = fmt.Sprintf("https://%s.openai.azure.com/openai/deployments/%s", 
            spec.Endpoint, spec.DeploymentID)
    }
    return p.BaseProvider.init(spec)
}

```

This ensures requests route to the correct Azure OpenAI Service deployment without manual URL configuration.

### Anthropic Adapter Pattern

The Anthropic provider implements custom request and response adapters to translate between Claude’s native API format and the OpenAI-compatible format used by `BaseProvider`. The conversion logic resides in [`pkg/object/aigatewaycontroller/aicontext/anthropic.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/aicontext/anthropic.go), mapping Claude’s `messages` and `content` structures to OpenAI’s `chat.completions` schema.

## Configuration Examples

### Pipeline YAML for OpenAI

Define an AI proxy pipeline in Easegress using the `AIProxy` kind:

```yaml
kind: AIProxy
metadata:
  name: openai-proxy
spec:
  provider:
    name: openai
    providerType: openai
    baseURL: https://api.openai.com/v1
    apiKey: ${OPENAI_API_KEY}
    headers:
      "OpenAI-Organization": "my-org"
  routes:
    - path: /v1/chat/completions
      method: POST

```

The gateway parses this spec, validates it via `ValidateSpec`, and instantiates an `OpenAIProvider` to handle matching requests.

### Programmatic Provider Creation

For custom controllers, create providers directly in Go:

```go
import (
    "github.com/megaease/easegress/v2/pkg/object/aigatewaycontroller/aicontext"
    "github.com/megaease/easegress/v2/pkg/object/aigatewaycontroller/providers"
)

spec := &aicontext.ProviderSpec{
    Name:         "my-openai",
    ProviderType: providers.OpenAIProviderType,
    BaseURL:      "https://api.openai.com/v1",
    APIKey:       os.Getenv("OPENAI_API_KEY"),
    Headers: map[string]string{
        "OpenAI-Organization": "my-org",
    },
}

if err := providers.ValidateSpec(spec); err != nil {
    log.Fatalf("invalid spec: %v", err)
}

provider := providers.NewProvider(spec)

```

## Summary

- The **Easegress AI Gateway** uses a provider registry pattern to abstract OpenAI, Azure, Anthropic, and other LLM backends behind a unified interface.
- **Provider registration** occurs at package initialization in files like [`openai.go`](https://github.com/megaease/easegress/blob/main/openai.go) and [`azure.go`](https://github.com/megaease/easegress/blob/main/azure.go), mapping type strings to factory functions.
- **Request handling** is centralized in `BaseProvider` from [`pkg/object/aigatewaycontroller/providers/base.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/base.go), which manages authentication, proxying, and token accounting.
- **Provider-specific customizations** are minimal: Azure overrides URL construction, while Anthropic implements request/response adapters to maintain OpenAI compatibility.
- **Token tracking** works across all providers, including streaming responses, enabling unified rate limiting and cost monitoring.

## Frequently Asked Questions

### What providers does the Easegress AI Gateway support?

The gateway supports OpenAI, Azure OpenAI, Anthropic (Claude), AWS Bedrock, Cohere, DeepSeek, Google Gemini, Mistral, Ollama, Qwen, and any other provider implementing an OpenAI-compatible HTTP API. Each provider registers its type string (e.g., `"openai"`, `"azure"`, `"anthropic"`) in the global `ProviderTypeRegistry` during initialization.

### How does the gateway handle authentication with OpenAI?

Authentication is handled by the `prepareRequest` function in [`pkg/object/aigatewaycontroller/providers/common.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/common.go). It automatically injects the `Authorization: Bearer <APIKey>` header using the API key defined in the provider specification. Additional headers, such as `OpenAI-Organization`, can be configured via the `headers` map in the YAML spec or `ProviderSpec` struct.

### Can I use the Easegress AI Gateway with self-hosted models like Ollama?

Yes. The gateway treats Ollama as a standard provider type that implements the OpenAI-compatible API. You configure it by setting the `providerType` to `"ollama"` and pointing the `baseURL` to your local Ollama instance (e.g., `http://localhost:11434/v1`). The `BaseProvider` logic handles the rest, including health checks against the `/models` endpoint.

### How does token usage tracking work across different providers?

Token accounting is unified through `BaseProvider.ParseTokens` in [`pkg/object/aigatewaycontroller/providers/base.go`](https://github.com/megaease/easegress/blob/main/pkg/object/aigatewaycontroller/providers/base.go). This method extracts `prompt_tokens` and `completion_tokens` from the JSON response body. For streaming responses, the gateway buffers the stream and extracts the final chunk using `getLastChunkFromOpenAIStream` to retrieve usage statistics. This normalized data enables consistent rate limiting and cost monitoring regardless of the backend provider.