How to Implement Cross-Cluster Routing in INFINI Gateway Using Index Patterns

INFINI Gateway implements cross-cluster routing by evaluating index patterns against declarative rules in the routing.switch filter, then forwarding requests to the appropriate Elasticsearch cluster via the X-Backend-Cluster header.

Cross-cluster routing in INFINI Gateway enables transparent request forwarding to different Elasticsearch clusters based on matching index patterns. The open-source proxy, maintained in the infinilabs/gateway repository, achieves this through a declarative configuration that maps regex or prefix patterns to logical cluster identifiers without requiring custom code modifications.

How Cross-Cluster Routing Works

Core Components

Three components handle the routing logic according to the source code:

• Routing filter (switch) – Evaluates index patterns and selects the target cluster. Located in proxy/filters/routing/switch.go.
• Elasticsearch lookup filter – Performs secondary queries against the routed cluster using specific index patterns. Located in proxy/filters/transform/elasticsearch_lookup.go.
• Reverse proxy – Adds the X-Backend-Cluster header and resolves the logical cluster to a physical host. Located in proxy/output/elastic/reverseproxy.go.

Request Flow

The routing process follows these steps:

1. URI Parsing – The gateway extracts the index name from the request URI (e.g., my-logs-2023 from /my-logs-2023/_search).
2. Pattern Matching – The routing.switch filter compares the index against configured rules using util.MatchPattern().
3. Context Storage – Upon matching, the filter stores the logical cluster name in the request context via ctx.SetUserValue("target_cluster", rule.Cluster) and sets the X-Backend-Cluster header.
4. Proxy Resolution – The reverse proxy reads the header (line 70 in reverseproxy.go) and routes to the concrete Elasticsearch endpoint defined in the clusters configuration.

Configuring Index Pattern Routing

Defining Routing Rules

Configure the routing.switch filter in your gateway.yml to map index patterns to cluster names:

filters:
  - name: routing.switch
    rules:
      - pattern: "^logs-.*"
        cluster: "cluster_a"
      - pattern: "^metrics-.*"
        cluster: "cluster_b"

Rules evaluate in order; the first match determines the target cluster.

Mapping Logical Clusters to Physical Endpoints

Define the actual Elasticsearch endpoints in the clusters section:

clusters:
  cluster_a:
    elasticsearch: "http://es-a.example.com:9200"
    balancer: roundrobin
  cluster_b:
    elasticsearch: "http://es-b.example.com:9200"
    balancer: roundrobin

The reverse proxy resolves the logical name assigned by the routing filter to these physical URLs defined in ProxyConfig.Elasticsearch.

Advanced Routing with ElasticsearchLookup

For secondary queries against the routed cluster, use the ElasticsearchLookup filter. This component respects the cluster selected by the routing logic while allowing you to specify a different index pattern for the lookup:

filters:
  - name: elasticsearch_lookup
    target:
      elasticsearch: "cluster_a"  # Inherits from routing decision

      index_pattern: "logs-*-raw"
      template:
        method: "POST"
        body: |
          {
            "size": 0,
            "aggs": {
              "hit_docs": {
                "terms": { "field": "{{JOIN_BY_FIELD_ARRAY_VALUES}}" }
              }
            }
          }

The target.elasticsearch field references the logical cluster name, while target.index_pattern defines which indices to query within that cluster.

Implementation Details

The core routing logic in proxy/filters/routing/switch.go extracts the index and evaluates patterns:

func (f *Switch) Filter(ctx *fasthttp.RequestCtx) {
    uri := ctx.Request.URI()
    parts := strings.Split(strings.Trim(uri.Path(), "/"), "/")
    if len(parts) < 1 {
        return
    }
    index := parts[0]

    for _, rule := range f.Config.Rules {
        if util.MatchPattern(rule.Pattern, index) {
            ctx.SetUserValue("target_cluster", rule.Cluster)
            ctx.Response.Header.Set("X-Backend-Cluster", rule.Cluster)
            break
        }
    }
}

The reverse proxy in proxy/output/elastic/reverseproxy.go consumes this header at line 70 to select the backend host from the clusters configuration or ProxyConfig.Elasticsearch.

Summary

• Cross-cluster routing relies on the routing.switch filter to evaluate index patterns against regex or prefix rules.
• The filter stores the target cluster in the request context and sets the X-Backend-Cluster header for downstream processing.
• Physical cluster endpoints are configured separately in the clusters section, enabling logical-to-physical resolution.
• The ElasticsearchLookup filter supports secondary queries against the routed cluster using distinct index patterns.
• All routing logic is declarative; no code changes are required to add or modify routing rules.

Frequently Asked Questions

How does INFINI Gateway determine which cluster to route a request to?

The gateway extracts the index name from the request URI and evaluates it against ordered rules in the routing.switch filter configuration. The first matching pattern determines the logical cluster name, which is stored in the request context and passed via the X-Backend-Cluster header to the reverse proxy for physical resolution.

Can I use regular expressions for index pattern matching?

Yes, the routing.switch filter supports regular expressions in the pattern field. The implementation uses util.MatchPattern() in proxy/filters/routing/switch.go to evaluate both regex and prefix matches against the extracted index name.

What happens if no routing rule matches the index pattern?

If no rules match, the request proceeds to the default cluster configured in proxy.elasticsearch or fails based on your specific configuration. The routing filter only overrides the destination when an explicit pattern matches the incoming index.

How does ElasticsearchLookup interact with the routing decision?

The ElasticsearchLookup filter references the logical cluster name selected by the routing filter through its target.elasticsearch configuration. It then executes queries against that same cluster using the target.index_pattern to specify which indices to search, enabling cross-cluster data enrichment while maintaining the routing context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}