Integrating INFINI Gateway with Kibana for Enhanced Dashboard Performance

To integrate INFINI Gateway with Kibana for enhanced performance, route all Kibana traffic through the gateway's HTTPS endpoint and implement the cache_first flow—get_cache → elasticsearch → set_cache—to serve repeat dashboard queries from an in-process memory cache, reducing response latency from over 200ms to under 10ms.

Routing Kibana through INFINI Gateway transforms dashboard performance by adding intelligent caching and load balancing between the UI and Elasticsearch. The infinilabs/gateway repository provides a high-performance reverse proxy specifically designed for Elasticsearch ecosystems, offering built-in query caching and TLS termination. This guide covers the recommended approach for integrating INFINI Gateway with Kibana to achieve secure, observable, and low-latency dashboard experiences.

Benefits of Routing Kibana Through INFINI Gateway

Unified Entry Point with Security

All Kibana requests are sent to the gateway instead of directly to Elasticsearch. The gateway handles TLS termination, basic authentication, request-level routing, and load balancing before traffic reaches the Elasticsearch cluster.

Built-in Query Cache

The gateway's cache_first flow stores results of repeat Kibana queries in an in-process cache. This dramatically reduces latency for dashboard panels that issue identical searches, such as time-range visualizations.

Enhanced Observability

The gateway logs every request and response, feeding them into logging pipelines and exposing metrics via the internal API. This provides clear visibility into Kibana-generated load and query patterns.

Architecture Overview

The integration creates a three-tier architecture where Kibana communicates with Elasticsearch exclusively through the gateway:

• Kibana UI sends HTTPS requests to the gateway entry point
• INFINI Gateway processes requests through configured filters and cache layers
• Elasticsearch Cluster receives proxied traffic from the gateway's reverse proxy

Key components implemented in the source code include:

• ReverseProxy (proxy/output/elastic/reverseproxy.go): Creates and refreshes host clients for Elasticsearch nodes, handling node discovery and connection pooling via the refreshNodes logic.
• ProxyConfig (proxy/output/elastic/config.go): Holds per-entry settings including weights, tls, and filter sections that define caching behavior.
• Balancer: Distributes requests across healthy Elasticsearch nodes using the balancer.NewBalancer implementation.
• Filters: Process requests through get_cache, elasticsearch, and set_cache stages.

The request flow for a Kibana search follows this pattern:

1. Kibana sends an HTTPS request to the gateway's configured entry (e.g., 0.0.0.0:8000).
2. The gateway evaluates the flow, first checking get_cache for existing results.
3. On cache miss, the reverse proxy forwards the request to an Elasticsearch node selected by the load balancer.
4. The response passes through set_cache to store results for subsequent identical queries before returning to Kibana.

Step-by-Step Integration Guide

Configure Kibana to Use the Gateway

Edit kibana.yml to point at the gateway instead of Elasticsearch directly:

elasticsearch.hosts: ["https://<gateway-ip>:8000"]
elasticsearch.customHeaders: { "app": "kibana" }

This ensures all Kibana queries travel through the gateway, leveraging the custom header for request identification and routing.

Create a Gateway Entry with Caching

In gateway.yml, define an entry that implements the cache-first flow:

entry:
  - name: kibana_entry
    enabled: true
    router: kibana_router
    max_concurrency: 10000
    network:
      binding: 0.0.0.0:8000
    tls:
      enabled: true
    flow:
      - name: kibana_cache_flow
        filter:
          - get_cache:
              cache_ttl: 30s
              max_cache_items: 200000
          - elasticsearch:
              elasticsearch: prod
              max_connection_per_node: 1000
          - set_cache:
              cache_ttl: 30s

The ProxyConfig struct defined in proxy/output/elastic/config.go parses these parameters to configure the reverse proxy behavior.

Enable Authentication and TLS

Add security layers within the same entry configuration:

          - basic_auth:
              valid_users:
                admin: secret123

Enable TLS under the entry's tls block to encrypt traffic between the browser and gateway, and between gateway and Elasticsearch.

Tune Cache Parameters

Adjust cache_ttl (e.g., 30s) and max_cache_items based on your dashboard's query frequency. A short TTL is sufficient for rapidly refreshed dashboard panels while preventing stale data from serving outdated visualizations.

Verify Cache Performance

Open Kibana, reload a dashboard, and monitor the gateway logs. You should see DEBUG get_cache messages indicating cache hits after the first request, confirming the filter is actively intercepting duplicate queries.

Performance Benefits

• Latency Reduction: After the first execution, identical queries are served from memory, typically dropping response time from over 200ms to under 10ms.
• Cluster Load Decrease: Cache hits avoid hitting Elasticsearch entirely, freeing cluster resources for indexing and complex analytics workloads.
• Scalability: The balancer spreads traffic across all available Elasticsearch nodes. If a node fails, the gateway automatically removes it from the pool via the refreshNodes logic in proxy/output/elastic/reverseproxy.go.

Key Implementation Files

Understanding these source files helps with advanced configuration:

• proxy/output/elastic/reverseproxy.go: Contains the ReverseProxy implementation handling node discovery, load balancing, TLS, and connection pooling.
• proxy/output/elastic/config.go: Defines the ProxyConfig struct with tunable parameters for TLS, caching, and node weights.
• docs/content.en/docs/tutorial/proxy_kibana.md: Official tutorial for TLS and basic authentication setup.
• docs/content.en/docs/references/filters/cache.md: Complete reference for get_cache and set_cache filter options.

Summary

• Route Kibana to INFINI Gateway by updating elasticsearch.hosts in kibana.yml to point to the gateway endpoint.
• Implement the cache-first flow (get_cache → elasticsearch → set_cache) in your gateway entry configuration to accelerate dashboard queries.
• Enable TLS and basic_auth filters in gateway.yml for secure production deployments.
• Tune cache_ttl and max_cache_items based on dashboard refresh rates; 30 seconds is optimal for most real-time dashboards.
• Monitor gateway logs for DEBUG get_cache messages to verify cache hits and performance improvements.

Frequently Asked Questions

How does the cache-first flow improve Kibana dashboard performance?

The cache-first flow intercepts identical queries from Kibana dashboard panels. When a user loads a dashboard, repeated time-range visualizations hit the get_cache filter first. If the query was executed recently (within cache_ttl), the gateway returns the cached result from memory in under 10ms instead of forwarding to Elasticsearch, eliminating network latency and cluster processing time for duplicate requests.

Can I use INFINI Gateway with multiple Elasticsearch clusters behind Kibana?

Yes. The gateway supports routing to different Elasticsearch clusters based on URL patterns or headers. Configure multiple elasticsearch backends in gateway.yml and use router rules to direct specific Kibana spaces or index patterns to different clusters, all while maintaining the caching and authentication layers at the gateway level.

What TLS configuration is required for production Kibana integration?

For production, enable TLS termination at the gateway by setting tls.enabled: true in the entry configuration. The gateway handles HTTPS traffic from Kibana browsers and can use separate TLS settings for upstream Elasticsearch connections. If using self-signed certificates for testing, you may need to adjust Kibana's elasticsearch.ssl.verificationMode setting, though production environments should use valid certificates.

How do I monitor cache hit rates for my Kibana dashboards?

Enable debug logging in the gateway configuration and watch for DEBUG get_cache log entries when dashboard panels refresh. The gateway exposes metrics via its internal API, allowing you to track cache hit ratios and latency improvements. High cache hit rates indicate that your cache_ttl settings align well with dashboard refresh intervals.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}