Optimizing INFINI Gateway Performance for High-Throughput Scenarios: 8 Core Strategies

To maximize throughput in INFINI Gateway, scale the bulk-indexing worker pool and HTTP connection limits, tune queue consumer batch sizes, implement weighted load balancing, and adjust rate-limiting thresholds while closely monitoring memory usage and host health detection.

INFINI Gateway acts as a high-performance proxy for Elasticsearch and OpenSearch clusters, designed to handle thousands of queries per second through intelligent request batching and parallel processing. When optimizing INFINI Gateway performance in high-throughput scenarios, understanding the underlying Go implementation and configuration parameters is essential for eliminating bottlenecks. This guide examines the specific source code implementations in infinilabs/gateway that control worker pools, connection management, and flow control to help you achieve maximum throughput.

Scale the Bulk-Indexing Worker Pool

The FastBulkIndexingProcessor is the primary component for aggregating individual index requests into efficient bulk operations. According to the source code in pipeline/fast_bulk_indexing/bulk_indexing.go (lines 66-84), the processor creates a configurable pool of workers controlled by NumOfWorkers (per-shard workers) and MaxWorkers (total concurrent queues).

For high-throughput environments, increase these values proportionally to your CPU count. The optimal setting typically follows the formula: NumOfWorkers = CPU cores × 2-4. This parallelization reduces round-trip latency by grouping many index requests into single bulk calls.

# gateway.yml configuration

fast_bulk_indexing:
  worker_size: 16          # NumOfWorkers

  max_worker_size: 64      # MaxWorkers (total concurrent queues)

Optimize HTTP Connection Pooling

Persistent connections eliminate TCP handshake overhead when communicating with Elasticsearch nodes. In pipeline/fast_bulk_indexing/bulk_indexing.go (lines 77-79), the MaxConnectionPerHost parameter controls the HTTP client pool size passed to elastic.NewBulkProcessor.

Increase this value from the default if your cluster nodes can handle more parallel connections:

fast_bulk_indexing:
  max_connection_per_node: 8   # Increase for high-throughput clusters

Implement Weighted Load Balancing

To prevent hot-spots and distribute traffic evenly across nodes with varying capacities, use the weighted round-robin balancer implemented in proxy/balancer/balancer.go. The balancer accepts a weight slice (ws []int) that reflects relative node capacity.

Configure weights proportional to your node specifications—higher values for nodes with more CPU/RAM:

// weights reflect node capacities: node-A (3), node-B (2), node-C (1)
weights := []int{3, 2, 1}
balancer := balancer.NewBalancer(weights)
hostIdx := balancer.Distribute()
selectedHost := hosts[hostIdx]

Configure Rate Limiting and Back-Pressure

Prevent downstream cluster overload during traffic spikes by tuning the per-path rate limiter in proxy/filters/throttle/request_path_limiter.go (lines 120-132). The implementation uses MaxQPS and Burst parameters to control request flow without choking the queue.

Raise these thresholds for high-traffic API paths:

proxy:
  filters:
    - type: request_path_limiter
      pattern: "^/api/v1/.*"
      max_qps: 2000      # Raise from default 500

      burst: 5000

Tune Queue Consumer Concurrency

When processing from Kafka-style queues, the FlowRunnerProcessor dequeue speed directly impacts latency. In pipeline/flow_runner/flow_runner.go (lines 92-100), the FetchMaxMessages and FetchMaxBytes parameters control batch sizes.

For high-throughput scenarios, increase these values to pull larger chunks from the queue:

flow_runner:
  consumer:
    fetch_max_messages: 2000
    fetch_max_bytes: 52428800   # 50 MiB

Manage Idle Timeouts and Commit Behavior

Balance between durability and performance by configuring flush timers. The IdleTimeoutInSecond parameter in pipeline/fast_bulk_indexing/bulk_indexing.go (lines 61-68) controls when workers flush buffers during low activity, while CommitTimeoutInSeconds in pipeline/flow_runner/flow_runner.go (lines 45-53) manages offset commit frequency.

Tune these to reduce commit chatter under load:

fast_bulk_indexing:
  idle_timeout_in_second: 2     # Flush faster under load

flow_runner:
  commit_timeout_in_seconds: 5  # Reduce commit frequency

Enable Fast Host Health Detection

Rapid failover to healthy nodes prevents request stalls. The elastic.IsHostDead and IsHostAvailable checks in proxy/output/elastic/reverseproxy.go (lines 298-312) combined with rate-limited logging ensure the system remains responsive during node failures.

Ensure your health check interval is sufficiently aggressive (less than 30 seconds) to detect failures quickly without generating excessive log noise.

Memory Management for Bulk Buffers

The gateway maintains a memory-efficient bulk buffer pool (bulkBufferPool) sized up to 1 GiB by default, initialized in pipeline/fast_bulk_indexing/bulk_indexing.go (lines 8-10). While increasing bulk sizes improves throughput, monitor heap usage carefully.

Provision at least 2 GiB of heap for the gateway process on high-throughput nodes, and only increase buffer sizes beyond defaults if you have sufficient RAM to prevent OOM conditions.

Complete Configuration Examples

High-Throughput Bulk Indexing Configuration

fast_bulk_indexing:
  worker_size: 16               # NumOfWorkers per shard

  max_worker_size: 64           # Total concurrent queues

  max_connection_per_node: 8    # HTTP connections per ES node

  bulk:
    bulk_size_in_bytes: 10485760   # 10 MiB per bulk request

    bulk_max_docs_count: 5000      # Doc count limit

  idle_timeout_in_second: 2        # Aggressive flush

  waiting_after: []                # Disable extra wait checks

  pause_when: null                 # No pause-condition in hot path

Load Balancer with Weighted Distribution

package main

import "github.com/infinilabs/gateway/proxy/balancer"

// Create balancer with weights matching node capacity
func createBalancer(hosts []string) *balancer.Balancer {
    weights := []int{3, 2, 1}  // High, medium, low capacity nodes
    return balancer.NewBalancer(weights)
}

Summary

• Scale workers linearly with CPU cores by setting NumOfWorkers to 2-4× core count in bulk_indexing.go
• Increase connection pools using MaxConnectionPerHost to eliminate TCP handshake overhead
• Use weighted balancing via balancer.go to distribute load according to actual node capacity
• Raise rate limits for hot paths in request_path_limiter.go to prevent artificial throttling
• Enlarge consumer batches by increasing FetchMaxMessages and FetchMaxBytes in flow_runner.go
• Tune timers by lowering IdleTimeoutInSecond for faster flushing and raising CommitTimeoutInSeconds to reduce commit frequency
• Monitor health aggressively using the checks in reverseproxy.go to ensure fast failover
• Maintain 2+ GiB heap to accommodate the 1 GiB bulk buffer pool without risking OOM errors

Frequently Asked Questions

What is the optimal number of workers for FastBulkIndexingProcessor?

Set NumOfWorkers to between 2 and 4 times the number of CPU cores available to the gateway process. This configuration, found in pipeline/fast_bulk_indexing/bulk_indexing.go, maximizes parallel bulk processing without excessive goroutine overhead. Monitor CPU utilization and adjust upward if cores remain underutilized during peak load.

How do I prevent memory issues when increasing bulk sizes?

The gateway maintains a 1 GiB bulkBufferPool as defined in pipeline/fast_bulk_indexing/bulk_indexing.go. When increasing BulkSizeInBytes beyond the default 5 MiB, ensure the gateway process has at least 2 GiB of heap memory available. Monitor garbage collection logs and reduce buffer sizes if you observe frequent GC pauses or OOM errors.

What is the difference between IdleTimeoutInSecond and CommitTimeoutInSeconds?

IdleTimeoutInSecond (in bulk_indexing.go) controls how long a bulk worker waits before flushing accumulated documents to Elasticsearch during periods of low activity. CommitTimeoutInSeconds (in flow_runner.go) determines how frequently the consumer commits offsets to Kafka, affecting durability guarantees. Lower the idle timeout for lower latency under load, and raise the commit timeout to reduce I/O overhead.

How does the gateway handle unhealthy Elasticsearch nodes?

The gateway uses elastic.IsHostDead and IsHostAvailable checks in proxy/output/elastic/reverseproxy.go (lines 298-312) to detect node failures. Combined with rate-limited logging via rate.GetRateLimiter, the system quickly marks failed hosts as unavailable and redirects traffic to healthy nodes, preventing request stalls during cluster instabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}