# Optimizing INFINI Gateway Performance for High-Throughput Scenarios: 8 Core Strategies

> Boost INFINI Gateway performance in high-throughput scenarios with 8 core strategies. Optimize worker pools, connections, batch sizes, load balancing, and rate limits.

- Repository: [INFINI Labs/gateway](https://github.com/infinilabs/gateway)
- Tags: best-practices
- Published: 2026-03-04

---

**To maximize throughput in INFINI Gateway, scale the bulk-indexing worker pool and HTTP connection limits, tune queue consumer batch sizes, implement weighted load balancing, and adjust rate-limiting thresholds while closely monitoring memory usage and host health detection.**

INFINI Gateway acts as a high-performance proxy for Elasticsearch and OpenSearch clusters, designed to handle thousands of queries per second through intelligent request batching and parallel processing. When optimizing INFINI Gateway performance in high-throughput scenarios, understanding the underlying Go implementation and configuration parameters is essential for eliminating bottlenecks. This guide examines the specific source code implementations in `infinilabs/gateway` that control worker pools, connection management, and flow control to help you achieve maximum throughput.

## Scale the Bulk-Indexing Worker Pool

The `FastBulkIndexingProcessor` is the primary component for aggregating individual index requests into efficient bulk operations. According to the source code in [`pipeline/fast_bulk_indexing/bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/fast_bulk_indexing/bulk_indexing.go) (lines 66-84), the processor creates a configurable pool of workers controlled by `NumOfWorkers` (per-shard workers) and `MaxWorkers` (total concurrent queues).

For high-throughput environments, increase these values proportionally to your CPU count. The optimal setting typically follows the formula: **NumOfWorkers** = CPU cores × 2-4. This parallelization reduces round-trip latency by grouping many index requests into single bulk calls.

```yaml

# gateway.yml configuration

fast_bulk_indexing:
  worker_size: 16          # NumOfWorkers

  max_worker_size: 64      # MaxWorkers (total concurrent queues)

```

## Optimize HTTP Connection Pooling

Persistent connections eliminate TCP handshake overhead when communicating with Elasticsearch nodes. In [`pipeline/fast_bulk_indexing/bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/fast_bulk_indexing/bulk_indexing.go) (lines 77-79), the `MaxConnectionPerHost` parameter controls the HTTP client pool size passed to `elastic.NewBulkProcessor`.

Increase this value from the default if your cluster nodes can handle more parallel connections:

```yaml
fast_bulk_indexing:
  max_connection_per_node: 8   # Increase for high-throughput clusters

```

## Implement Weighted Load Balancing

To prevent hot-spots and distribute traffic evenly across nodes with varying capacities, use the weighted round-robin balancer implemented in [`proxy/balancer/balancer.go`](https://github.com/infinilabs/gateway/blob/main/proxy/balancer/balancer.go). The balancer accepts a weight slice (`ws []int`) that reflects relative node capacity.

Configure weights proportional to your node specifications—higher values for nodes with more CPU/RAM:

```go
// weights reflect node capacities: node-A (3), node-B (2), node-C (1)
weights := []int{3, 2, 1}
balancer := balancer.NewBalancer(weights)
hostIdx := balancer.Distribute()
selectedHost := hosts[hostIdx]

```

## Configure Rate Limiting and Back-Pressure

Prevent downstream cluster overload during traffic spikes by tuning the per-path rate limiter in [`proxy/filters/throttle/request_path_limiter.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/throttle/request_path_limiter.go) (lines 120-132). The implementation uses `MaxQPS` and `Burst` parameters to control request flow without choking the queue.

Raise these thresholds for high-traffic API paths:

```yaml
proxy:
  filters:
    - type: request_path_limiter
      pattern: "^/api/v1/.*"
      max_qps: 2000      # Raise from default 500

      burst: 5000

```

## Tune Queue Consumer Concurrency

When processing from Kafka-style queues, the `FlowRunnerProcessor` dequeue speed directly impacts latency. In [`pipeline/flow_runner/flow_runner.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/flow_runner/flow_runner.go) (lines 92-100), the `FetchMaxMessages` and `FetchMaxBytes` parameters control batch sizes.

For high-throughput scenarios, increase these values to pull larger chunks from the queue:

```yaml
flow_runner:
  consumer:
    fetch_max_messages: 2000
    fetch_max_bytes: 52428800   # 50 MiB

```

## Manage Idle Timeouts and Commit Behavior

Balance between durability and performance by configuring flush timers. The `IdleTimeoutInSecond` parameter in [`pipeline/fast_bulk_indexing/bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/fast_bulk_indexing/bulk_indexing.go) (lines 61-68) controls when workers flush buffers during low activity, while `CommitTimeoutInSeconds` in [`pipeline/flow_runner/flow_runner.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/flow_runner/flow_runner.go) (lines 45-53) manages offset commit frequency.

Tune these to reduce commit chatter under load:

```yaml
fast_bulk_indexing:
  idle_timeout_in_second: 2     # Flush faster under load

flow_runner:
  commit_timeout_in_seconds: 5  # Reduce commit frequency

```

## Enable Fast Host Health Detection

Rapid failover to healthy nodes prevents request stalls. The `elastic.IsHostDead` and `IsHostAvailable` checks in [`proxy/output/elastic/reverseproxy.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/elastic/reverseproxy.go) (lines 298-312) combined with rate-limited logging ensure the system remains responsive during node failures.

Ensure your health check interval is sufficiently aggressive (less than 30 seconds) to detect failures quickly without generating excessive log noise.

## Memory Management for Bulk Buffers

The gateway maintains a memory-efficient bulk buffer pool (`bulkBufferPool`) sized up to 1 GiB by default, initialized in [`pipeline/fast_bulk_indexing/bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/fast_bulk_indexing/bulk_indexing.go) (lines 8-10). While increasing bulk sizes improves throughput, monitor heap usage carefully.

Provision at least 2 GiB of heap for the gateway process on high-throughput nodes, and only increase buffer sizes beyond defaults if you have sufficient RAM to prevent OOM conditions.

## Complete Configuration Examples

### High-Throughput Bulk Indexing Configuration

```yaml
fast_bulk_indexing:
  worker_size: 16               # NumOfWorkers per shard

  max_worker_size: 64           # Total concurrent queues

  max_connection_per_node: 8    # HTTP connections per ES node

  bulk:
    bulk_size_in_bytes: 10485760   # 10 MiB per bulk request

    bulk_max_docs_count: 5000      # Doc count limit

  idle_timeout_in_second: 2        # Aggressive flush

  waiting_after: []                # Disable extra wait checks

  pause_when: null                 # No pause-condition in hot path

```

### Load Balancer with Weighted Distribution

```go
package main

import "github.com/infinilabs/gateway/proxy/balancer"

// Create balancer with weights matching node capacity
func createBalancer(hosts []string) *balancer.Balancer {
    weights := []int{3, 2, 1}  // High, medium, low capacity nodes
    return balancer.NewBalancer(weights)
}

```

## Summary

- **Scale workers linearly** with CPU cores by setting `NumOfWorkers` to 2-4× core count in [`bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/bulk_indexing.go)
- **Increase connection pools** using `MaxConnectionPerHost` to eliminate TCP handshake overhead
- **Use weighted balancing** via [`balancer.go`](https://github.com/infinilabs/gateway/blob/main/balancer.go) to distribute load according to actual node capacity
- **Raise rate limits** for hot paths in [`request_path_limiter.go`](https://github.com/infinilabs/gateway/blob/main/request_path_limiter.go) to prevent artificial throttling
- **Enlarge consumer batches** by increasing `FetchMaxMessages` and `FetchMaxBytes` in [`flow_runner.go`](https://github.com/infinilabs/gateway/blob/main/flow_runner.go)
- **Tune timers** by lowering `IdleTimeoutInSecond` for faster flushing and raising `CommitTimeoutInSeconds` to reduce commit frequency
- **Monitor health aggressively** using the checks in [`reverseproxy.go`](https://github.com/infinilabs/gateway/blob/main/reverseproxy.go) to ensure fast failover
- **Maintain 2+ GiB heap** to accommodate the 1 GiB bulk buffer pool without risking OOM errors

## Frequently Asked Questions

### What is the optimal number of workers for FastBulkIndexingProcessor?

Set `NumOfWorkers` to between 2 and 4 times the number of CPU cores available to the gateway process. This configuration, found in [`pipeline/fast_bulk_indexing/bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/fast_bulk_indexing/bulk_indexing.go), maximizes parallel bulk processing without excessive goroutine overhead. Monitor CPU utilization and adjust upward if cores remain underutilized during peak load.

### How do I prevent memory issues when increasing bulk sizes?

The gateway maintains a 1 GiB `bulkBufferPool` as defined in [`pipeline/fast_bulk_indexing/bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/pipeline/fast_bulk_indexing/bulk_indexing.go). When increasing `BulkSizeInBytes` beyond the default 5 MiB, ensure the gateway process has at least 2 GiB of heap memory available. Monitor garbage collection logs and reduce buffer sizes if you observe frequent GC pauses or OOM errors.

### What is the difference between IdleTimeoutInSecond and CommitTimeoutInSeconds?

`IdleTimeoutInSecond` (in [`bulk_indexing.go`](https://github.com/infinilabs/gateway/blob/main/bulk_indexing.go)) controls how long a bulk worker waits before flushing accumulated documents to Elasticsearch during periods of low activity. `CommitTimeoutInSeconds` (in [`flow_runner.go`](https://github.com/infinilabs/gateway/blob/main/flow_runner.go)) determines how frequently the consumer commits offsets to Kafka, affecting durability guarantees. Lower the idle timeout for lower latency under load, and raise the commit timeout to reduce I/O overhead.

### How does the gateway handle unhealthy Elasticsearch nodes?

The gateway uses `elastic.IsHostDead` and `IsHostAvailable` checks in [`proxy/output/elastic/reverseproxy.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/elastic/reverseproxy.go) (lines 298-312) to detect node failures. Combined with rate-limited logging via `rate.GetRateLimiter`, the system quickly marks failed hosts as unavailable and redirects traffic to healthy nodes, preventing request stalls during cluster instabilities.