# How Bulk Indexing Acceleration Works in INFINI Gateway: Pipeline Deep Dive and Optimization Guide

> Discover how INFINI Gateway accelerates Elasticsearch bulk indexing through sub-buffer routing and async processing. Learn optimization techniques to eliminate bottlenecks and enhance performance.

- Repository: [INFINI Labs/gateway](https://github.com/infinilabs/gateway)
- Tags: deep-dive
- Published: 2026-03-04

---

**INFINI Gateway accelerates Elasticsearch bulk indexing by decomposing incoming `_bulk` requests into routed sub-buffers that are asynchronously reordered, batched, and delivered to target nodes, eliminating network bottlenecks and enabling back-pressure management through configurable throttling and partitioning stages.**

Bulk indexing acceleration in INFINI Gateway reimagines the data path between clients and Elasticsearch clusters. Instead of proxying `_bulk` requests verbatim, the gateway—implemented in the `infinilabs/gateway` repository—intercepts the NDJSON payload and processes it through a specialized pipeline that optimizes for both throughput and reliability. Understanding the internal flow through filters like `bulk_request_throttle`, `bulk_reshuffle`, and `bulk_request_resort` is essential for tuning high-volume ingestion workloads.

## How the Bulk Acceleration Pipeline Works

The pipeline consists of four distinct stages that transform a standard bulk request into an optimized, asynchronously delivered payload. Each stage is implemented as a dedicated filter in the gateway's proxy layer.

| Stage | Function | Source File |
|-------|----------|-------------|
| **Throttle** | Applies per-index rate limiting based on operation count and payload size | [`proxy/filters/throttle/bulk_request_throttle.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/throttle/bulk_request_throttle.go) |
| **Reshuffle** | Parses NDJSON payload, computes routing, splits into sub-bulks, and queues by partition | [`proxy/filters/elastic/bulk_reshuffle.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/elastic/bulk_reshuffle.go) |
| **Response Processing** | Generates synthetic bulk response to acknowledge client immediately | [`proxy/filters/elastic/bulk_reshuffle.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/elastic/bulk_reshuffle.go) |
| **Resort** | Reorders documents by version or timestamp, batches, and commits to downstream | [`proxy/filters/elastic/bulk_request_resort.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/elastic/bulk_request_resort.go) |

### Stage 1: Throttle (Early Back-Pressure)

The `ElasticsearchBulkRequestThrottle` filter acts as the first line of defense against downstream overload. Located in [`proxy/filters/throttle/bulk_request_throttle.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/throttle/bulk_request_throttle.go), this stage counts bulk operations and payload size per index using a **GenericLimiter**. When configured with per-index limits, it automatically delays or drops excess traffic before it enters the reshuffle queue, preventing memory pressure and Elasticsearch throttling on hot indices.

### Stage 2: Reshuffle (Splitting and Routing)

The core transformation happens in [`proxy/filters/elastic/bulk_reshuffle.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/elastic/bulk_reshuffle.go). Here, the filter parses the incoming `_bulk` NDJSON stream using **`elastic.WalkBulkRequests`** from the [`framework/core/elastic/bulk.go`](https://github.com/infinilabs/gateway/blob/main/framework/core/elastic/bulk.go) utility. For each document, it optionally fixes missing `_id` fields via **`FixNullID`** and collects metadata through **`IndexStatsAnalysis`** and **`ActionStatsAnalysis`**.

The filter then determines routing based on the configured **`Level`** parameter:
- **Cluster**: Routes to any available cluster node
- **Node**: Targets the specific node holding the primary shard
- **Index**: Groups by index name
- **Shard**: Targets specific shard IDs

When **`PartitionSize`** exceeds 1, the filter computes a deterministic partition ID using **xxhash**, creating queue keys like `async_bulk##node##es-1##node-abc##partition-3`. Each sub-bulk is buffered in a **`bytebufferpool.ByteBuffer`** (when `bytes_buffer_enabled` is true) to minimize GC overhead.

### Stage 3: Response Processing (Synthetic Acknowledgment)

Immediately after queuing, the reshuffle filter constructs a synthetic bulk response using `startPart`, `itemPart`, and `endPart` templates. This fake response streams back to the client with HTTP 200 status, allowing the client to proceed while the actual indexing happens asynchronously. The filter also injects **`bulk_index_stats`** and **`bulk_action_stats`** into the request context for downstream metric collection.

### Stage 4: Resort (Reordering and Batching)

The final stage runs in [`proxy/filters/elastic/bulk_request_resort.go`](https://github.com/infinilabs/gateway/blob/main/proxy/filters/elastic/bulk_request_resort.go) within dedicated goroutines (**`Sorter.run`**). Workers consume from per-partition output queues, assembling documents into new bulk payloads. They reorder documents using either **`SortDocumentsByVersion`** (for optimistic concurrency) or **`SortDocumentsByTime`** (for chronological sequencing).

Batches are flushed based on **`BatchSizeInDocs`**, **`BatchSizeInMB`**, or **`IdleTimeoutInSeconds`** thresholds. After successful delivery, the filter atomically commits offsets to guarantee at-least-once delivery while preserving ordering guarantees.

## Data Flow Summary

1. **Client** sends `_bulk` request → **Throttle** (optional rate limiting) → **Reshuffle** (split and queue)
2. **Reshuffle** returns synthetic 200 OK to client immediately
3. **Resort** workers consume queues, reorder, batch, and deliver real bulk requests to Elasticsearch nodes

## Performance Tuning and Optimization

Optimizing bulk indexing acceleration requires balancing memory usage, network efficiency, and delivery latency. The following configurations control the pipeline behavior.

### Throttle Configuration for Hot Index Protection

Configure per-index limits in `bulk_request_throttle` to protect specific indices from traffic spikes. Use the `indices` map to define `limit` (operations per second) and `burst` (maximum temporary exceedance) values.

```yaml
bulk_request_throttle:
  indices:
    "logs-*":
      limit: 5000
      burst: 10000

```

### Reshuffle Level and Partitioning Strategies

Set `level: node` in `bulk_reshuffle` to route documents directly to nodes holding primary shards, eliminating inter-node forwarding within Elasticsearch. For clusters exceeding 50 nodes, enable **`partition_size`** to distribute load across independent queues using xxhash-based partitioning.

```yaml
bulk_reshuffle:
  level: node
  partition_size: 8

```

### Buffer Pool and Memory Management

Enable **`bytes_buffer_enabled: true`** to reuse `ByteBuffer` instances from [`framework/lib/bytebufferpool/pool.go`](https://github.com/infinilabs/gateway/blob/main/framework/lib/bytebufferpool/pool.go) instead of allocating new buffers per request. Adjust **`max_buffer_size`** and **`max_buffer_count`** to match available heap memory—defaults of 1 GB and 10,000 buffers suit moderate workloads, but high-volume scenarios require increased limits.

```yaml
bulk_reshuffle:
  bytes_buffer_enabled: true
  max_buffer_size: 2000000
  max_buffer_count: 50000

```

### Resort Batch Sizing and Idle Timeouts

Tune **`batch_size_in_docs`** and **`batch_size_in_mb`** in `bulk_request_resort` to match Elasticsearch's `http.max_content_length` (default 100 MB). Larger batches improve throughput but increase latency. Set **`idle_timeout_in_seconds`** lower than the default 10s to ensure timely delivery during low-traffic periods.

```yaml
bulk_request_resort:
  batch_size_in_docs: 8000
  batch_size_in_mb: 20
  idle_timeout_in_seconds: "2s"
  commit_offset:
    interval: "8s"

```

## Complete Production Configuration

The following [`gateway.yml`](https://github.com/infinilabs/gateway/blob/main/gateway.yml) combines all optimization strategies for a high-throughput, low-latency pipeline:

```yaml
filters:
  - name: bulk_request_throttle
    config:
      indices:
        "logs-*":
          limit: 4000
          burst: 8000

  - name: bulk_reshuffle
    config:
      elasticsearch: es-cluster
      level: node
      partition_size: 4
      bytes_buffer_enabled: true
      max_buffer_size: 5000000
      fix_null_id: true
      index_stats_analysis: true
      action_stats_analysis: true

  - name: bulk_request_resort
    config:
      batch_size_in_docs: 6000
      batch_size_in_mb: 15
      idle_timeout_in_seconds: "5s"
      sort_by: version
      commit_offset:
        interval: "8s"

```

### Client Integration Example

No client changes are required. Applications continue using standard Elasticsearch bulk APIs while the gateway handles all acceleration transparently.

```go
// Standard HTTP client example - works unchanged with INFINI Gateway
bulkBody := `
{ "index": { "_index": "logs-2026.03", "_id": "123" } }
{ "msg": "entry", "ts": "2026-03-04T12:00:00Z" }
`
resp, err := http.Post("http://gateway:8080/_bulk", 
    "application/x-ndjson", 
    strings.NewReader(bulkBody))
// Returns 200 OK immediately (synthetic response)

```

## Summary

- **Throttle** applies per-index back-pressure via `GenericLimiter` in [`bulk_request_throttle.go`](https://github.com/infinilabs/gateway/blob/main/bulk_request_throttle.go) to prevent downstream overload.
- **Reshuffle** splits requests using `elastic.WalkBulkRequests`, routes by node/shard level, partitions via xxhash, and buffers using pooled `ByteBuffer` instances.
- **Resort** asynchronously reorders documents using `SortDocumentsByVersion` or `SortDocumentsByTime`, then batches according to size and idle timeout constraints before final delivery.
- **Optimization** requires tuning `partition_size` for parallelism, enabling buffer pooling to reduce GC, and configuring batch thresholds to balance latency against throughput.

## Frequently Asked Questions

### How does INFINI Gateway handle client responses during bulk acceleration?

The gateway immediately returns a synthetic bulk response constructed from `startPart`, `itemPart`, and `endPart` templates after queuing sub-bulks. This allows clients to proceed without waiting for actual Elasticsearch indexing, while background `Resort` workers handle the real delivery and retry logic.

### What is the purpose of the partition size setting in bulk reshuffle?

`partition_size` determines how many independent queues the gateway creates per routing level. When set above 1, the filter uses xxhash to distribute documents across partitions like `async_bulk##node##es-1##node-abc##partition-3`, enabling parallel consumption by multiple resort workers and preventing single-queue bottlenecks in large clusters.

### How can I prevent out-of-memory errors when processing large bulk payloads?

Enable `bytes_buffer_enabled` to reuse buffers from `bytebufferpool`, and configure `max_buffer_size` and `max_buffer_count` to cap memory consumption per partition. Additionally, use `bulk_request_throttle` to rate-limit hot indices and prevent unbounded queue growth during traffic spikes.

### Does the gateway preserve document ordering during bulk acceleration?

Yes, when configured appropriately. The `Resort` filter supports `SortDocumentsByVersion` for optimistic concurrency control or `SortDocumentsByTime` for chronological ordering. Documents within each partition are reordered before batching, and atomic offset commits ensure at-least-once delivery with ordering guarantees maintained per partition.