# How ds4-server Handles Batched Sessions for Multi-User Inference

> Discover how ds4-server efficiently handles batched sessions for multi-user inference. Learn how it boosts throughput by coalescing requests for GPU/Metal backends.

- Repository: [Salvatore Sanfilippo/ds4](https://github.com/antirez/ds4)
- Tags: internals
- Published: 2026-08-05

---

**`ds4-server` uses a batched mode where concurrent client requests are collected by a dedicated decode-worker thread, coalesced into a single array, and submitted together to GPU or Metal backends via `ds4_sessions_eval_batch`, dramatically improving throughput while maintaining thread-safety and atomic failure semantics.**

The `ds4-server` component of the [antirez/ds4](https://github.com/antirez/ds4) repository implements a sophisticated batching system designed to maximize hardware utilization when serving multiple simultaneous users. Instead of processing each inference request individually—which would leave GPU/Metal cores underutilized—the server aggregates pending evaluations and executes them as a single batch operation.

## The Three-Stage Batching Pipeline

Multi-user inference in `ds4-server` flows through three coordinated stages: request intake, batch assembly, and unified backend execution.

### Stage 1: Per-Request Entry Point (`server_eval_token`)

Every client request enters through **`server_eval_token`** in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) (lines 10801–10809). This function acts as a router that behaves differently depending on the `batched_mode` configuration:

- **Non-batched path**: The function acquires `inference_mu`, calls `ds4_session_eval` directly for single-session evaluation, and returns immediately.
- **Batched path**: The request is recorded in its assigned slot—storing the `decode_token`, setting `decode_pending = true`, and incrementing the global pending counter. The function then signals the decode-worker via `pthread_cond_broadcast(&s->model_cv)` and yields control.

This design allows the client thread to return to handling I/O while the heavy inference work is deferred to a specialized worker.

### Stage 2: Decode-Worker Thread (`decode_worker_main`)

The **`decode_worker_main`** function (lines 10855–10931 of [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c)) implements the core batching logic. Its operation follows a precise sequence:

1. **Wait for work**: The worker spins on `while (s->decode_pending == 0 && …)`, sleeping on `pthread_cond_timedwait` until at least one slot has `decode_pending` set.

2. **Optional coalescing**: To maximize batch size, the worker can wait an additional configurable window—controlled by **`DS4_SERVER_DECODE_COALESCE_US`**—allowing straggler requests to arrive before proceeding.

3. **Batch construction**: The worker iterates through all slots, collecting those with `decode_pending == true`. For each pending slot, it:
   - Clears `decode_pending` and sets `decode_in_flight = true`
   - Populates a `ds4_decode_item` struct with the session pointer and token
   - Stores the slot pointer in a `members[]` array for result distribution

4. **Batch execution**: With the `items[count]` array assembled, the worker acquires `inference_mu` and calls `ds4_sessions_eval_batch(items, count, …)`.

5. **Result distribution**: After the backend returns, the worker copies `decode_rc` and `decode_err` into each slot, sets `decode_done = true`, and broadcasts to wake waiting client threads.

### Stage 3: Core Library Batch Evaluation (`ds4_sessions_eval_batch`)

The **`ds4_sessions_eval_batch`** function in [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c) (lines 61333–61372) serves as the gateway to hardware acceleration. Its responsibilities include:

- **Validation**: Verifies that all sessions belong to the same `ds4_engine`, tokens are within vocabulary bounds, no session appears twice, and no session has exceeded its context window.

- **Single-item optimization**: If `count == 1`, delegates to `ds4_session_eval` to avoid batch overhead.

- **Backend dispatch**: For multi-item batches, routes to:
  - `ds4_sessions_eval_batch_cuda` for NVIDIA GPU execution
  - `ds4_sessions_eval_batch_metal` for Apple Silicon GPU execution

- **Atomic failure handling**: If the backend returns an error, the function invalidates **all** sessions in the batch, ensuring no partial progress contaminates the system state.

## Synchronization Architecture

The batching system employs a two-lock strategy to balance concurrency and correctness:

| Phase | Mechanism | Purpose |
|-------|-----------|---------|
| Slot state & pending count | `model_mu` + `pthread_cond_broadcast` | Protects request enqueue/dequeue and worker coordination |
| Inference engine state | `inference_mu` | Serializes access to GPU/Metal context and model weights |
| Worker sleep | `pthread_cond_timedwait` | Allows microsecond-precision coalescing without busy-waiting |

This separation enables the decode-worker to assemble batches under `model_mu` while only briefly holding `inference_mu` during actual execution—minimizing contention on the critical path.

## Practical Code Flow

**Client thread enqueueing a request:**

```c
/* Client thread – enqueue a token for batched evaluation */
int rc = server_eval_token(srv, slot, token, err_buf, sizeof(err_buf));
if (rc != 0) { /* handle error */ }

```

**Worker thread building and executing the batch:**

```c
/* Inside decode_worker_main – build the batch */
for (int i = 0; i < s->slot_count; i++) {
    server_slot *slot = &s->slots[i];
    if (!slot->decode_pending) continue;
    slot->decode_pending = false;
    slot->decode_in_flight = true;
    s->decode_pending--;
    members[count] = slot;
    items[count].session = slot->session;
    items[count].token   = slot->decode_token;
    count++;
}

/* Execute the batch */
char batch_err[160] = {0};
int rc = ds4_sessions_eval_batch(items, count, batch_err, sizeof(batch_err));

```

**Core library validation and dispatch:**

```c
/* ds4_sessions_eval_batch – validation & dispatch */
if (count == 1) return ds4_session_eval(items[0].session, items[0].token, err, errlen);
...
if (e->backend == DS4_BACKEND_CUDA)
    return ds4_sessions_eval_batch_cuda(items, count, err, errlen);
...

```

## Key Source Files

| File | Role in Batched Multi-User Inference |
|------|--------------------------------------|
| [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) | Implements `server_eval_token`, per-slot state machine, and `decode_worker_main` batching thread |
| [`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c) | Core library containing `ds4_sessions_eval_batch` with CUDA/Metal backend dispatch |
| [`ds4_session.c`](https://github.com/antirez/ds4/blob/main/ds4_session.c) | Defines `ds4_session` structure and non-batched `ds4_session_eval` fallback |
| [`ds4_tp.c`](https://github.com/antirez/ds4/blob/main/ds4_tp.c) | Thread-pool utilities for slot worker management |

## Summary

- **`ds4-server`** achieves high-throughput multi-user inference through explicit request batching rather than per-request GPU dispatch.
- The **decode-worker thread** (`decode_worker_main`) coalesces pending evaluations with microsecond-precision timing (`DS4_SERVER_DECODE_COALESCE_US`).
- **Two-lock synchronization** (`model_mu` for coordination, `inference_mu` for engine access) prevents contention while ensuring correctness.
- **`ds4_sessions_eval_batch`** validates inputs rigorously, optimizes single-item cases, and guarantees all-or-nothing failure semantics.
- Client threads experience blocking semantics identical to non-batched mode, hiding the complexity of asynchronous batch assembly.

## Frequently Asked Questions

### What is the purpose of `DS4_SERVER_DECODE_COALESCE_US`?

**`DS4_SERVER_DECODE_COALESCE_US`** configures a microsecond delay before batch execution, allowing the decode-worker to accumulate additional pending requests. This increases average batch size and GPU utilization at the cost of minor latency inflation. The trade-off is tunable based on workload characteristics and latency requirements.

### How does `ds4-server` prevent partial failures in batched sessions?

The **`ds4_sessions_eval_batch`** function validates all sessions before backend submission and implements all-or-nothing failure handling. If the CUDA or Metal backend returns an error, the function invalidates every session in the batch rather than allowing partial success. This prevents state corruption where some sessions advance their context while others fail.

### Can batched and non-batched modes coexist in the same server instance?

No—batching is controlled by a global **`batched_mode`** flag at server initialization. When disabled, `server_eval_token` acquires `inference_mu` and calls `ds4_session_eval` synchronously for each request. When enabled, all requests flow through the decode-worker thread. The mode is selected based on expected concurrency and latency requirements.

### What happens to client threads while their requests are being batched?

Client threads block on a per-slot condition variable after enqueueing their request. The thread yields after signaling `model_cv`, then sleeps until `decode_worker_main` completes the batch and broadcasts `model_cv` again. From the client's perspective, the latency appears as standard request processing time; the batching optimization is completely transparent.