# Debugging Distributed Inference in DS4: Layer Routing and Worker Communication

> Debug distributed inference in DS4 by enabling trace flags to analyze layer routing stalls, HELLO frames, and telemetry. Inspect worker communication and optimize performance.

- Repository: [Salvatore Sanfilippo/ds4](https://github.com/antirez/ds4)
- Tags: debugging
- Published: 2026-08-08

---

**Enable `DS4_DIST_TRACE=1` and `DS4_DIST_CONNECT_TRACE=1` to debug layer routing stalls, verify HELLO frames, and inspect telemetry fields like `downstream_wait_usec` in the antirez/ds4 distributed inference runtime.**

The ds4 project implements distributed LLM inference by splitting models into contiguous layer slices across a coordinator and multiple workers. When **debugging distributed inference in ds4**, understanding the binary protocol, route planning logic, and telemetry aggregation is essential for resolving communication failures and latency bottlenecks. This guide references the exact source locations in [`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c) and related files to help you diagnose issues in the message flow.

## Understanding the DS4 Distributed Architecture

DS4 splits inference across a **coordinator** that owns the main `ds4_session` and **workers** that execute assigned layer slices. Workers maintain private KV caches and communicate via a custom binary protocol defined in [`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c).

### Coordinator and Worker Roles

The **coordinator** accepts incoming connections, builds a `ds4_dist_route_plan`, and forwards work messages to workers. Key functions include `dist_coordinator_prefill_prompt` (lines 517–526) for streaming prompt batches and telemetry aggregation logic.

The **worker** receives frames via `dist_worker_handle_work` (declarations at lines 496–505), processes *WORK* frames, and returns *RESULT* frames containing hidden states or logits. Workers also handle snapshot save/load operations through `dist_worker_handle_snapshot_*` functions.

### Binary Protocol Overview

All communication uses a binary framing protocol starting with magic number `0x44533444` (ASCII "DS4D"):

| Message Type | Value | Purpose |
|--------------|-------|---------|
| `DS4_DIST_MSG_HELLO` | 1 | Worker announces model-id and layer range |
| `DS4_DIST_MSG_ERROR` | 2 | Protocol violation reporting |
| `DS4_DIST_MSG_WORK` | 3 | Coordinator sends token batches and activation data |
| `DS4_DIST_MSG_RESULT` | 4 | Worker returns logits, hidden states, and telemetry |

Frame headers are written with `dist_write_frame_header` and read with `dist_read_frame_header` (lines 30–46). The protocol constants are defined at lines 44–70 in [`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c).

### Layer Routing Mechanics

The `--layers` CLI option is parsed by `ds4_dist_parse_layers` to create a `ds4_dist_route_plan`. Each route entry uses the `ds4_dist_route_fixed` struct:

```c
typedef struct {
    uint32_t host_len;    // length of hostname string
    uint32_t port;        // TCP port
    uint32_t layer_start; // first layer in slice
    uint32_t layer_end;   // last layer (inclusive)
    uint32_t flags;       // e.g. DS4_DIST_ROUTE_F_OUTPUT_LOGITS
} ds4_dist_route_fixed;

```

Route serialization uses `dist_route_to_wire` and `dist_route_from_wire` (lines 84–97). The coordinator stores live socket file descriptors in `ds4_dist_route_entry` structures within the plan.

## Debugging Worker Communication Flow

### Connection Setup and HELLO Exchange

Workers connect via `dist_connect_endpoint`, which retries up to 200 times with low-latency socket options (`TCP_NODELAY`, `SO_KEEPALIVE`). Buffer sizes default to 128 MiB but can be overridden with `DS4_DIST_SOCKET_BUFFER_MB` (see socket setup at lines 107–121).

After connecting, the worker sends a `HELLO` frame serialized by `dist_hello_to_wire`, and the coordinator validates it with `dist_hello_from_wire` (lines 59–71). Verify that `layer_start` and `layer_end` in the HELLO payload match your `--layers` configuration.

### Prefill and Evaluation Loops

During prefill, the coordinator calls `dist_coordinator_prefill_prompt` to stream token batches as *WORK* frames via `dist_send_work_frame`. Each frame contains:

- Request ID and model ID
- Layer range for the slice
- Optional packed activations (controlled by `--activation-bits`)

The worker receives the frame in `dist_worker_handle_work`, decodes activations using `dist_decode_activation_payload` (lines 86–124), and executes the slice with `ds4_engine_eval_slice`. Results are returned via `dist_send_result_frame` containing a `ds4_dist_telemetry_fixed` struct (lines 141–152).

The coordinator maintains ordering guarantees through a *pending request* queue (`ds4_dist_pending_request`) tracked in `pending_head`, `pending_tail`, and `pending_count` fields (lines 88–102).

### Interpreting Telemetry Data

The `ds4_dist_telemetry_fixed` struct captures timing data for each inference step:

- **eval_usec**: Time spent in model evaluation
- **downstream_wait_usec**: Time waiting for downstream workers (high values indicate blocked sockets)
- **forward_send_usec**: Time spent sending results upstream

When `DS4_DIST_TRACE=1` is enabled, the coordinator prints summaries like:

```

ds4: distributed debug: result ack, telemetry: eval_usec=321, downstream_wait_usec=12, forward_send_usec=4

```

## Environment Variables for Debugging

| Variable | Effect |
|----------|--------|
| `DS4_DIST_TRACE=1` | Enables `DIST_DEBUG` macro for verbose protocol events |
| `DS4_DIST_CONNECT_TRACE=1` | Logs connect attempts, socket binds, and address selection |
| `DS4_DIST_SOCKET_BUFFER_MB` | Override default 128 MiB send/receive buffers |
| `DS4_DIST_PREFILL_SEND_DEPTH` | Prefill chunks queued ahead (default 2) |
| `DS4_DIST_WORKER_PREFETCH_DEPTH` | Worker-side prefetch window (default 2) |
| `DS4_DIST_WORKER_FORWARD_WINDOW` | Max pending forward requests per worker (default 4) |
| `DS4_DIST_DECODE_PROFILE=1` | Collects timing data for activation decode operations |

Set both `DS4_DIST_TRACE=1` and `DS4_DIST_CONNECT_TRACE=1` together for a complete view of socket creation and wire-level bytes.

## Step-by-Step Debugging Workflow

1. **Start the coordinator with tracing**

   ```bash
   DS4_DIST_TRACE=1 DS4_DIST_CONNECT_TRACE=1 ./ds4 \
       --role coordinator -p 5000 \
       --layers 0-11 --model mymodel.gguf
   ```

   The coordinator logs `dist_open_listener` and `dist_connect_endpoint_once` activity, then waits for workers.

2. **Start a worker pointing at the coordinator**

   ```bash
   DS4_DIST_TRACE=1 ./ds4 \
       --role worker -c localhost:5000 \
       --layers 12-23 --model mymodel.gguf
   ```

   Verify the worker logs show the HELLO frame transmission and successful connection.

3. **Inspect the HELLO payload**

   Locate the printed HELLO dump in logs (e.g., `model_id=1, layer_start=12, layer_end=23`). Confirm these values match the `--layers` argument.

4. **Validate activation bit-packing**

   If using `--activation-bits 8` or `16`, the coordinator invokes `dist_write_activation_payload`. Enable `DS4_DIST_DECODE_PROFILE=1` to verify bytes written match bytes decoded on the worker side.

5. **Check telemetry for stalls**

   Review the telemetry output after each token. Large `downstream_wait_usec` values often indicate network congestion or slow downstream workers.

6. **Test reconnection logic**

   Kill a worker process temporarily. The coordinator's `dist_connect_errno_retryable` logic (lines 152–158) automatically retries the connection. Observe the retry behavior in `DS4_DIST_CONNECT_TRACE` logs.

7. **Verify snapshot operations**

   Trigger a snapshot save (send `SIGUSR2` to the coordinator) and monitor the `DS4_DIST_MSG_SNAPSHOT_*` sequence (messages 5–8). The worker handles these via `dist_worker_handle_snapshot_save` and `dist_worker_handle_snapshot_load`. Errors surface as `DS4_DIST_MSG_ERROR` frames.

## Key Source Files and Functions

| File | Purpose | Key Locations |
|------|---------|---------------|
| **[`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c)** | Core distributed runtime, protocol framing, activation packing | Protocol constants L44–L70; activation helpers L86–124; socket setup L107–121; HELLO parsing L59–71 |
| **[`ds4_distributed.h`](https://github.com/antirez/ds4/blob/main/ds4_distributed.h)** | Public API for coordinator sessions and route structures | `ds4_dist_route_plan` definition L21–30 |
| **[`ds4_layer_pack.c`](https://github.com/antirez/ds4/blob/main/ds4_layer_pack.c)** | Route plan construction from `--layers` CLI | `ds4_dist_parse_layers`, `dist_route_to_wire` L84–97 |
| **[`ds4_tp.c`](https://github.com/antirez/ds4/blob/main/ds4_tp.c)** | Thread-pool utilities for parallel worker I/O | Used by coordinator forwarder threads |
| **[`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c)** | HTTP/REST server exposing distributed endpoints | Integration debugging with external clients |
| **[`ds4.c`](https://github.com/antirez/ds4/blob/main/ds4.c)** | Entry point for role selection (coordinator vs worker) | Wires sessions to coordinator state |
| **[`ds4_help.c`](https://github.com/antirez/ds4/blob/main/ds4_help.c)** | CLI documentation for distributed options | `--role`, `--layers`, `--activation-bits` help text |

## Summary

- **Enable tracing** with `DS4_DIST_TRACE=1` and `DS4_DIST_CONNECT_TRACE=1` to see protocol events and socket operations in real-time.
- **Verify HELLO frames** to ensure workers report correct `layer_start` and `layer_end` values matching the `--layers` configuration.
- **Monitor telemetry fields** (`eval_usec`, `downstream_wait_usec`, `forward_send_usec`) to identify bottlenecks; high `downstream_wait_usec` indicates socket blocking.
- **Tune performance** using `DS4_DIST_SOCKET_BUFFER_MB`, `DS4_DIST_PREFILL_SEND_DEPTH`, and `DS4_DIST_WORKER_FORWARD_WINDOW` to optimize throughput.
- **Focus debugging** on `dist_coordinator_prefill_prompt` and `dist_worker_handle_work` in [`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c) where most communication bugs manifest.

## Frequently Asked Questions

### How do I verify that workers are correctly reporting their layer ranges?

Enable `DS4_DIST_TRACE=1` and inspect the HELLO frame dump in the coordinator logs. The output shows `model_id`, `layer_start`, and `layer_end` values. Cross-reference these with the `--layers` argument you passed to the worker (e.g., `12-23`) to ensure the `ds4_dist_route_fixed` structure was parsed correctly in [`ds4_layer_pack.c`](https://github.com/antirez/ds4/blob/main/ds4_layer_pack.c).

### What does a high `downstream_wait_usec` value indicate in telemetry?

A large `downstream_wait_usec` value in the `ds4_dist_telemetry_fixed` struct indicates the worker spent excessive time waiting for a downstream worker to accept data. This usually signals a blocked socket, network congestion, or that the downstream worker's `DS4_DIST_WORKER_PREFETCH_DEPTH` window is saturated. Check the coordinator's pending request queue (`pending_count` field) for congestion.

### How can I debug activation packing errors when using `--activation-bits 8` or `16`?

Set `DS4_DIST_DECODE_PROFILE=1` to enable internal timing collection in `dist_decode_activation_payload` (lines 86–124). Compare the byte counts logged during `dist_write_activation_payload` (coordinator side) against the decoded bytes on the worker. Mismatches indicate corruption in the activation payload or incorrect bit-width negotiation between `dist_work_to_wire` and `dist_work_from_wire` (lines 60–81).

### Where is the route plan constructed and how can I inspect it?

The `ds4_dist_route_plan` is constructed in [`ds4_layer_pack.c`](https://github.com/antirez/ds4/blob/main/ds4_layer_pack.c) by `ds4_dist_parse_layers` when parsing the `--layers` CLI option. The plan contains an array of `ds4_dist_route_entry` structures with live socket file descriptors. You can inspect the wire-format representation using `dist_route_to_wire` and `dist_route_from_wire` (lines 84–97) to verify host, port, and layer range assignments before workers connect.