Debugging Distributed Inference in DS4: Layer Routing and Worker Communication

Enable DS4_DIST_TRACE=1 and DS4_DIST_CONNECT_TRACE=1 to debug layer routing stalls, verify HELLO frames, and inspect telemetry fields like downstream_wait_usec in the antirez/ds4 distributed inference runtime.

The ds4 project implements distributed LLM inference by splitting models into contiguous layer slices across a coordinator and multiple workers. When debugging distributed inference in ds4, understanding the binary protocol, route planning logic, and telemetry aggregation is essential for resolving communication failures and latency bottlenecks. This guide references the exact source locations in ds4_distributed.c and related files to help you diagnose issues in the message flow.

Understanding the DS4 Distributed Architecture

DS4 splits inference across a coordinator that owns the main ds4_session and workers that execute assigned layer slices. Workers maintain private KV caches and communicate via a custom binary protocol defined in ds4_distributed.c.

Coordinator and Worker Roles

The coordinator accepts incoming connections, builds a ds4_dist_route_plan, and forwards work messages to workers. Key functions include dist_coordinator_prefill_prompt (lines 517–526) for streaming prompt batches and telemetry aggregation logic.

The worker receives frames via dist_worker_handle_work (declarations at lines 496–505), processes WORK frames, and returns RESULT frames containing hidden states or logits. Workers also handle snapshot save/load operations through dist_worker_handle_snapshot_* functions.

Binary Protocol Overview

All communication uses a binary framing protocol starting with magic number 0x44533444 (ASCII "DS4D"):

Message Type	Value	Purpose
DS4_DIST_MSG_HELLO	1	Worker announces model-id and layer range
DS4_DIST_MSG_ERROR	2	Protocol violation reporting
DS4_DIST_MSG_WORK	3	Coordinator sends token batches and activation data
DS4_DIST_MSG_RESULT	4	Worker returns logits, hidden states, and telemetry

Frame headers are written with dist_write_frame_header and read with dist_read_frame_header (lines 30–46). The protocol constants are defined at lines 44–70 in ds4_distributed.c.

Layer Routing Mechanics

The --layers CLI option is parsed by ds4_dist_parse_layers to create a ds4_dist_route_plan. Each route entry uses the ds4_dist_route_fixed struct:

typedef struct {
    uint32_t host_len;    // length of hostname string
    uint32_t port;        // TCP port
    uint32_t layer_start; // first layer in slice
    uint32_t layer_end;   // last layer (inclusive)
    uint32_t flags;       // e.g. DS4_DIST_ROUTE_F_OUTPUT_LOGITS
} ds4_dist_route_fixed;

Route serialization uses dist_route_to_wire and dist_route_from_wire (lines 84–97). The coordinator stores live socket file descriptors in ds4_dist_route_entry structures within the plan.

Debugging Worker Communication Flow

Connection Setup and HELLO Exchange

Workers connect via dist_connect_endpoint, which retries up to 200 times with low-latency socket options (TCP_NODELAY, SO_KEEPALIVE). Buffer sizes default to 128 MiB but can be overridden with DS4_DIST_SOCKET_BUFFER_MB (see socket setup at lines 107–121).

After connecting, the worker sends a HELLO frame serialized by dist_hello_to_wire, and the coordinator validates it with dist_hello_from_wire (lines 59–71). Verify that layer_start and layer_end in the HELLO payload match your --layers configuration.

Prefill and Evaluation Loops

During prefill, the coordinator calls dist_coordinator_prefill_prompt to stream token batches as WORK frames via dist_send_work_frame. Each frame contains:

• Request ID and model ID
• Layer range for the slice
• Optional packed activations (controlled by --activation-bits)

The worker receives the frame in dist_worker_handle_work, decodes activations using dist_decode_activation_payload (lines 86–124), and executes the slice with ds4_engine_eval_slice. Results are returned via dist_send_result_frame containing a ds4_dist_telemetry_fixed struct (lines 141–152).

The coordinator maintains ordering guarantees through a pending request queue (ds4_dist_pending_request) tracked in pending_head, pending_tail, and pending_count fields (lines 88–102).

Interpreting Telemetry Data

The ds4_dist_telemetry_fixed struct captures timing data for each inference step:

• eval_usec: Time spent in model evaluation
• downstream_wait_usec: Time waiting for downstream workers (high values indicate blocked sockets)
• forward_send_usec: Time spent sending results upstream

When DS4_DIST_TRACE=1 is enabled, the coordinator prints summaries like:

ds4: distributed debug: result ack, telemetry: eval_usec=321, downstream_wait_usec=12, forward_send_usec=4

Environment Variables for Debugging

Variable	Effect
DS4_DIST_TRACE=1	Enables DIST_DEBUG macro for verbose protocol events
DS4_DIST_CONNECT_TRACE=1	Logs connect attempts, socket binds, and address selection
DS4_DIST_SOCKET_BUFFER_MB	Override default 128 MiB send/receive buffers
DS4_DIST_PREFILL_SEND_DEPTH	Prefill chunks queued ahead (default 2)
DS4_DIST_WORKER_PREFETCH_DEPTH	Worker-side prefetch window (default 2)
DS4_DIST_WORKER_FORWARD_WINDOW	Max pending forward requests per worker (default 4)
DS4_DIST_DECODE_PROFILE=1	Collects timing data for activation decode operations

Set both DS4_DIST_TRACE=1 and DS4_DIST_CONNECT_TRACE=1 together for a complete view of socket creation and wire-level bytes.

Step-by-Step Debugging Workflow

1. Start the coordinator with tracing

   DS4_DIST_TRACE=1 DS4_DIST_CONNECT_TRACE=1 ./ds4 \
       --role coordinator -p 5000 \
       --layers 0-11 --model mymodel.gguf

   The coordinator logs dist_open_listener and dist_connect_endpoint_once activity, then waits for workers.

2. Start a worker pointing at the coordinator

   DS4_DIST_TRACE=1 ./ds4 \
       --role worker -c localhost:5000 \
       --layers 12-23 --model mymodel.gguf

   Verify the worker logs show the HELLO frame transmission and successful connection.

3. Inspect the HELLO payload

   Locate the printed HELLO dump in logs (e.g., model_id=1, layer_start=12, layer_end=23). Confirm these values match the --layers argument.

4. Validate activation bit-packing

   If using --activation-bits 8 or 16, the coordinator invokes dist_write_activation_payload. Enable DS4_DIST_DECODE_PROFILE=1 to verify bytes written match bytes decoded on the worker side.

5. Check telemetry for stalls

   Review the telemetry output after each token. Large downstream_wait_usec values often indicate network congestion or slow downstream workers.

6. Test reconnection logic

   Kill a worker process temporarily. The coordinator's dist_connect_errno_retryable logic (lines 152–158) automatically retries the connection. Observe the retry behavior in DS4_DIST_CONNECT_TRACE logs.

7. Verify snapshot operations

   Trigger a snapshot save (send SIGUSR2 to the coordinator) and monitor the DS4_DIST_MSG_SNAPSHOT_* sequence (messages 5–8). The worker handles these via dist_worker_handle_snapshot_save and dist_worker_handle_snapshot_load. Errors surface as DS4_DIST_MSG_ERROR frames.

Key Source Files and Functions

File	Purpose	Key Locations
ds4_distributed.c	Core distributed runtime, protocol framing, activation packing	Protocol constants L44–L70; activation helpers L86–124; socket setup L107–121; HELLO parsing L59–71
ds4_distributed.h	Public API for coordinator sessions and route structures	ds4_dist_route_plan definition L21–30
ds4_layer_pack.c	Route plan construction from --layers CLI	ds4_dist_parse_layers, dist_route_to_wire L84–97
ds4_tp.c	Thread-pool utilities for parallel worker I/O	Used by coordinator forwarder threads
ds4_server.c	HTTP/REST server exposing distributed endpoints	Integration debugging with external clients
ds4.c	Entry point for role selection (coordinator vs worker)	Wires sessions to coordinator state
ds4_help.c	CLI documentation for distributed options	--role, --layers, --activation-bits help text

Summary

• Enable tracing with DS4_DIST_TRACE=1 and DS4_DIST_CONNECT_TRACE=1 to see protocol events and socket operations in real-time.
• Verify HELLO frames to ensure workers report correct layer_start and layer_end values matching the --layers configuration.
• Monitor telemetry fields (eval_usec, downstream_wait_usec, forward_send_usec) to identify bottlenecks; high downstream_wait_usec indicates socket blocking.
• Tune performance using DS4_DIST_SOCKET_BUFFER_MB, DS4_DIST_PREFILL_SEND_DEPTH, and DS4_DIST_WORKER_FORWARD_WINDOW to optimize throughput.
• Focus debugging on dist_coordinator_prefill_prompt and dist_worker_handle_work in ds4_distributed.c where most communication bugs manifest.

Frequently Asked Questions

How do I verify that workers are correctly reporting their layer ranges?

Enable DS4_DIST_TRACE=1 and inspect the HELLO frame dump in the coordinator logs. The output shows model_id, layer_start, and layer_end values. Cross-reference these with the --layers argument you passed to the worker (e.g., 12-23) to ensure the ds4_dist_route_fixed structure was parsed correctly in ds4_layer_pack.c.

What does a high downstream_wait_usec value indicate in telemetry?

A large downstream_wait_usec value in the ds4_dist_telemetry_fixed struct indicates the worker spent excessive time waiting for a downstream worker to accept data. This usually signals a blocked socket, network congestion, or that the downstream worker's DS4_DIST_WORKER_PREFETCH_DEPTH window is saturated. Check the coordinator's pending request queue (pending_count field) for congestion.

How can I debug activation packing errors when using --activation-bits 8 or 16?

Set DS4_DIST_DECODE_PROFILE=1 to enable internal timing collection in dist_decode_activation_payload (lines 86–124). Compare the byte counts logged during dist_write_activation_payload (coordinator side) against the decoded bytes on the worker. Mismatches indicate corruption in the activation payload or incorrect bit-width negotiation between dist_work_to_wire and dist_work_from_wire (lines 60–81).

Where is the route plan constructed and how can I inspect it?

The ds4_dist_route_plan is constructed in ds4_layer_pack.c by ds4_dist_parse_layers when parsing the --layers CLI option. The plan contains an array of ds4_dist_route_entry structures with live socket file descriptors. You can inspect the wire-format representation using dist_route_to_wire and dist_route_from_wire (lines 84–97) to verify host, port, and layer range assignments before workers connect.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}