How DS4 Handles Worker Registration and Rolling Hash Validation in Its Distributed Protocol

The DS4 distributed protocol uses a lightweight TCP-based handshake where workers send HELLO frames to register model slices with a coordinator, while protecting inference consistency through FNV-1a rolling hashes that validate token prefixes before KV-cache updates.

The antirez/ds4 repository implements a distributed inference engine that splits large language models across multiple workers. Understanding how the distributed protocol handles worker registration and rolling hash validation is essential for operating fault-tolerant, multi-node deployments without silent data corruption. The implementation centers on ds4_distributed.c, which defines the wire format, state machines, and consistency checks.

Worker Registration Protocol

The registration flow establishes a route plan that maps model layer ranges to specific worker endpoints. This plan enables the coordinator to dispatch prefilling and evaluation work to the correct nodes.

Establishing the TCP Connection

When a worker starts, it initiates a blocking connection attempt to the coordinator's endpoint. The function dist_connect_endpoint() (lines 01443‑01453) wraps the socket creation and implements retry logic with exponential backoff until the coordinator accepts the connection. Once established, the socket operates in blocking mode for the HELLO handshake, then switches to non-blocking for the main work loop.

HELLO Frame Structure and Serialization

After connecting, the worker constructs a ds4_dist_hello_fixed structure containing metadata about the model slice it owns. The structure includes:

• model_id and quant_bits – Identifies the model and quantization scheme.
• layer_start and layer_end – Defines the contiguous layer range this worker executes.
• has_output and has_hidden – Boolean flags indicating whether the slice produces logits or hidden states.
• listen_port – The port on which the worker accepts backward-pass connections from peers.

The function dist_hello_to_wire() (lines 01459‑01470) converts these fields to network byte order using htonl(), then the worker transmits the frame with dist_write_frame_header() specifying DS4_DIST_MSG_HELLO as the message type.

Coordinator Acceptance and Route Planning

On the coordinator side, the accept thread calls dist_read_frame_header() (lines 01439‑01455) to read the message type and payload length. It then deserializes the HELLO payload via dist_hello_from_wire() (lines 01471‑01483), which converts the network bytes back to host order.

The coordinator allocates a ds4_dist_worker_entry (defined at lines 00208‑00224) to store the worker's file descriptor, peer address, and layer metadata. This entry is inserted into a linked list at state->workers, forming the route plan used for subsequent work dispatch. After successful insertion, the coordinator sends a HELLO response to confirm the handshake, and the worker enters its main WORK loop.

Rolling Hash Validation for Consistency

To prevent out-of-order KV-cache updates when workers process split layers, the protocol embeds a 64-bit rolling hash of the token prefix in every work frame. Workers recompute this hash locally and reject mismatches before applying state updates.

FNV-1a Hash Algorithm Implementation

DS4 uses a non-cryptographic FNV-1a hash for speed and simplicity. The implementation defines:

• DS4_DIST_TOKEN_HASH_INIT – The offset basis value (14695981039346656037 ULL).
• DS4_DIST_TOKEN_HASH_PRIME – The FNV prime (1099511628211 ULL).

These constants appear at lines 01489‑01493. The function dist_token_hash_update() (lines 01495‑01506) mixes individual token IDs into the running hash, while dist_token_hash_prefix() (lines 01509‑01511) computes the hash over a span of tokens by iterating the update function.

For session-wide validation, dist_session_token_hash_prefix() (lines 01513‑01526) extracts the token array from a ds4_session object, verifies the token count exceeds the prefix length, and returns the computed 64-bit hash.

Embedding Hashes in Work Frames

When dispatching inference work, the coordinator calls dist_session_token_hash_prefix() to obtain the hash of the prompt prefix the worker will process. It then splits the 64-bit value into high and low 32-bit halves using dist_u64_to_halves() (lines 01532‑01535), storing them in the prefix_hash_hi and prefix_hash_lo fields of the ds4_dist_work_fixed structure. This frame is transmitted with message type DS4_DIST_MSG_WORK.

Worker-Side Validation and Error Handling

Upon receiving a WORK frame, the worker deserializes it via dist_work_from_wire() and immediately validates the prefix hash. Inside dist_worker_handle_work() (around line 01845), the worker computes the expected hash using its local session state and compares it against the coordinator-supplied values:

uint64_t expected_prefix_hash;
if (dist_session_token_hash_prefix(session,
        work.n_tokens, &expected_prefix_hash, err, sizeof(err)) != 0 ||
    expected_prefix_hash != dist_u64_from_halves(work.prefix_hash_hi,
                                                  work.prefix_hash_lo)) {
    ds4_log(stderr, DS4_LOG_ERROR,
            "hash mismatch: worker %s expected %016lx got %016lx",
            we->peer_host, expected_prefix_hash,
            dist_u64_from_halves(work.prefix_hash_hi, work.prefix_hash_lo));
    return dist_worker_upstream_send_work_error(upstream,
            dist_u64_from_halves(work.request_hi, work.request_lo),
            "hash mismatch");
}

If the hashes differ, the worker logs the discrepancy and sends a DS4_DIST_MSG_ERROR frame upstream, aborting the work without modifying its KV cache. This prevents silent corruption from mismatched token sequences.

Result Hash Verification

After computing outputs, workers generate a result hash over the generated tokens using the same FNV-1a algorithm. The function dist_result_to_wire() (lines 01611‑01622) serializes the ds4_dist_result_fixed structure, including result_hash_hi and result_hash_lo, allowing the coordinator to verify that the worker's output matches the expected token sequence before integrating it into the global session state.

Implementation Examples

Worker Registration Client Side

/* Build the HELLO payload */
ds4_dist_hello_fixed hello = {
    .model_id      = htonl(state->model_id),
    .quant_bits    = htonl(state->quant_bits),
    .layer_start   = htonl(state->layer_start),
    .layer_end     = htonl(state->layer_end),
    .has_output    = htonl(state->has_output),
    .has_hidden    = htonl(state->has_hidden),
    .ctx_size      = htonl(state->ctx_size),
    .n_layers      = htonl(state->n_layers),
    .listen_port   = htonl(state->listen_port),
    .model_name_len= htonl(strlen(state->model_name)),
};

/* Encode and send */
dist_hello_to_wire(&hello);
dist_write_frame_header(fd, DS4_DIST_MSG_HELLO,
                        sizeof(hello) + hello.model_name_len);
dist_write_full(fd, &hello, sizeof(hello));
dist_write_full(fd, state->model_name, hello.model_name_len);

Source: dist_hello_to_wire() – lines 01459‑01470

Coordinator Handling HELLO Registration

/* After reading the frame header */
dist_hello_from_wire(&hello);
if (hello.model_id != expected_model_id) {
    ds4_log(stderr, DS4_LOG_ERROR, "unexpected model id from worker");
    dist_send_error(fd, "model mismatch");
    return;
}

ds4_dist_worker_entry *we = calloc(1, sizeof(*we));
we->fd = fd;
we->model_id = hello.model_id;
we->layer_start = hello.layer_start;
we->layer_end   = hello.layer_end;
we->has_output  = hello.has_output;
we->has_hidden  = hello.has_hidden;

/* Insert into linked list */
we->next = state->workers;
state->workers = we;

Source: Worker entry definition – lines 00208‑00224

Rolling-Hash Validation in Worker

/* In dist_worker_handle_work() */
dist_work_from_wire(&work);
uint64_t expected_prefix_hash;

if (dist_session_token_hash_prefix(session,
        work.n_tokens, &expected_prefix_hash, err, sizeof(err)) != 0 ||
    expected_prefix_hash != dist_u64_from_halves(work.prefix_hash_hi,
                                                  work.prefix_hash_lo)) {
    ds4_log(stderr, DS4_LOG_ERROR,
            "hash mismatch: worker %s expected %016lx got %016lx",
            we->peer_host, expected_prefix_hash,
            dist_u64_from_halves(work.prefix_hash_hi, work.prefix_hash_lo));
    return dist_worker_upstream_send_work_error(upstream,
            dist_u64_from_halves(work.request_hi, work.request_lo),
            "hash mismatch");
}

Source: Hash comparison inside dist_worker_handle_work() – around line 01845

Result Hash Transmission

/* After computing result tokens */
uint64_t result_hash = dist_token_hash_prefix(result_tokens, n_result);

ds4_dist_result_fixed res = {
    .request_hi = htonl(work.request_hi),
    .request_lo = htonl(work.request_lo),
    .result_hash_hi = htonl((uint32_t)(result_hash >> 32)),
    .result_hash_lo = htonl((uint32_t)result_hash),
    .status = htonl(0),               /* DS4_OK */
    .result_kind = htonl(DS4_DIST_RESULT_LOGITS),
    .payload_bytes = htonl(payload_bytes),
    .payload_bits  = htonl(payload_bits),
    .telemetry_count = htonl(1),
    .telemetry_bytes = htonl(sizeof(telemetry)),
};

dist_result_to_wire(&res);
dist_write_frame_header(fd, DS4_DIST_MSG_RESULT,
                        sizeof(res) + payload_bytes);
dist_write_full(fd, &res, sizeof(res));
dist_write_full(fd, payload, payload_bytes);

Source: Result conversion – dist_result_to_wire() – lines 01611‑01622

Summary

• Worker registration relies on a HELLO handshake over TCP, where workers advertise their layer ranges via ds4_dist_hello_fixed frames and the coordinator stores them in a linked list of ds4_dist_worker_entry structures.
• Route planning occurs automatically as the coordinator populates state->workers, mapping specific layer ranges to file descriptors for subsequent work dispatch.
• Rolling hash validation uses FNV-1a to compute 64-bit checksums of token prefixes, embedding these in WORK frames via prefix_hash_hi/lo fields.
• Consistency enforcement happens at the worker level; mismatches trigger immediate DS4_DIST_MSG_ERROR responses, preventing KV-cache corruption from out-of-order tokens.
• Result verification allows the coordinator to confirm worker outputs through result_hash fields in RESULT frames, ensuring end-to-end integrity.

Frequently Asked Questions

What hash algorithm does DS4 use for token validation?

DS4 implements the FNV-1a (Fowler-Noll-Vo) non-cryptographic hash algorithm using a 64-bit state space. The implementation uses an offset basis of 14695981039346656037 and a prime multiplier of 1099511628211, defined in ds4_distributed.c at lines 01489‑01493. This provides a fast, compact checksum suitable for detecting token sequence mismatches without the overhead of cryptographic hashing.

How does the coordinator handle worker failures during registration?

If a worker sends a HELLO frame with a mismatched model_id or malformed data, the coordinator calls dist_send_error() and closes the connection immediately without inserting the worker into state->workers. The dist_connect_endpoint() function on the worker side implements exponential backoff retry logic, allowing workers to reconnect if the coordinator temporarily rejects the handshake or if network partitions occur during startup.

What happens when a worker detects a hash mismatch?

When dist_worker_handle_work() detects that the locally computed prefix hash differs from the coordinator-supplied prefix_hash_hi/lo values, it logs the mismatch via ds4_log() and returns dist_worker_upstream_send_work_error(). This sends a DS4_DIST_MSG_ERROR frame upstream with the error string "hash mismatch", causing the coordinator to abort or retry the request without applying the corrupted KV state. The worker skips all inference computation for that frame.

Can the distributed protocol support heterogeneous model slices?

Yes. The ds4_dist_hello_fixed structure includes fields for quant_bits, layer_start, layer_end, and capability flags (has_output, has_hidden), allowing workers to register different quantization levels or layer spans. The coordinator stores these capabilities in the ds4_dist_worker_entry and uses them to build a route plan that accounts for heterogeneous slices, though all workers must share the same base model_id to pass validation in dist_hello_from_wire().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}