# How DS4 Handles Worker Registration and Rolling Hash Validation in Its Distributed Protocol

> Discover how the DS4 distributed protocol manages worker registration via HELLO frames and ensures inference consistency with FNV-1a rolling hashes for token prefix validation before KV-cache updates.

- Repository: [Salvatore Sanfilippo/ds4](https://github.com/antirez/ds4)
- Tags: internals
- Published: 2026-08-09

---

**The DS4 distributed protocol uses a lightweight TCP-based handshake where workers send HELLO frames to register model slices with a coordinator, while protecting inference consistency through FNV-1a rolling hashes that validate token prefixes before KV-cache updates.**

The antirez/ds4 repository implements a distributed inference engine that splits large language models across multiple workers. Understanding how the **distributed protocol handles worker registration and rolling hash validation** is essential for operating fault-tolerant, multi-node deployments without silent data corruption. The implementation centers on [`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c), which defines the wire format, state machines, and consistency checks.

## Worker Registration Protocol

The registration flow establishes a **route plan** that maps model layer ranges to specific worker endpoints. This plan enables the coordinator to dispatch prefilling and evaluation work to the correct nodes.

### Establishing the TCP Connection

When a worker starts, it initiates a blocking connection attempt to the coordinator's endpoint. The function `dist_connect_endpoint()` (lines [01443‑01453](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1443-L1453)) wraps the socket creation and implements retry logic with exponential backoff until the coordinator accepts the connection. Once established, the socket operates in blocking mode for the HELLO handshake, then switches to non-blocking for the main work loop.

### HELLO Frame Structure and Serialization

After connecting, the worker constructs a `ds4_dist_hello_fixed` structure containing metadata about the model slice it owns. The structure includes:

- **model_id** and **quant_bits** – Identifies the model and quantization scheme.
- **layer_start** and **layer_end** – Defines the contiguous layer range this worker executes.
- **has_output** and **has_hidden** – Boolean flags indicating whether the slice produces logits or hidden states.
- **listen_port** – The port on which the worker accepts backward-pass connections from peers.

The function `dist_hello_to_wire()` (lines [01459‑01470](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1459-L1470)) converts these fields to network byte order using `htonl()`, then the worker transmits the frame with `dist_write_frame_header()` specifying `DS4_DIST_MSG_HELLO` as the message type.

### Coordinator Acceptance and Route Planning

On the coordinator side, the accept thread calls `dist_read_frame_header()` (lines [01439‑01455](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1439-L1455)) to read the message type and payload length. It then deserializes the HELLO payload via `dist_hello_from_wire()` (lines [01471‑01483](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1471-L1483)), which converts the network bytes back to host order.

The coordinator allocates a `ds4_dist_worker_entry` (defined at lines [00208‑00224](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L208-224)) to store the worker's file descriptor, peer address, and layer metadata. This entry is inserted into a linked list at `state->workers`, forming the **route plan** used for subsequent work dispatch. After successful insertion, the coordinator sends a HELLO response to confirm the handshake, and the worker enters its main WORK loop.

## Rolling Hash Validation for Consistency

To prevent out-of-order KV-cache updates when workers process split layers, the protocol embeds a 64-bit rolling hash of the token prefix in every work frame. Workers recompute this hash locally and reject mismatches before applying state updates.

### FNV-1a Hash Algorithm Implementation

DS4 uses a non-cryptographic **FNV-1a** hash for speed and simplicity. The implementation defines:

- `DS4_DIST_TOKEN_HASH_INIT` – The offset basis value (14695981039346656037 ULL).
- `DS4_DIST_TOKEN_HASH_PRIME` – The FNV prime (1099511628211 ULL).

These constants appear at lines [01489‑01493](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1489-L1493). The function `dist_token_hash_update()` (lines [01495‑01506](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1495-L1506)) mixes individual token IDs into the running hash, while `dist_token_hash_prefix()` (lines [01509‑01511](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1509-L1511)) computes the hash over a span of tokens by iterating the update function.

For session-wide validation, `dist_session_token_hash_prefix()` (lines [01513‑01526](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1513-L1526)) extracts the token array from a `ds4_session` object, verifies the token count exceeds the prefix length, and returns the computed 64-bit hash.

### Embedding Hashes in Work Frames

When dispatching inference work, the coordinator calls `dist_session_token_hash_prefix()` to obtain the hash of the prompt prefix the worker will process. It then splits the 64-bit value into high and low 32-bit halves using `dist_u64_to_halves()` (lines [01532‑01535](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1532-L1535)), storing them in the `prefix_hash_hi` and `prefix_hash_lo` fields of the `ds4_dist_work_fixed` structure. This frame is transmitted with message type `DS4_DIST_MSG_WORK`.

### Worker-Side Validation and Error Handling

Upon receiving a WORK frame, the worker deserializes it via `dist_work_from_wire()` and immediately validates the prefix hash. Inside `dist_worker_handle_work()` (around line [01845](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1845)), the worker computes the expected hash using its local session state and compares it against the coordinator-supplied values:

```c
uint64_t expected_prefix_hash;
if (dist_session_token_hash_prefix(session,
        work.n_tokens, &expected_prefix_hash, err, sizeof(err)) != 0 ||
    expected_prefix_hash != dist_u64_from_halves(work.prefix_hash_hi,
                                                  work.prefix_hash_lo)) {
    ds4_log(stderr, DS4_LOG_ERROR,
            "hash mismatch: worker %s expected %016lx got %016lx",
            we->peer_host, expected_prefix_hash,
            dist_u64_from_halves(work.prefix_hash_hi, work.prefix_hash_lo));
    return dist_worker_upstream_send_work_error(upstream,
            dist_u64_from_halves(work.request_hi, work.request_lo),
            "hash mismatch");
}

```

If the hashes differ, the worker logs the discrepancy and sends a `DS4_DIST_MSG_ERROR` frame upstream, aborting the work without modifying its KV cache. This prevents silent corruption from mismatched token sequences.

### Result Hash Verification

After computing outputs, workers generate a **result hash** over the generated tokens using the same FNV-1a algorithm. The function `dist_result_to_wire()` (lines [01611‑01622](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1611-L1622)) serializes the `ds4_dist_result_fixed` structure, including `result_hash_hi` and `result_hash_lo`, allowing the coordinator to verify that the worker's output matches the expected token sequence before integrating it into the global session state.

## Implementation Examples

### Worker Registration Client Side

```c
/* Build the HELLO payload */
ds4_dist_hello_fixed hello = {
    .model_id      = htonl(state->model_id),
    .quant_bits    = htonl(state->quant_bits),
    .layer_start   = htonl(state->layer_start),
    .layer_end     = htonl(state->layer_end),
    .has_output    = htonl(state->has_output),
    .has_hidden    = htonl(state->has_hidden),
    .ctx_size      = htonl(state->ctx_size),
    .n_layers      = htonl(state->n_layers),
    .listen_port   = htonl(state->listen_port),
    .model_name_len= htonl(strlen(state->model_name)),
};

/* Encode and send */
dist_hello_to_wire(&hello);
dist_write_frame_header(fd, DS4_DIST_MSG_HELLO,
                        sizeof(hello) + hello.model_name_len);
dist_write_full(fd, &hello, sizeof(hello));
dist_write_full(fd, state->model_name, hello.model_name_len);

```

*Source:* `dist_hello_to_wire()` – lines [01459‑01470](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1459-L1470)

### Coordinator Handling HELLO Registration

```c
/* After reading the frame header */
dist_hello_from_wire(&hello);
if (hello.model_id != expected_model_id) {
    ds4_log(stderr, DS4_LOG_ERROR, "unexpected model id from worker");
    dist_send_error(fd, "model mismatch");
    return;
}

ds4_dist_worker_entry *we = calloc(1, sizeof(*we));
we->fd = fd;
we->model_id = hello.model_id;
we->layer_start = hello.layer_start;
we->layer_end   = hello.layer_end;
we->has_output  = hello.has_output;
we->has_hidden  = hello.has_hidden;

/* Insert into linked list */
we->next = state->workers;
state->workers = we;

```

*Source:* Worker entry definition – lines [00208‑00224](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L208-224)

### Rolling-Hash Validation in Worker

```c
/* In dist_worker_handle_work() */
dist_work_from_wire(&work);
uint64_t expected_prefix_hash;

if (dist_session_token_hash_prefix(session,
        work.n_tokens, &expected_prefix_hash, err, sizeof(err)) != 0 ||
    expected_prefix_hash != dist_u64_from_halves(work.prefix_hash_hi,
                                                  work.prefix_hash_lo)) {
    ds4_log(stderr, DS4_LOG_ERROR,
            "hash mismatch: worker %s expected %016lx got %016lx",
            we->peer_host, expected_prefix_hash,
            dist_u64_from_halves(work.prefix_hash_hi, work.prefix_hash_lo));
    return dist_worker_upstream_send_work_error(upstream,
            dist_u64_from_halves(work.request_hi, work.request_lo),
            "hash mismatch");
}

```

*Source:* Hash comparison inside `dist_worker_handle_work()` – around line [01845](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1845)

### Result Hash Transmission

```c
/* After computing result tokens */
uint64_t result_hash = dist_token_hash_prefix(result_tokens, n_result);

ds4_dist_result_fixed res = {
    .request_hi = htonl(work.request_hi),
    .request_lo = htonl(work.request_lo),
    .result_hash_hi = htonl((uint32_t)(result_hash >> 32)),
    .result_hash_lo = htonl((uint32_t)result_hash),
    .status = htonl(0),               /* DS4_OK */
    .result_kind = htonl(DS4_DIST_RESULT_LOGITS),
    .payload_bytes = htonl(payload_bytes),
    .payload_bits  = htonl(payload_bits),
    .telemetry_count = htonl(1),
    .telemetry_bytes = htonl(sizeof(telemetry)),
};

dist_result_to_wire(&res);
dist_write_frame_header(fd, DS4_DIST_MSG_RESULT,
                        sizeof(res) + payload_bytes);
dist_write_full(fd, &res, sizeof(res));
dist_write_full(fd, payload, payload_bytes);

```

*Source:* Result conversion – `dist_result_to_wire()` – lines [01611‑01622](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1611-L1622)

## Summary

- **Worker registration** relies on a HELLO handshake over TCP, where workers advertise their layer ranges via `ds4_dist_hello_fixed` frames and the coordinator stores them in a linked list of `ds4_dist_worker_entry` structures.
- **Route planning** occurs automatically as the coordinator populates `state->workers`, mapping specific layer ranges to file descriptors for subsequent work dispatch.
- **Rolling hash validation** uses FNV-1a to compute 64-bit checksums of token prefixes, embedding these in WORK frames via `prefix_hash_hi/lo` fields.
- **Consistency enforcement** happens at the worker level; mismatches trigger immediate `DS4_DIST_MSG_ERROR` responses, preventing KV-cache corruption from out-of-order tokens.
- **Result verification** allows the coordinator to confirm worker outputs through `result_hash` fields in RESULT frames, ensuring end-to-end integrity.

## Frequently Asked Questions

### What hash algorithm does DS4 use for token validation?

DS4 implements the **FNV-1a** (Fowler-Noll-Vo) non-cryptographic hash algorithm using a 64-bit state space. The implementation uses an offset basis of 14695981039346656037 and a prime multiplier of 1099511628211, defined in [`ds4_distributed.c`](https://github.com/antirez/ds4/blob/main/ds4_distributed.c) at lines [01489‑01493](https://github.com/antirez/ds4/blob/main/ds4_distributed.c#L1489-L1493). This provides a fast, compact checksum suitable for detecting token sequence mismatches without the overhead of cryptographic hashing.

### How does the coordinator handle worker failures during registration?

If a worker sends a HELLO frame with a mismatched `model_id` or malformed data, the coordinator calls `dist_send_error()` and closes the connection immediately without inserting the worker into `state->workers`. The `dist_connect_endpoint()` function on the worker side implements exponential backoff retry logic, allowing workers to reconnect if the coordinator temporarily rejects the handshake or if network partitions occur during startup.

### What happens when a worker detects a hash mismatch?

When `dist_worker_handle_work()` detects that the locally computed prefix hash differs from the coordinator-supplied `prefix_hash_hi/lo` values, it logs the mismatch via `ds4_log()` and returns `dist_worker_upstream_send_work_error()`. This sends a `DS4_DIST_MSG_ERROR` frame upstream with the error string "hash mismatch", causing the coordinator to abort or retry the request without applying the corrupted KV state. The worker skips all inference computation for that frame.

### Can the distributed protocol support heterogeneous model slices?

Yes. The `ds4_dist_hello_fixed` structure includes fields for `quant_bits`, `layer_start`, `layer_end`, and capability flags (`has_output`, `has_hidden`), allowing workers to register different quantization levels or layer spans. The coordinator stores these capabilities in the `ds4_dist_worker_entry` and uses them to build a route plan that accounts for heterogeneous slices, though all workers must share the same base `model_id` to pass validation in `dist_hello_from_wire()`.