# How the VoiceStudio Remote Worker Scheduler Handles Capacity, Breaker States, and Inbound-Node Mode

> Discover how the VoiceStudio remote worker scheduler manages capacity, breaker states, and inbound-node mode to prevent system overload and ensure reliable connections. Learn more.

- Repository: [Palash Debnath/VoiceStudio](https://github.com/debpalash/VoiceStudio)
- Tags: internals
- Published: 2026-09-13

---

**The VoiceStudio remote worker scheduler prevents system overload by treating capacity limits as circuit-breaker states that filter workers before strategy selection, while inbound-node mode uses epoch-based fencing to ignore stale RPC messages during reconnections.**

The `debpalash/VoiceStudio` repository provides a distributed infrastructure for voice processing workloads where remote workers must be protected from overload and connection churn. This deep dive examines how the scheduler in [`backend/worker/scheduler.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py) implements **capacity breaker states** to shield workers from repeatedly failing models, and how **inbound-node mode** maintains correctness when client panels initiate connections into the node rather than the reverse.

## Circuit-Breaker States as Hard Filters

The scheduler treats the **capacity breaker** as a circuit-breaker mechanism that guards individual workers against repeatedly attempting to execute specific models they cannot handle. This protection runs as a **hard filter** during the worker enumeration phase, meaning it executes before any user-defined placement strategy can influence the decision.

### Pre-Strategy Worker Filtering

When the scheduler enumerates eligible workers for a task, it first consults the breaker pool to verify the worker can accept the specific model workload. In [`backend/worker/scheduler.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py) at lines 1050-1057, the implementation checks the breaker state before applying any strategy logic:

```python

# backend/worker/scheduler.py

# (lines 1050-1057)

if not self.pool.breakers.allows(worker.worker_id, model_key, now=stamp):
    continue

```

Here, `model_key` represents the capacity slot key for the `(engine, model_id)` pair. The method `self.pool.breakers.allows` returns **False** when the breaker for that specific worker-model combination is open, effectively removing the worker from consideration regardless of its current load or other attributes.

### Recording Failures and Successes

After task completion, the scheduler updates the breaker state to reflect the outcome. The `record_failure` and `record_success` methods in [`backend/worker/breaker.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/breaker.py) maintain counters that determine when a breaker should open or close. Lines 912-924 in [`backend/worker/scheduler.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py) demonstrate this post-task update logic:

```python

# backend/worker/scheduler.py

# (lines 912-924)

self.pool.breakers.record_failure(
    worker.worker_id, model_key, error, now=stamp
)
…
self.pool.breakers.record_success(
    worker.worker_id,
    WorkerCapacity.slot_key(task.engine, task.model_id),
    now=stamp,
)

```

When `record_failure` detects that the failure threshold for a worker-model pair has been reached, the breaker **opens**, and subsequent tasks for that combination are filtered out by the `allows` check. Conversely, successful executions increment the success counter, potentially closing an opened breaker and restoring eligibility.

### Impact on Scheduling Decisions

Because the breaker check operates as a **hard filter** within `eligible_workers`, no user-defined strategy can override a closed breaker. Workers with open breakers become invisible to the scheduling algorithm. When all workers are filtered out, the scheduler raises `NoEligibleWorker` with `retryable=False`, forcing the task to fail fast rather than queue indefinitely behind a broken worker.

## Inbound-Node Mode and Message Integrity

**Inbound-node mode** reverses the typical connection topology, allowing a panel (remote client) to connect into a VoiceStudio node rather than the node dialing out. This mode requires additional safeguards to handle connection churn and message ordering.

### Epoch Fencing with `_fenced`

Every inbound RPC carries a **session epoch** that identifies the worker's current connection attempt. The scheduler uses the private helper `_fenced` (defined at lines 666-678 in [`backend/worker/scheduler.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py)) to discard messages from stale or replayed connections:

```python

# backend/worker/scheduler.py

# (lines 666-678)

def _fenced(self, task_id: str, attempt_id: str, epoch: Optional[int]) -> Optional[Task]:
    task = self._tasks.get(task_id)
    …
    if epoch is not None and attempt.session_epoch != epoch:
        logger.debug("Dropping message for %s from stale epoch %s", task_id, epoch)
        return None
    return task

```

This mechanism protects the system from applying "heartbeat", progress update, or result messages to the wrong task attempt when a node reconnects. All inbound callbacks—including `on_accepted`, `on_progress`, and `on_result`—first invoke `_fenced` to validate the epoch before processing the payload.

### Inbound Submission Flow

When [`backend/worker/inbound/listener.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/inbound/listener.py) receives a new task from an inbound panel, it forwards the request through the standard `submit` pathway. The scheduler still executes the full **capacity check** (including the breaker filter) for inbound tasks, ensuring that inbound-node mode does not bypass overload protection. The gRPC server implementation in the listener module handles the transport layer while delegating scheduling decisions to the core scheduler.

### Worker-Side Session Isolation

On the worker side, [`backend/worker/transport/client.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/transport/client.py) prepares inbound sessions through `prepare_inbound_session`. During an active inbound session, the client ensures that **no outbound calls** are initiated, keeping capacity bookkeeping isolated from the standard outbound flow. This separation prevents capacity accounting conflicts when a worker acts as both an inbound endpoint and outbound client.

### Breaker Logic in Inbound Context

Inbound tasks traverse the identical `eligible_workers` filter used for outbound requests. Consequently, if a specific model repeatedly fails on a node serving as an inbound endpoint, the circuit breaker opens and the scheduler stops dispatching further inbound tasks for that model-worker pair until the failure statistics clear. This unified protection ensures that inbound connections cannot destabilize workers through repeated bad assignments.

## Summary

The VoiceStudio remote worker scheduler combines circuit-breaker logic with connection-oriented fencing to provide robust task distribution:

- **Hard-filter breaker checks** in [`backend/worker/scheduler.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py) prevent overloaded workers from receiving tasks they are likely to fail, with no provision for strategy override.
- **Success and failure recording** in [`backend/worker/breaker.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/breaker.py) automatically opens and closes breakers based on runtime performance metrics.
- **Epoch fencing** via `_fenced` ensures inbound-node mode safely handles reconnections by ignoring stale RPC messages.
- **Unified eligibility logic** applies identical capacity and breaker checks to both inbound and outbound task submissions.

## Frequently Asked Questions

### What triggers a circuit breaker to open in VoiceStudio?

A circuit breaker opens when the `record_failure` method in [`backend/worker/breaker.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/breaker.py) detects that the failure threshold for a specific worker-model pair has been exceeded. Once opened, the breaker prevents that worker from receiving additional tasks for the failed model until sufficient successful executions occur to close it.

### Can user-defined scheduling strategies override the capacity breaker?

No. The breaker check operates as a **hard filter** during the `eligible_workers` enumeration phase, executing before any user-defined strategy is applied. This design ensures that system stability protections cannot be bypassed by custom scheduling logic.

### How does inbound-node mode differ from standard outbound connections?

In standard outbound mode, the VoiceStudio node initiates connections to remote panels. In **inbound-node mode**, the panel initiates the connection into the node, and the node acts as a server. This reversal requires epoch-based message fencing to protect against stale messages during reconnections, though both modes use identical capacity and breaker checks for task assignment.

### What happens when all workers are filtered out by breakers?

When the hard filter removes all eligible workers due to open breakers, the scheduler raises a `NoEligibleWorker` exception with `retryable=False`. This causes the task to fail immediately rather than remaining in the queue, alerting operators to a systemic capacity issue rather than allowing indefinite retries against broken workers.