# How to Debug Module RPC Failures in DimOS When Using Forkserver Worker Processes

> Debug module RPC failures in DimOS with forkserver workers. Enable debug logs, verify RPC decorators, and check worker PIDs for crashed processes to resolve cryptic pipe errors.

- Repository: [Dimensional/dimos](https://github.com/dimensionalOS/dimos)
- Tags: how-to-guide
- Published: 2026-03-15

---

**Enable `DIMOS_LOG_LEVEL=DEBUG` to capture worker-side tracebacks, verify your methods are decorated with `@rpc`, and inspect `Worker.pid` to detect crashed forkserver processes before they cause cryptic pipe errors.**

DimOS (dimensionalOS/dimos) isolates compute-heavy modules in separate processes using Python’s `forkserver` multiprocessing context to prevent CUDA context corruption. When modules expose methods via the `@rpc` decorator, calls cross a process boundary through pipes, creating failure modes ranging from `AttributeError` to `BrokenPipeError` that require systematic diagnosis of the worker lifecycle and serialization stack.


## How RPC Works in DimOS Forkserver Workers

Understanding the request flow is essential for pinpointing where failures occur.

### Module Registration and the @rpc Decorator

In [`dimos/core/module.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/module.py), the `ModuleBase` class collects RPC-exposed methods via the `@rpc` decorator (defined in [`dimos/core/core.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/core.py)). When a module initializes, it creates an **LCMRPC** transport and calls `rpc.serve_module_rpc(self)`, which makes the module’s `rpcs` dictionary (built in `ModuleBase.rpcs` at lines 19-27) reachable from other processes. Only methods present in this dictionary can be invoked remotely.

### The Actor-to-Worker Request Flow

The RPC path spans two processes:

1. **Parent process** – An `Actor` proxy serializes method calls into request dictionaries (`{"type":"call_method", ...}`) and sends them through a pipe via `Actor._send_request_to_worker` (lines 93-108 in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py)).
2. **Worker process** – The `_worker_loop` (lines 34-42 in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py)) receives the dict, resolves the target module from its `instances` map, invokes the method, and marshals back a `{result, error}` response.

All workers are spawned via `multiprocessing.get_context("forkserver")` inside `get_forkserver_context` (lines 28-33 in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py)). This separate process boundary is where most RPC failures surface.


## Common RPC Failure Modes in Forkserver Workers

| Symptom | Root Cause | Diagnostic Location |
|---------|------------|---------------------|
| **AttributeError: method not found** | Method missing `@rpc` decorator or module initialization failed before RPC registration. | `Actor._send_request_to_worker` error response (lines 101-107). |
| **BrokenPipeError / EOFError** | Worker process died (crash, deadlock, or unhandled exception) or pipe closed during shutdown. | Pipe send/recv in `Actor._send_request_to_worker`. |
| **Timeout / hanging call** | Worker blocked inside the RPC method (long-running blocking call or deadlocked lock). | Parent waiting indefinitely on `conn.recv()`. |
| **RuntimeError: Worker error** | Unhandled exception inside `_worker_loop` caught and marshaled back as `"error"`. | Response handling in `Actor._send_request_to_worker`. |
| **PicklingError** | Arguments or return values are not pickle-serializable. | Serialization layer in [`dimos/protocol/rpc.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/protocol/rpc.py). |


## Step-by-Step Debugging Workflow

Follow this sequence to isolate and fix RPC failures in forkserver workers.

### 1. Enable Detailed Logging

Set the environment variable `DIMOS_LOG_LEVEL=DEBUG` before launching your blueprint, or call `setup_logger(level="DEBUG")` from `dimos.utils.logging_config`. This captures the exact traceback from the worker side when `_worker_loop` catches an exception, printing `logger.error("Worker process error: …")` with the full stack trace.

### 2. Verify RPC Method Registration

Inside a running module, call `self.get_rpc_method_names()` (exposed via `@rpc` at `ModuleBase.get_rpc_method_names`, lines 48-50 in [`dimos/core/module.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/module.py)). If your method is missing, ensure it uses the `@rpc` decorator and that `__init__` completed without raising.

### 3. Check Worker Health

A `None` PID indicates the forkserver process died:

```python
from dimos.core.worker import Worker

print("Worker PID:", w.pid)          # None → worker died

print("Modules loaded:", w.module_names)

```

The `Worker.pid` property (lines 61-73 in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py)) returns `None` when the process terminates unexpectedly.

### 4. Inspect and Reset Forkserver Context

Stale forkserver state in tests causes “cannot start new process” errors. Call `reset_forkserver_context()` (lines 35-39 in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py)) before creating a new `Worker` to ensure a fresh process context.

### 5. Reproduce in Isolation

Create a minimal script spawning a single worker to isolate the failure from blueprint wiring:

```python
from dimos.core.worker import Worker
from dimos.core.module import Module

class TestModule(Module):
    @rpc
    def echo(self, msg):
        return msg

w = Worker()
w.start_process()
actor = w.deploy_module(TestModule)
print(actor.echo("test").result())  # Should return "test"

w.shutdown()

```

### 6. Validate Serialization

RPC arguments and return values must be picklable (or JSON-serializable for LCMRPC). If you see `RuntimeError: Worker error: PicklingError`, simplify the payload or implement a custom serializer in your RPC transport defined in [`dimos/protocol/rpc.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/protocol/rpc.py).

### 7. Guard Against Deadlocks

Modules often share a `threading.Lock` (`self._lock`) when sending requests. Never acquire the same lock across parent ↔ worker boundaries, such as calling a blocking RPC from within a module’s own thread that already holds the lock.

### 8. Validate Startup Order

The worker process must be started via `Worker.start_process()` before deploying any modules. Attempting to deploy a module before calling `start_process` raises “Worker process not started”.


## Diagnostic Code Examples

### Listing Available RPC Methods

Verify a target method is registered before invoking it:

```python
from dimos.core.blueprints import autoconnect
from dimos.robot.unitree.unitree_skill_container import UnitreeSkillContainer

bp = autoconnect(UnitreeSkillContainer())
actor = bp.build().run()

skill_mod = actor.unitree_skill_container  # attribute name matches class name

print("Available RPCs:", skill_mod.get_rpc_method_names())

# Output: ['start', 'stop', 'move', 'set_pose', ...]

```

Reference: `ModuleBase.get_rpc_method_names` in [`dimos/core/module.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/module.py).

### Capturing Worker Tracebacks with Debug Logging

```bash
export DIMOS_LOG_LEVEL=DEBUG
dimos run unitree-go2-agentic

```

When an RPC fails, the logs reveal the exact line:

```

2026-03-15 12:34:56,789 | ERROR | Worker process error: ValueError: invalid command
Traceback (most recent call last):
  File ".../dimos/core/worker.py", line 84, in _worker_loop
    result = method(*request.get("args", ()), **request.get("kwargs", {}))
ValueError: invalid command

```

### Resetting Forkserver Context in Tests

Prevent cross-test pollution by resetting the forkserver between cases:

```python
import pytest
from dimos.core.worker import reset_forkserver_context, Worker

@pytest.fixture(autouse=True)
def clean_forkserver():
    reset_forkserver_context()
    yield
    reset_forkserver_context()

def test_rpc():
    w = Worker()
    w.start_process()
    actor = w.deploy_module(SomeModule)
    result = actor.some_rpc_method().result()
    assert result == "expected"
    w.shutdown()

```

Reference: `reset_forkserver_context` in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py).

### Detecting and Restarting Dead Workers

```python
from dimos.core.worker import Worker

w = Worker()
w.start_process()
actor = w.deploy_module(MyModule)

try:
    actor.crash().result()
except RuntimeError as e:
    print("Worker error:", e)

if w.pid is None:
    print("Worker died; restarting...")
    w.shutdown()
    w = Worker()
    w.start_process()
    actor = w.deploy_module(MyModule)

```


## Summary

- **Enable debug logging** via `DIMOS_LOG_LEVEL=DEBUG` to view worker-side tracebacks in [`dimos/core/worker.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/worker.py).
- **Verify method exposure** using `get_rpc_method_names()` to confirm the `@rpc` decorator registered the method in `ModuleBase.rpcs`.
- **Monitor process health** by checking `Worker.pid`; a `None` value indicates the forkserver process crashed.
- **Reset state between tests** with `reset_forkserver_context()` to avoid stale process errors.
- **Ensure serialization compatibility** for all arguments and return values crossing the LCMRPC transport layer.


## Frequently Asked Questions

### What causes "AttributeError: method not found" in DimOS RPC calls?

This error occurs when the requested method is not present in the module’s `rpcs` dictionary, either because the method lacks the `@rpc` decorator defined in [`dimos/core/core.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/core.py) or the module’s `__init__` raised an exception before completing registration. Call `get_rpc_method_names()` to verify the method appears in the RPC surface.

### How do I detect if a forkserver worker process has crashed?

Check the `Worker.pid` property. If it returns `None`, the process has terminated. You can also catch `BrokenPipeError` or `EOFError` when attempting to send requests through `Actor._send_request_to_worker`, as these indicate the pipe closed because the worker died.

### Why do I see BrokenPipeError when calling module methods?

`BrokenPipeError` indicates the worker process died unexpectedly (segfault, unhandled exception, or explicit `os._exit`) or the pipe was closed during shutdown. Enable `DIMOS_LOG_LEVEL=DEBUG` to capture the traceback from `_worker_loop` that preceded the crash, then inspect the worker-side code for blocking operations or resource leaks.

### Can I reset the forkserver context between test cases?

Yes. Import `reset_forkserver_context` from `dimos.core.worker` and invoke it in a pytest fixture before and after each test. This clears stale process state and prevents "cannot start new process" errors caused by lingering forkserver resources from previous test runs.