How to Debug Module RPC Failures in DimOS When Using Forkserver Worker Processes

Enable DIMOS_LOG_LEVEL=DEBUG to capture worker-side tracebacks, verify your methods are decorated with @rpc, and inspect Worker.pid to detect crashed forkserver processes before they cause cryptic pipe errors.

DimOS (dimensionalOS/dimos) isolates compute-heavy modules in separate processes using Python’s forkserver multiprocessing context to prevent CUDA context corruption. When modules expose methods via the @rpc decorator, calls cross a process boundary through pipes, creating failure modes ranging from AttributeError to BrokenPipeError that require systematic diagnosis of the worker lifecycle and serialization stack.

How RPC Works in DimOS Forkserver Workers

Understanding the request flow is essential for pinpointing where failures occur.

Module Registration and the @rpc Decorator

In dimos/core/module.py, the ModuleBase class collects RPC-exposed methods via the @rpc decorator (defined in dimos/core/core.py). When a module initializes, it creates an LCMRPC transport and calls rpc.serve_module_rpc(self), which makes the module’s rpcs dictionary (built in ModuleBase.rpcs at lines 19-27) reachable from other processes. Only methods present in this dictionary can be invoked remotely.

The Actor-to-Worker Request Flow

The RPC path spans two processes:

1. Parent process – An Actor proxy serializes method calls into request dictionaries ({"type":"call_method", ...}) and sends them through a pipe via Actor._send_request_to_worker (lines 93-108 in dimos/core/worker.py).
2. Worker process – The _worker_loop (lines 34-42 in dimos/core/worker.py) receives the dict, resolves the target module from its instances map, invokes the method, and marshals back a {result, error} response.

All workers are spawned via multiprocessing.get_context("forkserver") inside get_forkserver_context (lines 28-33 in dimos/core/worker.py). This separate process boundary is where most RPC failures surface.

Common RPC Failure Modes in Forkserver Workers

Symptom	Root Cause	Diagnostic Location
AttributeError: method not found	Method missing @rpc decorator or module initialization failed before RPC registration.	Actor._send_request_to_worker error response (lines 101-107).
BrokenPipeError / EOFError	Worker process died (crash, deadlock, or unhandled exception) or pipe closed during shutdown.	Pipe send/recv in Actor._send_request_to_worker.
Timeout / hanging call	Worker blocked inside the RPC method (long-running blocking call or deadlocked lock).	Parent waiting indefinitely on conn.recv().
RuntimeError: Worker error	Unhandled exception inside _worker_loop caught and marshaled back as "error".	Response handling in Actor._send_request_to_worker.
PicklingError	Arguments or return values are not pickle-serializable.	Serialization layer in dimos/protocol/rpc.py.

Step-by-Step Debugging Workflow

Follow this sequence to isolate and fix RPC failures in forkserver workers.

1. Enable Detailed Logging

Set the environment variable DIMOS_LOG_LEVEL=DEBUG before launching your blueprint, or call setup_logger(level="DEBUG") from dimos.utils.logging_config. This captures the exact traceback from the worker side when _worker_loop catches an exception, printing logger.error("Worker process error: …") with the full stack trace.

2. Verify RPC Method Registration

Inside a running module, call self.get_rpc_method_names() (exposed via @rpc at ModuleBase.get_rpc_method_names, lines 48-50 in dimos/core/module.py). If your method is missing, ensure it uses the @rpc decorator and that __init__ completed without raising.

3. Check Worker Health

A None PID indicates the forkserver process died:

from dimos.core.worker import Worker

print("Worker PID:", w.pid)          # None → worker died

print("Modules loaded:", w.module_names)

The Worker.pid property (lines 61-73 in dimos/core/worker.py) returns None when the process terminates unexpectedly.

4. Inspect and Reset Forkserver Context

Stale forkserver state in tests causes “cannot start new process” errors. Call reset_forkserver_context() (lines 35-39 in dimos/core/worker.py) before creating a new Worker to ensure a fresh process context.

5. Reproduce in Isolation

Create a minimal script spawning a single worker to isolate the failure from blueprint wiring:

from dimos.core.worker import Worker
from dimos.core.module import Module

class TestModule(Module):
    @rpc
    def echo(self, msg):
        return msg

w = Worker()
w.start_process()
actor = w.deploy_module(TestModule)
print(actor.echo("test").result())  # Should return "test"

w.shutdown()

6. Validate Serialization

RPC arguments and return values must be picklable (or JSON-serializable for LCMRPC). If you see RuntimeError: Worker error: PicklingError, simplify the payload or implement a custom serializer in your RPC transport defined in dimos/protocol/rpc.py.

7. Guard Against Deadlocks

Modules often share a threading.Lock (self._lock) when sending requests. Never acquire the same lock across parent ↔ worker boundaries, such as calling a blocking RPC from within a module’s own thread that already holds the lock.

8. Validate Startup Order

The worker process must be started via Worker.start_process() before deploying any modules. Attempting to deploy a module before calling start_process raises “Worker process not started”.

Diagnostic Code Examples

Listing Available RPC Methods

Verify a target method is registered before invoking it:

from dimos.core.blueprints import autoconnect
from dimos.robot.unitree.unitree_skill_container import UnitreeSkillContainer

bp = autoconnect(UnitreeSkillContainer())
actor = bp.build().run()

skill_mod = actor.unitree_skill_container  # attribute name matches class name

print("Available RPCs:", skill_mod.get_rpc_method_names())

# Output: ['start', 'stop', 'move', 'set_pose', ...]

Reference: ModuleBase.get_rpc_method_names in dimos/core/module.py.

Capturing Worker Tracebacks with Debug Logging

export DIMOS_LOG_LEVEL=DEBUG
dimos run unitree-go2-agentic

When an RPC fails, the logs reveal the exact line:

2026-03-15 12:34:56,789 | ERROR | Worker process error: ValueError: invalid command
Traceback (most recent call last):
  File ".../dimos/core/worker.py", line 84, in _worker_loop
    result = method(*request.get("args", ()), **request.get("kwargs", {}))
ValueError: invalid command

Resetting Forkserver Context in Tests

Prevent cross-test pollution by resetting the forkserver between cases:

import pytest
from dimos.core.worker import reset_forkserver_context, Worker

@pytest.fixture(autouse=True)
def clean_forkserver():
    reset_forkserver_context()
    yield
    reset_forkserver_context()

def test_rpc():
    w = Worker()
    w.start_process()
    actor = w.deploy_module(SomeModule)
    result = actor.some_rpc_method().result()
    assert result == "expected"
    w.shutdown()

Reference: reset_forkserver_context in dimos/core/worker.py.

Detecting and Restarting Dead Workers

from dimos.core.worker import Worker

w = Worker()
w.start_process()
actor = w.deploy_module(MyModule)

try:
    actor.crash().result()
except RuntimeError as e:
    print("Worker error:", e)

if w.pid is None:
    print("Worker died; restarting...")
    w.shutdown()
    w = Worker()
    w.start_process()
    actor = w.deploy_module(MyModule)

Summary

• Enable debug logging via DIMOS_LOG_LEVEL=DEBUG to view worker-side tracebacks in dimos/core/worker.py.
• Verify method exposure using get_rpc_method_names() to confirm the @rpc decorator registered the method in ModuleBase.rpcs.
• Monitor process health by checking Worker.pid; a None value indicates the forkserver process crashed.
• Reset state between tests with reset_forkserver_context() to avoid stale process errors.
• Ensure serialization compatibility for all arguments and return values crossing the LCMRPC transport layer.

Frequently Asked Questions

What causes "AttributeError: method not found" in DimOS RPC calls?

This error occurs when the requested method is not present in the module’s rpcs dictionary, either because the method lacks the @rpc decorator defined in dimos/core/core.py or the module’s __init__ raised an exception before completing registration. Call get_rpc_method_names() to verify the method appears in the RPC surface.

How do I detect if a forkserver worker process has crashed?

Check the Worker.pid property. If it returns None, the process has terminated. You can also catch BrokenPipeError or EOFError when attempting to send requests through Actor._send_request_to_worker, as these indicate the pipe closed because the worker died.

Why do I see BrokenPipeError when calling module methods?

BrokenPipeError indicates the worker process died unexpectedly (segfault, unhandled exception, or explicit os._exit) or the pipe was closed during shutdown. Enable DIMOS_LOG_LEVEL=DEBUG to capture the traceback from _worker_loop that preceded the crash, then inspect the worker-side code for blocking operations or resource leaks.

Can I reset the forkserver context between test cases?

Yes. Import reset_forkserver_context from dimos.core.worker and invoke it in a pytest fixture before and after each test. This clears stale process state and prevents "cannot start new process" errors caused by lingering forkserver resources from previous test runs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}