Parallel Twitter and Reddit Simulation Architecture in mirofish: Async Orchestration and IPC Design

The mirofish backend runs Twitter and Reddit simulations concurrently using an asyncio-driven orchestrator that launches platform-specific coroutines with asyncio.gather, wraps environments in PlatformSimulation containers, and enables post-simulation interaction via a file-based IPC mechanism.

The mirofish open-source project implements a sophisticated backend system for managing parallel simulations of Twitter and Reddit through a unified async architecture. By leveraging Python's asyncio library and a modular orchestrator design, the system can execute both social media platform simulations simultaneously while maintaining separate state, logging, and inter-process communication channels for each environment.

Async Orchestrator and Entry Point

The simulation lifecycle begins in backend/scripts/run_parallel_simulation.py, where the main() function (lines 24-78) handles CLI argument parsing and configuration loading. The orchestrator accepts a unified simulation_config.json that defines parameters for both platforms, then initializes a centralized SimulationLogManager to coordinate logging across the parallel environments.

When invoked without platform restrictions, the orchestrator determines that both Twitter and Reddit simulations should execute. Rather than running sequentially, the system uses asyncio.gather to achieve true concurrency.

Concurrent Platform Execution

The core parallel execution occurs at lines 84-89 of the orchestrator script, where the awaitable coroutines are launched simultaneously:

await asyncio.gather(
    run_twitter_simulation(config, log_manager, shutdown_event),
    run_reddit_simulation(config, log_manager, shutdown_event)
)

Each platform simulation follows an identical structural pattern implemented in backend/app/services/simulation_runner.py:

• run_twitter_simulation (lines 1011-1086): Constructs an OASIS environment via oasis.make, initializes a platform-specific SQLite database, generates an agent graph from Twitter profile configurations, and executes a round-based simulation loop with action logging and database extraction.

• run_reddit_simulation (lines 1394-1470): Mirrors the Twitter implementation but operates on Reddit-specific profiles and database schemas, maintaining complete isolation between the two simulation states.

Both functions return a PlatformSimulation container object that encapsulates the live environment and agent graph for downstream processing.

PlatformSimulation Wrapper and State Management

The PlatformSimulation class, defined at lines 217-225 in the orchestrator script, serves as the primary abstraction for managing simulation state:

class PlatformSimulation:
    def __init__(self, env, agent_graph, platform_name: str):
        self.env = env
        self.agent_graph = agent_graph
        self.platform_name = platform_name
        self.db_path = None

This wrapper preserves references to the OASIS environment (env) and the agent relationship graph (agent_graph) after the simulation loop completes. When the orchestrator runs both platforms, it instantiates two separate PlatformSimulation objects, each with its own database connection and memory space, preventing cross-platform contamination while allowing unified management.

Inter-Process Communication and Command Mode

After simulation completion, the orchestrator can remain alive in command mode (unless --no-wait is specified) to accept external instructions. At lines 304-311, the system instantiates a ParallelIPCHandler with references to both platform environments and graphs:

ipc_handler = ParallelIPCHandler(
    twitter_sim=twitter_result,
    reddit_sim=reddit_result,
    command_dir=command_dir,
    response_dir=response_dir
)

The ParallelIPCHandler class (defined in backend/app/services/simulation_ipc.py and referenced from line 217 onward in the orchestrator) implements a file-based IPC protocol:

1. Status Publication: Creates env_status.json and two directories (ipc_commands, ipc_responses) under the simulation root.
2. Command Polling: The process_commands method repeatedly scans the command folder for JSON instruction files.
3. Dispatch Logic: Parses commands such as interview, batch_interview, and close_env, routing them to the appropriate platform's ManualAction interface.
4. Response Writing: Generates response files in ipc_responses/ and removes processed commands from the queue.

Example command structure for interviewing a specific agent:

{
  "command_id": "001",
  "command_type": "interview",
  "args": {
    "agent_id": 3,
    "prompt": "What do you think about the new policy?",
    "platform": "twitter"
  }
}

Graceful Shutdown Handling

The architecture implements cooperative cancellation through a global asyncio.Event named _shutdown_event. The setup_signal_handlers() function (lines 53-78) registers handlers for SIGTERM and SIGINT that set this event, signaling all concurrent loops to terminate gracefully.

The shutdown flag propagates through three critical paths:

• The main orchestration loop checks the event between platform executions.
• The IPC polling loop in ParallelIPCHandler monitors the flag during command processing.
• The per-round simulation loops within run_twitter_simulation and run_reddit_simulation verify the event state between rounds, ensuring OASIS environments close properly without data corruption.

Practical Usage Examples

Execute both simulations in parallel with IPC mode enabled (default):

python backend/scripts/run_parallel_simulation.py \
    --config path/to/simulation_config.json

Run only the Twitter simulation and exit immediately after completion:

python backend/scripts/run_parallel_simulation.py \
    --config path/to/simulation_config.json \
    --twitter-only \
    --no-wait

Send an interview command to a running simulation:

# Create command file while orchestrator is waiting

echo '{"command_id": "001", "command_type": "interview", "args": {"agent_id": 3, "prompt": "Opinion on policy changes?", "platform": "twitter"}}' \
  > sim_dir/ipc_commands/001_interview.json

The handler processes the file, executes the interview against the Twitter OASIS environment, writes the response to sim_dir/ipc_responses/001.json, and removes the command file.

Summary

• Async Concurrency: The orchestrator uses asyncio.gather in backend/scripts/run_parallel_simulation.py (lines 84-89) to run Twitter and Reddit simulations simultaneously rather than sequentially.
• State Isolation: Each platform runs in its own PlatformSimulation wrapper (lines 217-225) with dedicated SQLite databases and agent graphs generated via backend/app/services/graph_builder.py.
• File-Based IPC: The ParallelIPCHandler enables post-simulation interaction through JSON command files in ipc_commands/ and response files in ipc_responses/, supporting operations like agent interviews and environment closures.
• Graceful Termination: Signal handlers (lines 53-78) set a global _shutdown_event that coordinates clean shutdown across the main loop, IPC polling, and per-round simulation logic.
• Modular Services: Core logic resides in backend/app/services/simulation_runner.py for platform execution and backend/app/services/simulation_ipc.py for command handling, with centralized logging via backend/app/utils/logger.py.

Frequently Asked Questions

How does mirofish prevent the Twitter and Reddit simulations from interfering with each other?

Each simulation runs in complete isolation through separate PlatformSimulation instances that encapsulate distinct OASIS environments, SQLite database connections, and agent graphs. Because run_twitter_simulation and run_reddit_simulation are independent coroutines launched via asyncio.gather, they maintain separate memory spaces and database handles, ensuring that actions in one platform never affect the state of the other.

What types of commands can be sent to running simulations via the IPC mechanism?

The ParallelIPCHandler supports three primary command types: interview (query a specific agent with a custom prompt), batch_interview (interview multiple agents simultaneously), and close_env (gracefully terminate a specific platform environment). These commands are submitted as JSON files in the ipc_commands/ directory, processed by polling loops, and responses are written to corresponding files in ipc_responses/.

Can the simulations be run independently without the parallel orchestrator?

Yes. While the parallel orchestrator in backend/scripts/run_parallel_simulation.py provides unified management, the underlying run_twitter_simulation and run_reddit_simulation functions in backend/app/services/simulation_runner.py are self-contained and can be imported and executed individually. The orchestrator simply adds concurrency management, unified configuration loading, and IPC capabilities on top of these core simulation runners.

How does the system handle unexpected shutdowns or interruptions?

The architecture implements cooperative multitasking with a global _shutdown_event asyncio Event object. Signal handlers for SIGTERM and SIGINT (lines 53-78) set this event, which is checked at regular intervals by the main orchestration loop, the IPC polling mechanism, and the per-round simulation loops. When triggered, each component begins graceful termination, ensuring OASIS environments close properly and database transactions complete before the process exits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}