How to Configure iii Engine with Custom Workers and Modules in config.yaml

Configure the iii engine by defining modules and workers arrays in engine/config.yaml, then register custom implementations via EngineBuilder::register_worker() before calling build().

The iii engine (from the iii-hq/iii repository) boots by deserializing a single YAML configuration file that declares both built-in workers and custom Rust modules. Understanding how to structure this file and wire it into the EngineBuilder API is essential for extending the engine’s processing capabilities.

Understanding the config.yaml Structure

The engine expects a top-level configuration object with two arrays: modules and workers. According to the source in src/workers/config.rs【L28-L35】, these arrays deserialize into the EngineConfig struct:

pub struct EngineConfig {
    pub modules: Vec<WorkerEntry>,
    pub workers: Vec<WorkerEntry>,
}

Each entry follows the WorkerEntry structure【L70-L77】, which supports:

• name: The worker identifier (e.g., my::CustomWorker or iii-http)
• image: Optional external binary image (for non-built-in workers)
• config: Optional JSON/YAML configuration object passed to the worker’s create method

The loader expands ${ENV} variables during deserialization【L78-L101】 and merges both arrays into a single internal list before instantiation.

Creating a Custom Worker Module

To register a custom module, you must implement the Worker trait and register it with the EngineBuilder before loading the configuration file.

Implement the Worker Trait

Define your module in Rust by implementing the async Worker trait from src/workers/traits.rs:

use async_trait::async_trait;
use serde_json::Value;
use std::sync::Arc;
use iii::engine::Engine;
use iii::workers::traits::Worker;

#[derive(Debug)]
struct AnalyticsProcessor;

#[async_trait]
impl Worker for AnalyticsProcessor {
    fn name(&self) -> &'static str {
        "my::AnalyticsProcessor"
    }

    async fn create(
        _engine: Arc<Engine>,
        config: Option<Value>
    ) -> anyhow::Result<Box<dyn Worker>> {
        // Config block from config.yaml arrives here as JSON
        if let Some(cfg) = config {
            println!("Initializing with config: {}", cfg);
        }
        Ok(Box::new(AnalyticsProcessor))
    }

    async fn initialize(&self) -> anyhow::Result<()> {
        Ok(())
    }
}

Register and Build

Use EngineBuilder::register_worker() to map the name to your implementation, then load the YAML file:

use iii::engine::EngineBuilder;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    EngineBuilder::new()
        .register_worker::<AnalyticsProcessor>("my::AnalyticsProcessor")
        .config_file("engine/config.yaml")?
        .build()
        .await?
        .serve()
        .await
}

The WorkerRegistry (populated via the inventory macro for built-ins and manual registration for customs) invokes create_worker【L86-L108】 to instantiate your module when it encounters the matching name in config.yaml.

Configuring Workers in config.yaml

The YAML file distinguishes between modules (Rust/TS/Python implementations) and workers (built-in or external processes).

Basic Configuration Example

modules:
  - name: "my::AnalyticsProcessor"
    config:
      batch_size: 1000
      flush_interval_sec: 30

workers:
  - name: iii-http
    config:
      port: 8080
      host: "0.0.0.0"

  - name: iii-queue
    config:
      concurrency: 4

Key rules:

• Built-in workers (like iii-http) must not specify an image field; the engine errors if they do.
• Custom modules listed under modules resolve to types registered via register_worker().
• The config block is passed verbatim to the worker’s create method as a serde_json::Value.

External Worker Binaries

To override a built-in worker or add an external binary, provide a unique name and an image field:

workers:
  - name: "custom::QueueWorker"
    image: "docker.io/tenant/custom-queue:1.2.3"
    config:
      redis_url: "redis://localhost:6379"

The engine launches external images via iii-worker start as implemented in WorkerRegistry::create_worker.

Programmatic Configuration Without YAML

If you prefer compile-time configuration, bypass config_file() and use add_worker():

EngineBuilder::new()
    .register_worker::<AnalyticsProcessor>("my::AnalyticsProcessor")
    .add_worker(
        "my::AnalyticsProcessor",
        Some(serde_json::json!({
            "batch_size": 500,
            "flush_interval_sec": 60
        }))
    )
    .default_config()  // Loads built-in defaults for unspecified workers
    .build()
    .await?
    .serve()
    .await

This approach is equivalent to a YAML entry but hardcoded in the binary, useful for testing or locked-down deployments.

Summary

• The iii engine reads modules and workers arrays from engine/config.yaml into EngineConfig defined in src/workers/config.rs.
• Each entry is a WorkerEntry containing a name, optional image, and configuration object.
• Register custom Rust modules via EngineBuilder::register_worker() before calling config_file() or add_worker().
• Built-in workers resolve through WorkerRegistry using the inventory macro; custom workers require explicit registration.
• External binaries require an image field and a unique name distinct from built-in worker IDs.

Frequently Asked Questions

What is the difference between the modules and workers arrays in config.yaml?

Both arrays deserialize into Vec<WorkerEntry> and are merged during engine startup. By convention, modules holds custom Rust/TS/Python implementations you register via EngineBuilder::register_worker(), while workers holds built-in engine workers (like iii-http) or external binary images. Functionally, they behave identically after the merge step in src/workers/config.rs【L81-L86】.

How do I use environment variables in config.yaml?

The engine automatically expands ${ENV_VAR} syntax during YAML deserialization. For example, port: ${PORT} resolves to the value of the PORT environment variable before the configuration is validated. This expansion logic is implemented in EngineConfig::config_file【L78-L101】.

Can I register a custom worker written in TypeScript or Python?

Yes, provided the engine binary includes the appropriate runtime bindings. You must register the foreign worker type through EngineBuilder::register_worker() using a Rust wrapper that implements the Worker trait and delegates to the TS/Python runtime. The config.yaml entry resembles a native Rust module with the name matching your registered identifier.

Why does the engine fail when I specify an image for a built-in worker?

Built-in workers (such as iii-http or iii-queue) are compiled into the engine binary and instantiated via factory functions in WorkerRegistry. Specifying an image field triggers the external binary boot logic, which is incompatible with built-in types. Use the image field only for custom external workers with unique names.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}