How the III Configuration Worker Manages Dynamic Configuration with File-Based Adapters

The ConfigurationWorker in the iii-hq/iii engine delegates persistence to the FsAdapter, which stores each configuration entry as an individual YAML file and leverages the notify crate to detect external changes, automatically triggering registered functions when files are modified on disk.

The iii-hq/iii engine treats configuration as a first-class resource that supports runtime updates and automatic trigger invocation. By leveraging file-based adapters, specifically the FsAdapter, the system enables external configuration management through simple YAML files while maintaining an in-memory cache for performance. This architecture allows operators to edit configuration files directly on disk and have changes propagate immediately to running functions.

Bootstrapping the Configuration Worker

When the III engine starts, the ConfigurationWorker initializes through the create method defined in engine/src/workers/configuration/configuration.rs. This method constructs the worker using the adapter specified in engine/config.yaml, defaulting to the fs adapter if no alternative is configured.

During the initialize phase, the worker performs three critical operations:

1. Primes the in-memory cache by calling store.prime_from_adapter(), which walks the configuration directory and loads every YAML file into the ConfigurationStore.

2. Registers the trigger type so that configuration:* triggers can be attached to functions.

3. Starts the external-change watcher by invoking store.adapter().watch(tx), spawning a background task that monitors the directory for modifications.

let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<ExternalChange>();
self.store.adapter().watch(tx).await?;
tokio::spawn(async move {
    while let Some(change) = rx.recv().await {
        worker.handle_external_change(change).await;
    }
});

This unbounded channel receives ExternalChange events from the adapter and routes them to handle_external_change, ensuring the engine reacts to any external file modifications.

The FsAdapter Implementation

The file-system adapter (FsAdapter) implements the adapter contract in engine/src/workers/configuration/adapters/fs.rs, providing durable persistence and real-time change detection.

File Structure and Atomic Operations

Each configuration entry maps to a single YAML file stored at <directory>/<id>.yaml. By default, the adapter uses ./data/configuration as the root directory. The adapter maintains an internal RwLock<HashMap<String, ConfigurationEntry>> cache to ensure fast read operations.

When writing data, the adapter guarantees atomicity between disk and cache:

• register and set methods write the entire entry to disk via write_entry before updating the in-memory cache under a write lock.

• get, list, and delete methods operate primarily on the cache, with delete also removing the underlying file from disk.

External Change Detection with Notify

The watch method uses the notify crate to monitor the configuration directory for Create, Modify, and Remove events. A debounce mechanism (500 ms) prevents rapid successive triggers by snapshotting the directory, diffing it against the cached state, and emitting structured ExternalChange events.

let (raw_tx, mut raw_rx) = tokio::sync::mpsc::unbounded_channel::<()>();
let mut watcher = notify::recommended_watcher(move |res| {
    if let Ok(event) = res {
        if matches!(event.kind, EventKind::Modify(_) | EventKind::Create(_) | EventKind::Remove(_)) {
            let _ = raw_tx.send(());
        }
    }
})?;
watcher.watch(&directory, RecursiveMode::NonRecursive)?;

The watcher runs continuously until the destroy method clears it, transforming any external file edit into an ExternalChange enum variant (Registered, Updated, or Deleted).

Propagating Changes to Triggers

When ConfigurationWorker::handle_external_change receives an event from the adapter, it applies the change to the store via store.apply_external(&change) and converts the raw change into a ConfigurationEventData structure.

self.store.apply_external(&change).await;
let event = match change {
    ExternalChange::Registered(entry) => entry_to_event(entry, ConfigurationEventType::Registered, None, Some(entry.value.clone())),
    // Updated and Deleted variants handled similarly
};
self.fan_out(event).await;

The fan_out method lookup triggers matching the configuration ID, evaluates any conditional logic, and invokes the target functions with the event payload. This mechanism enables configuration triggers to fire automatically when a YAML file is edited on disk, creating a seamless bridge between file-system operations and function execution.

Configuration Lifecycle and TTL

Beyond external edits, the worker supports TTL-based expiration through the expire_configuration method. When a configuration's TTL expires, the worker deletes the entry from both the cache and the file system, then emits a configuration:deleted event via the trigger system. This ensures that temporary configurations do not persist indefinitely, maintaining clean state management across the engine.

Practical Implementation Examples

Enabling the File Adapter

Configure the engine to use the file-based adapter by specifying it in engine/config.yaml:

configuration:
  ttl_seconds: 0
  adapter:
    name: "fs"
    config:
      directory: "./data/configuration"

Registering Configuration via SDK

Use the Node.js SDK to register a configuration entry that persists as a YAML file:

import { iii } from "iii-sdk";

await iii.registerFunction({
  id: "my-config::on_change",
  description: "Handle config updates",
});

await iii.registerTrigger({
  type: "configuration",
  function_id: "my-config::on_change",
  config: { id: "my-service", event_types: ["configuration:updated"] },
});

await iii.call("configuration::register", {
  id: "my-service",
  name: "My Service",
  description: "Runtime configuration for My Service",
  schema: { type: "object", properties: { port: { type: "integer" } } },
  initial_value: { port: 8080 },
});

This creates my-service.yaml in the configured directory with the initial value.

Reacting to External Edits

Modify the file directly using standard shell commands:

echo "port: 9090" > ./data/configuration/my-service.yaml

The notify watcher detects the change, the worker updates its cache, and the registered trigger fires immediately:

async function on_change(event) {
  console.log("Configuration updated:", event.new_value);
}

Summary

• The ConfigurationWorker coordinates dynamic configuration by delegating persistence to swappable adapters, with FsAdapter providing file-based storage in engine/src/workers/configuration/adapters/fs.rs.

• FsAdapter stores each entry as an individual YAML file in ./data/configuration and maintains an in-memory RwLock cache for performance.

• The notify crate powers real-time change detection with a 500 ms debounce, converting file system events into structured ExternalChange messages.

• Atomic write operations guarantee consistency between disk and cache by writing files via write_entry before updating memory.

• The trigger system automatically invokes functions when configuration changes through fan_out, bridging external file edits with internal engine events.

• TTL support enables automatic cleanup of expired configurations from both cache and disk via expire_configuration.

Frequently Asked Questions

How does the ConfigurationWorker detect changes to configuration files?

The worker delegates change detection to the active adapter's watch method. In FsAdapter, this uses the notify crate to subscribe to Create, Modify, and Remove events on the configuration directory. A 500 millisecond debounce prevents excessive trigger firings, and the adapter diffs the directory state against its internal cache to generate precise ExternalChange events.

What happens if two processes modify the same configuration file simultaneously?

The FsAdapter employs atomic write operations where changes are persisted to disk via write_entry before updating the in-memory cache. While the underlying file system handles atomicity at the OS level, the last write will prevail in the cache. The notify watcher will detect the final state and propagate a single ExternalChange::Updated event to the worker.

Can I use a custom directory for configuration storage?

Yes. The directory path is configurable in engine/config.yaml under the adapter.config.directory key. The FsAdapter::new method creates the specified directory (defaulting to ./data/configuration) during initialization if it does not exist.

How do I configure automatic cleanup of temporary configurations?

Set the ttl_seconds field in engine/config.yaml to a non-zero value, or specify a TTL when registering configuration entries via the SDK. When the TTL expires, the worker's expire_configuration method automatically deletes the entry from both the in-memory cache and the underlying YAML file, emitting a configuration:deleted event to any registered triggers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}