iii-State vs iii-Queue Workers: Choosing the Right Engine for Stateful Workflows

iii-state provides a distributed key-value store with reactive triggers for immediate data consistency, while iii-queue offers an asynchronous job queue with retry logic and ordering guarantees for reliable background processing.

The iii-hq/iii engine provides two distinct built-in workers for managing stateful workflows: iii-state and iii-queue. While both handle data and execution flow, they serve fundamentally different architectural purposes—one acts as a reactive data store, the other as a deferred execution system. Understanding the difference between iii-state and iii-queue workers ensures you select the right primitives for consistency, reactivity, and scalability in your applications.

Core Architectural Differences

Data Model and Persistence Layer

iii-state implements a distributed key-value store where data persists as arbitrary JSON values under scoped keys (<scope>/<key>). According to the source documentation in docs/0-11-0/workers/iii-state.mdx, values support atomic operations including set, update, delete, and list, with persistence handled by pluggable adapters such as in_memory, file_based, or redis.

iii-queue functions as an asynchronous job queue that schedules function calls for later processing. As detailed in docs/0-11-0/workers/iii-queue.mdx, jobs are transient payload objects attached to named queues or topics, with no persistent data stored beyond the job lifecycle itself. The queue supports both topic-based (pub/sub) and named-queue modes, configurable via queue_configs that define max retries, concurrency limits, FIFO ordering, and dead-letter queues.

Reactivity and Trigger Semantics

The workers differ fundamentally in how they notify your application of changes. iii-state fires state triggers server-side whenever a value is created, updated, or deleted. Handlers registered in engine/src/workers/state/state.rs receive detailed event contexts including event_type, old_value, and new_value, enabling immediate reactive workflows.

iii-queue triggers fire only when a job is dequeued and the target function invoked. There is no built-in notification mechanism for data changes; you must explicitly emit a job if you want downstream reactions. This makes iii-queue ideal for fire-and-forget operations rather than reactive state propagation.

Consistency Guarantees

iii-state provides atomic consistency per operation. The state::update function supports atomic multi-operation transactions with validation and per-operation error reporting, ensuring that state changes are immediately visible across all workers sharing the same adapter.

iii-queue offers at-least-once delivery semantics with configurable retry logic. Jobs are eventually processed with exponential back-off, and FIFO queues enforce strict ordering guarantees for series of related operations. The implementation in engine/src/workers/queue/queue.rs handles dequeue logic, retry counting, and dead-letter queue routing.

When to Use iii-State for Reactive State Management

Use iii-state when your workflow requires immediate reaction to data changes or shared mutable state that multiple functions must observe. Typical scenarios include user profile caches, feature flags, session storage, or any data where consistency and reactivity outweigh the need for background processing delays.

import { registerWorker, TriggerAction } from "iii-sdk";

const worker = registerWorker(process.env.III_URL!, {
  workerName: "profile-service",
});

// Store reactive state
await worker.trigger({
  function_id: "state::set",
  payload: {
    scope: "users",
    key: "user-123",
    value: { name: "Alice", email: "alice@example.com", theme: "dark" },
  },
  action: TriggerAction.Void(),
});

// React immediately to value changes
const fn = worker.registerFunction(
  { id: "state::onProfileChange" },
  async (event) => {
    console.log("Profile changed:", event.event_type, event.key);
    
    if (
      event.event_type === "state:updated" &&
      event.old_value?.email !== event.new_value?.email
    ) {
      await sendVerificationEmail(event.new_value.email);
    }
    return {};
  },
);

worker.registerTrigger({
  type: "state",
  function_id: fn.id,
  config: { scope: "users", key: "user-123" },
});

This pattern, demonstrated in sdk/packages/node/iii/tests/state.test.ts, ensures that your application responds synchronously to state mutations.

When to Use iii-Queue for Asynchronous Processing

Use iii-queue for background tasks that must survive process restarts, require retry logic, or need to defer execution to prevent blocking critical paths. Ideal use cases include email delivery, image processing, payment handling, or ledger entries requiring strict FIFO ordering.

import { registerWorker, TriggerAction } from "iii-sdk";

const worker = registerWorker(process.env.III_URL!, {
  workerName: "email-service",
});

// Enqueue background job with retry configuration
await worker.trigger({
  function_id: "email::sendVerification",
  payload: { userId: "user-123", email: "alice@example.com" },
  action: TriggerAction.Enqueue({ queue: "email" }),
});

// Consumer processes asynchronously with automatic retries
const emailFn = worker.registerFunction(
  { id: "email::sendVerification" },
  async (payload) => {
    await sendVerificationEmail(payload.email);
    return {};
  },
);

The queue adapter configuration determines scalability: the builtin adapter works only for single engine instances, while the rabbitmq or redis adapters enable distributed queue sharing across multiple instances.

Combining Both Workers in Stateful Workflows

Production stateful workflows typically require both primitives working in concert. Use iii-state to maintain the authoritative state and trigger immediate reactions, then use iii-queue to handle side effects that require reliability guarantees without blocking the state update.

const worker = registerWorker(process.env.III_URL!, {
  workerName: "stateful-workflow",
});

// React to state changes by enqueueing background work
const stateHandler = worker.registerFunction(
  { id: "state::onProfileChange" },
  async (event) => {
    if (event.event_type === "state:updated") {
      // State is already persisted atomically
      // Now schedule reliable background processing
      await worker.trigger({
        function_id: "email::sendVerification",
        payload: { 
          userId: event.key, 
          email: event.new_value.email 
        },
        action: TriggerAction.Enqueue({ 
          queue: "email",
          // Configurable in queue_configs: max_retries, fifo, etc.
        }),
      });
    }
    return {};
  },
);

worker.registerTrigger({
  type: "state",
  function_id: stateHandler.id,
  config: { scope: "users", key: "user-123" },
});

This pattern, documented in docs/tutorials/incremental-adoption/offload-to-queue.mdx, separates concerns: iii-state guarantees atomic persistence while iii-queue guarantees reliable, retryable delivery of side effects.

Configuration and Scalability Considerations

State scalability depends on scoping and adapter choice. State is sharded by scope, with all workers seeing the same data through the configured adapter. The redis adapter enables multi-instance state sharing, while in_memory or file_based adapters suit single-instance deployments.

Queue scalability depends on queue modes and adapter selection. Topic-based queues provide fan-out capabilities for broadcasting to multiple independent workers, while named queues with FIFO enabled enforce strict ordering for financial or audit workflows. The rabbitmq adapter provides true distributed queue semantics, whereas the builtin adapter restricts queues to a single engine instance.

Reference implementations in engine/src/workers/state/state.rs and engine/src/workers/queue/queue.rs demonstrate how the engine coordinates persistence and trigger firing respectively.

Summary

• iii-state maintains reactive, consistent key-value data with immediate trigger firing, suitable for caching, configuration, and session management.
• iii-queue provides durable, retryable background job processing with optional FIFO ordering, ideal for side effects and long-running operations.
• Combined usage patterns leverage state triggers to enqueue jobs, separating atomic data persistence from reliable execution.
• Adapter selection determines scalability: Redis adapters enable multi-instance deployments for both workers, while builtin adapters suit single-node scenarios.
• Consistency models differ fundamentally: iii-state offers atomic updates; iii-queue provides at-least-once delivery with configurable retry policies.

Frequently Asked Questions

Can I use iii-queue without iii-state for stateful workflows?

You can implement basic stateful workflows using only iii-queue by storing state within job payloads and chaining jobs together. However, this approach lacks reactive triggers and atomic update semantics. Without iii-state, your workflow cannot observe data changes in real-time or guarantee consistency during concurrent updates. For true stateful workflows requiring reactive patterns, combine both workers as shown in docs/tutorials/incremental-adoption/offload-to-queue.mdx.

How do I ensure data consistency when using both workers together?

Use iii-state as the source of truth for your data, performing all mutations through state::set or state::update operations which provide atomic consistency. Then use state triggers to enqueue jobs in iii-queue for side effects. The state update succeeds or fails atomically before any queue job executes, ensuring your background processing always references committed state. Handle potential duplicate queue executions by implementing idempotency keys in your job handlers.

What are the performance implications of state triggers versus queue processing?

State triggers execute synchronously during the state mutation, adding latency to the state::set or state::update call. According to benchmarks in sdk/packages/node/iii/tests/state.test.ts, trigger execution time directly impacts API response latency. Queue jobs execute asynchronously with minimal enqueue latency (typically sub-millisecond), but job execution occurs later with eventual consistency. For high-throughput scenarios, prefer enqueueing lightweight jobs from state triggers rather than performing heavy computation within the trigger handler itself.

Which adapter should I choose for multi-instance deployments?

For iii-state in multi-instance deployments, select the redis adapter to ensure all engine instances share consistent state through a central Redis cluster. For iii-queue, choose rabbitmq for distributed queue semantics across instances, or redis for simpler shared queue requirements. Avoid the builtin or in_memory adapters in production multi-node configurations, as these restrict both state and queues to individual engine instances, causing data fragmentation and job processing duplication.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}