How iii-observability Integrates with OpenTelemetry for Traces, Metrics, and Logs

The iii-observability layer unifies OpenTelemetry data collection across Rust, Node.js, and Python SDKs by streaming OTLP JSON payloads over a dedicated /otel WebSocket to the iii engine, using custom exporters for spans, metrics, and logs.

The iii-observability module in the iii-hq/iii repository serves as the shared telemetry pipeline for every iii SDK. It standardizes how distributed systems export OpenTelemetry data back to the iii engine, enabling real-time observability without requiring external collectors.

Architecture Overview

The iii-observability integration follows a consistent six-step pattern across all supported languages:

1. Unified /otel WebSocket endpoint – Each SDK establishes a dedicated telemetry-only WebSocket (ws://…/otel) that isolates observability traffic from regular worker registry connections.
2. Resource definition – The SDK constructs an OpenTelemetry Resource containing the service name, version, instance ID, and SDK metadata.
3. Global propagator – A composite of the W3C Trace Context and Baggage propagators is installed to enable distributed trace propagation across HTTP boundaries.
4. Span exporter – The custom EngineSpanExporter serializes spans as OTLP JSON and transmits them over the shared WebSocket.
5. Metrics exporter – EngineMetricsExporter streams OTLP metric points on the same socket, opt-out via OTEL_METRICS_ENABLED=false.
6. Log exporter – EngineLogExporter batches structured logs as OpenTelemetry LogRecords, opt-out via OTEL_LOGS_ENABLED=false.

The engine deserializes these OTLP JSON payloads into an in-memory buffer or forwards them to an external OTLP/gRPC collector based on configuration.

Implementation by Language

Rust

In the Rust SDK, init_otel(config).await in sdk/packages/rust/observability/src/telemetry/mod.rs initializes the telemetry stack. It creates a SharedEngineConnection using append_otel_path(ws_url), builds a Resource, registers a BaggageSpanProcessor, and installs an SdkTracerProvider with the EngineSpanExporter.

use iii_sdk::iii::{self, OtelConfig};

#[tokio::main]
async fn main() {
    let cfg = OtelConfig {
        service_name: Some("order-service".into()),
        ..Default::default()
    };
    iii::init_otel(cfg).await;
    
    iii::with_span("process-order", None, None, || async {
        // business logic
        Ok::<_, Box<dyn std::error::Error>>(())
    }).await.unwrap();
}

Node.js

The Node.js implementation in sdk/packages/node/observability/src/telemetry-system/index.ts exposes initOtel(config), which constructs the Resource, opens the SharedEngineConnection, and wires the EngineSpanExporter to the NodeTracerProvider.

import { initOtel, withSpan, SpanKind } from "iii-sdk/observability";

initOtel({ serviceName: "payment-gateway" });

async function charge(cardId: string, amount: number) {
  await withSpan("charge-card", { kind: SpanKind.CLIENT }, async span => {
    const response = await fetch("https://payments.example/charge", {
      method: "POST",
      body: JSON.stringify({ cardId, amount })
    });
    span.setAttribute("http.status_code", response.status);
  });
}

Python

Python SDK users call init_otel(config) from sdk/packages/python/observability/src/iii_observability/telemetry.py, which creates the SharedEngineConnection, configures the Resource, and sets up the TracerProvider with EngineSpanExporter.

from iii_observability import init_otel, with_span

async def main():
    init_otel()
    
    async def fetch_user(user_id: str):
        # Automatically traced via fetch instrumentation

        return await aiohttp.request("GET", f"https://api.example/users/{user_id}")
    
    await with_span("handle-request", lambda span: fetch_user("abc123"))

asyncio.run(main())

Engine-Side Processing

On the engine side, engine/src/workers/observability/otel.rs configures the global OpenTelemetry layer and registers the same composite propagator used by the SDKs. The worker supports three exporter modes configured via OTEL_EXPORTER_TYPE:

• memory – Stores spans in an in-memory buffer accessible via engine::traces skills.
• otlp – Forwards data to an external OTLP/gRPC collector.
• both – Uses a tee exporter to simultaneously store locally and forward externally.

Structured log events are converted to StoredSpanEvent objects and stored alongside trace spans.

Configuration and Environment Variables

Control the iii-observability behavior using environment variables or the engine/config.yaml file:

• OTEL_ENABLED – Master switch to enable/disable telemetry.
• OTEL_METRICS_ENABLED – Opt-out flag for EngineMetricsExporter.
• OTEL_LOGS_ENABLED – Opt-out flag for EngineLogExporter.
• OTEL_EXPORTER_TYPE – Selects memory, otlp, or both export strategies.

All telemetry streams share the same WebSocket connection, guaranteeing ordering and unified graceful shutdown via shutdown_otel().

Code Examples

Querying Stored Traces via CLI

Access the in-memory span storage populated by the SDK exporters using the iii CLI:

iii cloud logs query --filter 'service.name == "order-service"' --limit 10

This command interfaces with the engine’s observability module, which reads from the storage layer defined in engine/src/workers/observability/otel.rs.

Complete Rust Setup with Shutdown

use iii_sdk::iii::{self, OtelConfig};

#[tokio::main]
async fn main() {
    let cfg = OtelConfig {
        service_name: Some("inventory-service".into()),
        ..Default::default()
    };
    
    // Initialize connection and exporters
    iii::init_otel(cfg).await;
    
    // Execute traced operation
    iii::with_span("update-stock", None, None, || async {
        // database operations
        Ok::<_, Box<dyn std::error::Error>>(())
    }).await.unwrap();
    
    // Graceful shutdown flushes remaining spans
    iii::shutdown_otel().await;
}

Summary

• iii-observability provides a unified OpenTelemetry integration across Rust, Node.js, and Python SDKs in the iii-hq/iii repository.
• All SDKs stream OTLP JSON data over a dedicated /otel WebSocket using SharedEngineConnection.
• Custom exporters (EngineSpanExporter, EngineMetricsExporter, EngineLogExporter) handle serialization and transmission.
• The engine stores data in-memory for skill queries or forwards to external collectors via the configuration in otel.rs.
• Distributed tracing uses W3C Trace Context and Baggage propagators for cross-service correlation.

Frequently Asked Questions

What protocol does iii-observability use to send data?

The system uses OTLP (OpenTelemetry Protocol) in JSON format over WebSocket connections. Unlike standard OTLP which typically uses HTTP or gRPC, iii-observability streams JSON-encoded spans, metrics, and logs through a persistent WebSocket at the /otel endpoint to ensure low-latency, bidirectional communication with the iii engine.

How do I disable metrics or logs while keeping traces?

Set the environment variables OTEL_METRICS_ENABLED=false or OTEL_LOGS_ENABLED=false before initializing the SDK. The init_otel function checks these flags and skips instantiating the EngineMetricsExporter or EngineLogExporter, while keeping the EngineSpanExporter active for trace data.

How does distributed trace propagation work between workers?

The SDKs install a composite propagator combining W3C Trace Context and Baggage propagators during initialization. This propagator automatically injects trace headers into outgoing HTTP requests and extracts them from incoming requests, ensuring spans generated across different iii workers or external services share the same trace ID.

Where are traces stored after the engine receives them?

According to engine/src/workers/observability/otel.rs, the engine stores received spans in an in-memory buffer when configured with OTEL_EXPORTER_TYPE=memory or both. This buffer powers the built-in observability skills like engine::traces and engine::log. When using otlp mode, data is forwarded immediately to an external collector without local storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}