# How iii-observability Integrates with OpenTelemetry for Traces, Metrics, and Logs

> Learn how iii-observability unifies OpenTelemetry data collection for traces, metrics, and logs across Rust, Node.js, and Python via OTLP JSON payloads over WebSockets to the iii engine.

- Repository: [iii/iii](https://github.com/iii-hq/iii)
- Tags: how-to-guide
- Published: 2026-05-28

---

**The iii-observability layer unifies OpenTelemetry data collection across Rust, Node.js, and Python SDKs by streaming OTLP JSON payloads over a dedicated `/otel` WebSocket to the iii engine, using custom exporters for spans, metrics, and logs.**

The iii-observability module in the iii-hq/iii repository serves as the shared telemetry pipeline for every iii SDK. It standardizes how distributed systems export OpenTelemetry data back to the iii engine, enabling real-time observability without requiring external collectors.

## Architecture Overview

The iii-observability integration follows a consistent six-step pattern across all supported languages:

1. **Unified `/otel` WebSocket endpoint** – Each SDK establishes a dedicated telemetry-only WebSocket (`ws://…/otel`) that isolates observability traffic from regular worker registry connections.
2. **Resource definition** – The SDK constructs an OpenTelemetry **Resource** containing the service name, version, instance ID, and SDK metadata.
3. **Global propagator** – A composite of the W3C Trace Context and Baggage propagators is installed to enable distributed trace propagation across HTTP boundaries.
4. **Span exporter** – The custom `EngineSpanExporter` serializes spans as OTLP JSON and transmits them over the shared WebSocket.
5. **Metrics exporter** – `EngineMetricsExporter` streams OTLP metric points on the same socket, opt-out via `OTEL_METRICS_ENABLED=false`.
6. **Log exporter** – `EngineLogExporter` batches structured logs as OpenTelemetry LogRecords, opt-out via `OTEL_LOGS_ENABLED=false`.

The engine deserializes these OTLP JSON payloads into an in-memory buffer or forwards them to an external OTLP/gRPC collector based on configuration.

## Implementation by Language

### Rust

In the Rust SDK, `init_otel(config).await` in [`sdk/packages/rust/observability/src/telemetry/mod.rs`](https://github.com/iii-hq/iii/blob/main/sdk/packages/rust/observability/src/telemetry/mod.rs) initializes the telemetry stack. It creates a `SharedEngineConnection` using `append_otel_path(ws_url)`, builds a `Resource`, registers a `BaggageSpanProcessor`, and installs an `SdkTracerProvider` with the `EngineSpanExporter`.

```rust
use iii_sdk::iii::{self, OtelConfig};

#[tokio::main]
async fn main() {
    let cfg = OtelConfig {
        service_name: Some("order-service".into()),
        ..Default::default()
    };
    iii::init_otel(cfg).await;
    
    iii::with_span("process-order", None, None, || async {
        // business logic
        Ok::<_, Box<dyn std::error::Error>>(())
    }).await.unwrap();
}

```

### Node.js

The Node.js implementation in [`sdk/packages/node/observability/src/telemetry-system/index.ts`](https://github.com/iii-hq/iii/blob/main/sdk/packages/node/observability/src/telemetry-system/index.ts) exposes `initOtel(config)`, which constructs the `Resource`, opens the `SharedEngineConnection`, and wires the `EngineSpanExporter` to the `NodeTracerProvider`.

```typescript
import { initOtel, withSpan, SpanKind } from "iii-sdk/observability";

initOtel({ serviceName: "payment-gateway" });

async function charge(cardId: string, amount: number) {
  await withSpan("charge-card", { kind: SpanKind.CLIENT }, async span => {
    const response = await fetch("https://payments.example/charge", {
      method: "POST",
      body: JSON.stringify({ cardId, amount })
    });
    span.setAttribute("http.status_code", response.status);
  });
}

```

### Python

Python SDK users call `init_otel(config)` from [`sdk/packages/python/observability/src/iii_observability/telemetry.py`](https://github.com/iii-hq/iii/blob/main/sdk/packages/python/observability/src/iii_observability/telemetry.py), which creates the `SharedEngineConnection`, configures the `Resource`, and sets up the `TracerProvider` with `EngineSpanExporter`.

```python
from iii_observability import init_otel, with_span

async def main():
    init_otel()
    
    async def fetch_user(user_id: str):
        # Automatically traced via fetch instrumentation

        return await aiohttp.request("GET", f"https://api.example/users/{user_id}")
    
    await with_span("handle-request", lambda span: fetch_user("abc123"))

asyncio.run(main())

```

## Engine-Side Processing

On the engine side, [`engine/src/workers/observability/otel.rs`](https://github.com/iii-hq/iii/blob/main/engine/src/workers/observability/otel.rs) configures the global OpenTelemetry layer and registers the same composite propagator used by the SDKs. The worker supports three exporter modes configured via `OTEL_EXPORTER_TYPE`:

- **`memory`** – Stores spans in an in-memory buffer accessible via `engine::traces` skills.
- **`otlp`** – Forwards data to an external OTLP/gRPC collector.
- **`both`** – Uses a tee exporter to simultaneously store locally and forward externally.

Structured log events are converted to `StoredSpanEvent` objects and stored alongside trace spans.

## Configuration and Environment Variables

Control the iii-observability behavior using environment variables or the [`engine/config.yaml`](https://github.com/iii-hq/iii/blob/main/engine/config.yaml) file:

- `OTEL_ENABLED` – Master switch to enable/disable telemetry.
- `OTEL_METRICS_ENABLED` – Opt-out flag for `EngineMetricsExporter`.
- `OTEL_LOGS_ENABLED` – Opt-out flag for `EngineLogExporter`.
- `OTEL_EXPORTER_TYPE` – Selects `memory`, `otlp`, or `both` export strategies.

All telemetry streams share the same WebSocket connection, guaranteeing ordering and unified graceful shutdown via `shutdown_otel()`.

## Code Examples

### Querying Stored Traces via CLI

Access the in-memory span storage populated by the SDK exporters using the iii CLI:

```bash
iii cloud logs query --filter 'service.name == "order-service"' --limit 10

```

This command interfaces with the engine’s observability module, which reads from the storage layer defined in [`engine/src/workers/observability/otel.rs`](https://github.com/iii-hq/iii/blob/main/engine/src/workers/observability/otel.rs).

### Complete Rust Setup with Shutdown

```rust
use iii_sdk::iii::{self, OtelConfig};

#[tokio::main]
async fn main() {
    let cfg = OtelConfig {
        service_name: Some("inventory-service".into()),
        ..Default::default()
    };
    
    // Initialize connection and exporters
    iii::init_otel(cfg).await;
    
    // Execute traced operation
    iii::with_span("update-stock", None, None, || async {
        // database operations
        Ok::<_, Box<dyn std::error::Error>>(())
    }).await.unwrap();
    
    // Graceful shutdown flushes remaining spans
    iii::shutdown_otel().await;
}

```

## Summary

- **iii-observability** provides a unified OpenTelemetry integration across Rust, Node.js, and Python SDKs in the iii-hq/iii repository.
- All SDKs stream OTLP JSON data over a dedicated `/otel` WebSocket using `SharedEngineConnection`.
- Custom exporters (`EngineSpanExporter`, `EngineMetricsExporter`, `EngineLogExporter`) handle serialization and transmission.
- The engine stores data in-memory for skill queries or forwards to external collectors via the configuration in [`otel.rs`](https://github.com/iii-hq/iii/blob/main/otel.rs).
- Distributed tracing uses W3C Trace Context and Baggage propagators for cross-service correlation.

## Frequently Asked Questions

### What protocol does iii-observability use to send data?

The system uses **OTLP (OpenTelemetry Protocol) in JSON format** over WebSocket connections. Unlike standard OTLP which typically uses HTTP or gRPC, iii-observability streams JSON-encoded spans, metrics, and logs through a persistent WebSocket at the `/otel` endpoint to ensure low-latency, bidirectional communication with the iii engine.

### How do I disable metrics or logs while keeping traces?

Set the environment variables `OTEL_METRICS_ENABLED=false` or `OTEL_LOGS_ENABLED=false` before initializing the SDK. The `init_otel` function checks these flags and skips instantiating the `EngineMetricsExporter` or `EngineLogExporter`, while keeping the `EngineSpanExporter` active for trace data.

### How does distributed trace propagation work between workers?

The SDKs install a **composite propagator** combining W3C Trace Context and Baggage propagators during initialization. This propagator automatically injects trace headers into outgoing HTTP requests and extracts them from incoming requests, ensuring spans generated across different iii workers or external services share the same trace ID.

### Where are traces stored after the engine receives them?

According to [`engine/src/workers/observability/otel.rs`](https://github.com/iii-hq/iii/blob/main/engine/src/workers/observability/otel.rs), the engine stores received spans in an **in-memory buffer** when configured with `OTEL_EXPORTER_TYPE=memory` or `both`. This buffer powers the built-in observability skills like `engine::traces` and `engine::log`. When using `otlp` mode, data is forwarded immediately to an external collector without local storage.