Understanding Magnitude's OpenTelemetry Motel Tracing System: How to Inspect Traces and Spans

Magnitude's OpenTelemetry Motel is a lightweight, local collector that receives telemetry from ACN and ICN processes via OTLP HTTP/JSON when MAGNITUDE_OTEL=1 is set, exposing traces and spans through a queryable HTTP API at http://127.0.0.1:27686.

Magnitude ships with Motel, a purpose-built OpenTelemetry collector designed for local development and debugging. When enabled, every agent daemon (ACN) and inference server (ICN) instance exports distributed traces to this collector, giving developers granular visibility into RPC calls, span attributes, and log records without external dependencies. This article explains how Motel works internally in the magnitudedev/magnitude codebase and demonstrates the exact HTTP endpoints and code patterns for inspecting telemetry data.

What Is Motel and How It Works

Motel is implemented as a standalone binary that starts automatically with Magnitude's daemon processes. It functions as a minimal OTLP receiver and storage backend, accepting spans over HTTP/JSON and persisting them in memory for rapid querying.

Service Identity and Activation

According to design/inference/observability.md, telemetry is gated by a single environment variable:

• MAGNITUDE_OTEL=1 — enables the OTLP exporter in both ACN and ICN processes
• Service name: magnitude-icn (used for all inference engine traces)
• Default collector endpoint: http://127.0.0.1:27686

This design ensures zero configuration overhead—set one variable and traces flow to the local collector.

Querying Traces via the Motel HTTP API

The Motel collector exposes a RESTful HTTP API documented in AGENTS.md (lines 84-100). All endpoints return JSON and support standard HTTP clients.

Core Endpoints

Endpoint	Purpose
GET /api/health	Verify collector status
GET /api/services	List registered services with telemetry
GET /api/traces?service=&lookback=&limit=	Retrieve recent trace summaries
GET /api/traces/{traceId}	Fetch complete trace tree with spans
GET /api/spans/{spanId}	Get single span with attributes and logs
GET /api/logs?service=&lookback=&limit=	Query recent log records

Direct curl Examples

Verify the collector is running:

curl http://127.0.0.1:27686/api/health

{ "status": "ok" }

List services that have exported telemetry:

curl http://127.0.0.1:27686/api/services

[{ "name": "magnitude-icn", "type": "traces", "version": "..." }]

Retrieve recent traces (last 4 hours, max 40):

curl "http://127.0.0.1:27686/api/traces?service=magnitude-icn&lookback=4h&limit=40"

{
  "data": [
    {
      "traceId": "...",
      "rootOperationName": "RpcServer.someMethod",
      "startTime": "...",
      "durationMs": 42
    }
  ]
}

Fetch a complete trace tree:

TRACE_ID=1234567890abcdef
curl "http://127.0.0.1:27686/api/traces/$TRACE_ID"

{
  "traceId": "123456...",
  "spans": [
    {
      "spanId": "...",
      "name": "RpcServer.someMethod",
      "parentSpanId": null,
      "attributes": { "rpc.method": "someMethod", "rpc.service": "RpcServer" },
      "events": [...]
    }
  ]
}

Inspect a single span with its log records:

SPAN_ID=abcdef1234567890
curl "http://127.0.0.1:27686/api/spans/$SPAN_ID"

{
  "spanId": "abcdef...",
  "attributes": { "key": "value" },
  "logs": [
    { "timestamp": "...", "message": "Processing request", "level": "INFO" }
  ]
}

Query recent logs directly:

curl "http://127.0.0.1:27686/api/logs?service=magnitude-icn&lookback=4h&limit=50"

Programmatic Access via the ACN Dashboard

The ACN Dashboard (packages/acn-dashboard/src/server.ts) wraps Motel's HTTP API with TypeScript helpers, handling URL construction and type mapping from raw MotelTraceSummary objects to the internal RpcTraceSummary format used by the web UI.

In packages/acn-dashboard/src/server.ts (lines 11-13), the collector URL is configured:

const MOTEL_URL = process.env.MAGNITUDE_MOTEL_URL || 'http://127.0.0.1:27686';

The trace-listing helper (lines 15-26) forwards requests to Motel and transforms responses:

import { listRpcTraces } from '@magnitudedev/acn-dashboard/lib/api';

// Enable OTEL before starting any Magnitude processes
process.env.MAGNITUDE_OTEL = '1';

// Fetch RPC-specific traces with typed return values
const traces = await listRpcTraces();
console.log('Recent RPC traces:', traces);

This abstraction is useful when building custom dashboards or integrating trace inspection into CI pipelines.

Architecture and Key Source Files

File	Responsibility	Key Lines
AGENTS.md	Motel HTTP API specification	L84-L100
design/inference/observability.md	OTEL activation and service naming	L18-L27, L20-L27
packages/acn-dashboard/src/server.ts	Dashboard server, Motel client, listRpcTraces helper	L11-L13, L15-L26

The collector runs in-process with Magnitude's daemon launcher, binding to port 27686 on startup. No separate Docker container or external service is required, making it ideal for local development workflows.

Summary

• Motel is Magnitude's built-in OpenTelemetry collector, active when MAGNITUDE_OTEL=1
• Default endpoint: http://127.0.0.1:27686 with six documented HTTP endpoints for health, services, traces, spans, and logs
• The ACN Dashboard (packages/acn-dashboard/src/server.ts) provides typed TypeScript helpers around the raw HTTP API
• Traces from the inference engine are tagged with service name magnitude-icn as defined in design/inference/observability.md
• All telemetry is stored in-memory for fast querying of recent data (typically hours, not days)

Frequently Asked Questions

How do I enable tracing in Magnitude?

Set the environment variable MAGNITUDE_OTEL=1 before starting any ACN or ICN processes. This activates the OTLP HTTP/JSON exporter, which begins sending spans to the Motel collector at http://127.0.0.1:27686.

What service name should I filter for when querying inference traces?

Use service=magnitude-icn in your API queries. This is the hardcoded service identifier for all telemetry emitted by Magnitude's inference server, as documented in design/inference/observability.md.

Can I use Motel in production or CI environments?

Motel is optimized for local development with in-memory storage. For production observability, export traces from Magnitude processes directly to a persistent OpenTelemetry backend (Jaeger, Honeycomb, etc.) using the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

How do I access traces programmatically from my own code?

Either make direct HTTP calls to http://127.0.0.1:27686/api/traces/* endpoints, or import the listRpcTraces helper from @magnitudedev/acn-dashboard/lib/api for a typed interface that handles response parsing automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}