Macro Observability Stack: OpenTelemetry and Datadog Implementation Guide

Macro uses an OpenTelemetry-based tracing ecosystem for Rust with configurable backends—Jaeger for local development and Datadog APM for production—to achieve end-to-end observability with automatic trace-log correlation.

The macro-inc/macro repository implements a production-ready observability stack centered on the OpenTelemetry (OTel) protocol. This architecture decouples instrumentation from backend storage, allowing the same codebase to run locally with Jaeger's UI or in production with Datadog's enterprise APM.

Core OpenTelemetry Components

The foundation rests on three Rust crates from the OpenTelemetry project, wired together in crates/macro_entrypoint/src/lib.rs:

• opentelemetry – core SDK for span creation and context propagation
• opentelemetry-otlp – OTLP (OpenTelemetry Protocol) exporter implementation
• opentelemetry-sdk – tracer provider and span processing pipelines

These dependencies enable vendor-neutral instrumentation. The actual backend—whether Jaeger or Datadog—is determined at runtime via environment configuration.

Tracing Bridge: From tracing Crate to OTel Spans

Macro uses the ubiquitous tracing crate for structured logging throughout its codebase. The macro_entrypoint crate bridges this to OpenTelemetry through a custom layer defined in crates/macro_entrypoint/src/lib.rs at lines 46-58:

// Simplified representation of the otel_layer_with_error_mapping function
fn otel_layer_with_error_mapping<S>() -> OpenTelemetryLayer<S, Tracer>
where
    S: tracing::Subscriber + for<'span> LookupSpan<'span>,
{
    tracing_opentelemetry::layer()
        .with_error_records_to_exceptions(true)
}

The tracing_opentelemetry::OpenTelemetryLayer performs three critical functions:

1. Converts tracing spans into OTel span representations
2. Propagates context across async boundaries
3. Maps errors to exceptions—tracing::error! events become OTLP exceptions with full stack traces visible in Datadog APM

Datadog Integration and Trace-Log Correlation

Production deployments target Datadog APM. Two mechanisms enable deep integration:

OTLP Endpoint Routing

The OTEL_EXPORTER_OTLP_ENDPOINT environment variable (default: http://127.0.0.1:4317) controls where spans are exported. In production Kubernetes environments, this resolves to a Datadog agent side-car container. The constant is defined in crates/macro_entrypoint/src/lib.rs at lines 36-38:

pub const DEFAULT_OTLP_ENDPOINT: &str = "http://127.0.0.1:4317";

DatadogFormat Log Injection

The custom formatter in crates/macro_entrypoint/src/datadog_fmt.rs injects Datadog-specific fields into every JSON log line. It reads the active span context and adds:

• dd.trace_id – Datadog-compatible trace identifier
• dd.span_id – Span identifier for precise correlation

This enables Datadog's automatic trace-log correlation, where clicking a span in APM surfaces related logs instantly.

// Conceptual usage of DatadogFormat
let fmt_layer = fmt::layer()
    .json()
    .event_format(DatadogFormat::new(
        Format::default().json(),
    ));

Local Development with Jaeger

For local debugging, Macro provides a Jaeger profile via Docker Compose. The docker/docker-compose.yml file (lines 13-45) defines:

• Jaeger all-in-one container exposing OTLP on port 4317 and UI on http://localhost:16686
• Datadog agent container (optional) for testing production-like configurations

The CLI in tooling/xtask/crates/xtask_local/src/local/cli.rs exposes a --traces flag that selects the appropriate backend based on the active profile.

Analytics Proxy: Browser Telemetry Without Ad-Blocker Interference

A unique production consideration: browser-based telemetry must bypass ad-blockers and protect API keys. The analytics_proxy service defined in docker/docker-compose.yml (lines 50-60) handles this:

• Exposes /i/otlp endpoint for browser-span submission
• Forwards to the configured collector (Jaeger or Datadog)
• Hides Datadog API keys from client-side code

Initialization and Shutdown Pattern

Applications using Macro's observability stack follow a consistent lifecycle:

use macro_entrypoint::MacroEntrypoint;
use macro_env::Environment;

// Initialize tracing, OTel, and Datadog formatting
let entry = MacroEntrypoint::new(Environment::Production).init();

// Instrumented business logic
#[tracing::instrument(fields(document_id = %id))]
fn process_document(id: i64) -> Result<(), anyhow::Error> {
    tracing::info!("starting document processing");
    // ... work happens ...
    tracing::info!("document processed successfully");
    Ok(())
}

// Graceful shutdown ensures all spans are exported
entry.shutdown();

The #[tracing::instrument] attribute automatically creates OTel spans. Log emissions within these spans receive dd.trace_id and dd.span_id injection via the DatadogFormat layer.

Cloudflare Workers Extension

For edge compute scenarios, crates/worker-rs-otel/src/lib.rs provides a lightweight OTel exporter compatible with Cloudflare Workers' constrained runtime. This shares the same tracing instrumentation but uses a custom export path suitable for V8 isolates.

Summary

• Core stack: OpenTelemetry Rust SDK (opentelemetry, opentelemetry-otlp, opentelemetry-sdk) with tracing crate integration
• Bridge layer: tracing_opentelemetry::OpenTelemetryLayer with custom error mapping in macro_entrypoint/src/lib.rs
• Production backend: Datadog APM via OTLP export to agent side-car, with DatadogFormat enabling trace-log correlation
• Local backend: Jaeger (UI at localhost:16686) via Docker Compose profile
• Browser telemetry: analytics_proxy service prevents ad-blocker interference and secures API keys

Frequently Asked Questions

How does Macro correlate logs with traces in Datadog?

The DatadogFormat struct in crates/macro_entrypoint/src/datadog_fmt.rs reads the current OpenTelemetry span context during log formatting and injects dd.trace_id and dd.span_id fields into each JSON log line. Datadog's ingestion pipeline recognizes these fields and automatically links logs to their corresponding APM spans.

Can I run Macro's observability stack without Datadog?

Yes. By setting the appropriate Docker Compose profile, developers can route spans to Jaeger instead. The OTEL_EXPORTER_OTLP_ENDPOINT environment variable controls the destination, and the jaeger profile in docker/docker-compose.yml provides a complete local stack with web UI at http://localhost:16686.

What happens to tracing::error! calls in this stack?

The custom otel_layer_with_error_mapping function configures tracing_opentelemetry to convert error-level log records into OTLP exception events. These appear in Datadog APM with full stack traces, allowing engineers to diagnose errors without switching between logs and traces.

How does the analytics proxy protect Datadog API keys?

The analytics_proxy service defined in docker/docker-compose.yml accepts browser-originated OTLP traffic on /i/otlp and forwards it to the actual collector. This indirection keeps sensitive Datadog API keys server-side while allowing client-side instrumentation to submit spans without triggering ad-blocker rules that target known Datadog endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}