# Macro Observability Stack: OpenTelemetry and Datadog Implementation Guide

> Discover Macro's observability stack using OpenTelemetry and Datadog. Achieve end-to-end visibility with automatic trace-log correlation for Rust applications.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: how-to-guide
- Published: 2026-08-16

---

**Macro uses an OpenTelemetry-based tracing ecosystem for Rust with configurable backends—Jaeger for local development and Datadog APM for production—to achieve end-to-end observability with automatic trace-log correlation.**

The `macro-inc/macro` repository implements a production-ready **observability stack** centered on the OpenTelemetry (OTel) protocol. This architecture decouples instrumentation from backend storage, allowing the same codebase to run locally with Jaeger's UI or in production with Datadog's enterprise APM.

## Core OpenTelemetry Components

The foundation rests on three Rust crates from the OpenTelemetry project, wired together in [`crates/macro_entrypoint/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/crates/macro_entrypoint/src/lib.rs):

- **`opentelemetry`** – core SDK for span creation and context propagation
- **`opentelemetry-otlp`** – OTLP (OpenTelemetry Protocol) exporter implementation
- **`opentelemetry-sdk`** – tracer provider and span processing pipelines

These dependencies enable vendor-neutral instrumentation. The actual backend—whether Jaeger or Datadog—is determined at runtime via environment configuration.

## Tracing Bridge: From `tracing` Crate to OTel Spans

Macro uses the ubiquitous `tracing` crate for structured logging throughout its codebase. The `macro_entrypoint` crate bridges this to OpenTelemetry through a custom layer defined in [`crates/macro_entrypoint/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/crates/macro_entrypoint/src/lib.rs) at lines 46-58:

```rust
// Simplified representation of the otel_layer_with_error_mapping function
fn otel_layer_with_error_mapping<S>() -> OpenTelemetryLayer<S, Tracer>
where
    S: tracing::Subscriber + for<'span> LookupSpan<'span>,
{
    tracing_opentelemetry::layer()
        .with_error_records_to_exceptions(true)
}

```

The `tracing_opentelemetry::OpenTelemetryLayer` performs three critical functions:

1. **Converts `tracing` spans** into OTel span representations
2. **Propagates context** across async boundaries
3. **Maps errors to exceptions**—`tracing::error!` events become OTLP exceptions with full stack traces visible in Datadog APM

## Datadog Integration and Trace-Log Correlation

Production deployments target Datadog APM. Two mechanisms enable deep integration:

### OTLP Endpoint Routing

The `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable (default: `http://127.0.0.1:4317`) controls where spans are exported. In production Kubernetes environments, this resolves to a Datadog agent side-car container. The constant is defined in [`crates/macro_entrypoint/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/crates/macro_entrypoint/src/lib.rs) at lines 36-38:

```rust
pub const DEFAULT_OTLP_ENDPOINT: &str = "http://127.0.0.1:4317";

```

### DatadogFormat Log Injection

The custom formatter in [`crates/macro_entrypoint/src/datadog_fmt.rs`](https://github.com/macro-inc/macro/blob/main/crates/macro_entrypoint/src/datadog_fmt.rs) injects Datadog-specific fields into every JSON log line. It reads the active span context and adds:

- `dd.trace_id` – Datadog-compatible trace identifier
- `dd.span_id` – Span identifier for precise correlation

This enables Datadog's automatic **trace-log correlation**, where clicking a span in APM surfaces related logs instantly.

```rust
// Conceptual usage of DatadogFormat
let fmt_layer = fmt::layer()
    .json()
    .event_format(DatadogFormat::new(
        Format::default().json(),
    ));

```

## Local Development with Jaeger

For local debugging, Macro provides a Jaeger profile via Docker Compose. The [`docker/docker-compose.yml`](https://github.com/macro-inc/macro/blob/main/docker/docker-compose.yml) file (lines 13-45) defines:

- **Jaeger all-in-one** container exposing OTLP on port 4317 and UI on `http://localhost:16686`
- **Datadog agent** container (optional) for testing production-like configurations

The CLI in [`tooling/xtask/crates/xtask_local/src/local/cli.rs`](https://github.com/macro-inc/macro/blob/main/tooling/xtask/crates/xtask_local/src/local/cli.rs) exposes a `--traces` flag that selects the appropriate backend based on the active profile.

## Analytics Proxy: Browser Telemetry Without Ad-Blocker Interference

A unique production consideration: browser-based telemetry must bypass ad-blockers and protect API keys. The `analytics_proxy` service defined in [`docker/docker-compose.yml`](https://github.com/macro-inc/macro/blob/main/docker/docker-compose.yml) (lines 50-60) handles this:

- Exposes `/i/otlp` endpoint for browser-span submission
- Forwards to the configured collector (Jaeger or Datadog)
- Hides Datadog API keys from client-side code

## Initialization and Shutdown Pattern

Applications using Macro's observability stack follow a consistent lifecycle:

```rust
use macro_entrypoint::MacroEntrypoint;
use macro_env::Environment;

// Initialize tracing, OTel, and Datadog formatting
let entry = MacroEntrypoint::new(Environment::Production).init();

// Instrumented business logic
#[tracing::instrument(fields(document_id = %id))]
fn process_document(id: i64) -> Result<(), anyhow::Error> {
    tracing::info!("starting document processing");
    // ... work happens ...
    tracing::info!("document processed successfully");
    Ok(())
}

// Graceful shutdown ensures all spans are exported
entry.shutdown();

```

The `#[tracing::instrument]` attribute automatically creates OTel spans. Log emissions within these spans receive `dd.trace_id` and `dd.span_id` injection via the `DatadogFormat` layer.

## Cloudflare Workers Extension

For edge compute scenarios, [`crates/worker-rs-otel/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/crates/worker-rs-otel/src/lib.rs) provides a lightweight OTel exporter compatible with Cloudflare Workers' constrained runtime. This shares the same tracing instrumentation but uses a custom export path suitable for V8 isolates.

## Summary

- **Core stack**: OpenTelemetry Rust SDK (`opentelemetry`, `opentelemetry-otlp`, `opentelemetry-sdk`) with `tracing` crate integration
- **Bridge layer**: `tracing_opentelemetry::OpenTelemetryLayer` with custom error mapping in [`macro_entrypoint/src/lib.rs`](https://github.com/macro-inc/macro/blob/main/macro_entrypoint/src/lib.rs)
- **Production backend**: Datadog APM via OTLP export to agent side-car, with `DatadogFormat` enabling trace-log correlation
- **Local backend**: Jaeger (UI at `localhost:16686`) via Docker Compose profile
- **Browser telemetry**: `analytics_proxy` service prevents ad-blocker interference and secures API keys

## Frequently Asked Questions

### How does Macro correlate logs with traces in Datadog?

The `DatadogFormat` struct in [`crates/macro_entrypoint/src/datadog_fmt.rs`](https://github.com/macro-inc/macro/blob/main/crates/macro_entrypoint/src/datadog_fmt.rs) reads the current OpenTelemetry span context during log formatting and injects `dd.trace_id` and `dd.span_id` fields into each JSON log line. Datadog's ingestion pipeline recognizes these fields and automatically links logs to their corresponding APM spans.

### Can I run Macro's observability stack without Datadog?

Yes. By setting the appropriate Docker Compose profile, developers can route spans to **Jaeger** instead. The `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable controls the destination, and the `jaeger` profile in [`docker/docker-compose.yml`](https://github.com/macro-inc/macro/blob/main/docker/docker-compose.yml) provides a complete local stack with web UI at `http://localhost:16686`.

### What happens to `tracing::error!` calls in this stack?

The custom `otel_layer_with_error_mapping` function configures `tracing_opentelemetry` to convert error-level log records into OTLP exception events. These appear in Datadog APM with full stack traces, allowing engineers to diagnose errors without switching between logs and traces.

### How does the analytics proxy protect Datadog API keys?

The `analytics_proxy` service defined in [`docker/docker-compose.yml`](https://github.com/macro-inc/macro/blob/main/docker/docker-compose.yml) accepts browser-originated OTLP traffic on `/i/otlp` and forwards it to the actual collector. This indirection keeps sensitive Datadog API keys server-side while allowing client-side instrumentation to submit spans without triggering ad-blocker rules that target known Datadog endpoints.