# How Easegress Implements Distributed Tracing with OpenTelemetry

> Discover how Easegress integrates distributed tracing with OpenTelemetry. Learn about span creation, context propagation, and multi-protocol export via centralized configuration.

- Repository: [MegaEase/easegress](https://github.com/megaease/easegress)
- Tags: how-to-guide
- Published: 2026-03-07

---

**Easegress implements distributed tracing with OpenTelemetry by wrapping the OpenTelemetry SDK into a configurable `Tracer` abstraction that handles span creation, context propagation, and multi-protocol export (Jaeger, Zipkin, OTLP) through centralized configuration in [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go).**

Easegress is a cloud-native traffic orchestration system that provides built-in observability features for modern microservices. The platform implements distributed tracing with OpenTelemetry through a unified tracing layer defined in [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go), enabling seamless integration with popular backends like Jaeger and Zipkin. This implementation follows the standard OpenTelemetry architecture while adding Easegress-specific conveniences such as Cloudflare request tracing and flexible header format propagation.

## Architecture Components

The tracing system in `megaease/easegress` follows the standard OpenTelemetry pipeline architecture, with six core responsibilities implemented as discrete layers.

### Trace Exporters

The `ExporterSpec` configuration selects the appropriate backend protocol. In [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go), the `newExporters()` function instantiates concrete exporter objects based on the specification:

- **Jaeger**: Uses `jaeger.New` for direct collector submission
- **Zipkin**: Uses `zipkin.New` for Zipkin-compatible endpoints  
- **OTLP**: Uses `otlptracegrpc.New` or `otlptracehttp.New` for OpenTelemetry Protocol transmission

### Tracer Provider Configuration

The `New(spec *Spec)` function in [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go) constructs an `sdktrace.TracerProvider` with pipeline options derived from the configuration spec. This includes:

- **Span limits**: Maximum attributes per span
- **Sampler**: Rate-based sampling via `sampleRate`
- **Batch processor**: Configurable via `batchLimits` (maxQueueSize, batchTimeout, exportTimeout, maxExportBatchSize)
- **Resource attributes**: Service name and custom tags

### Context Propagation

Propagation is handled by a stored `propagation.TextMapPropagator` created in `newPropagator()`. The implementation supports:

- **W3C Trace-Context** (default): Standards-compliant traceparent/tracestate headers
- **B3 single-header**: Legacy Zipkin format activated when using legacy Zipkin configuration

The propagator is stored in the `Tracer` struct and used for both extraction (incoming) and injection (outgoing) operations.

### Span Creation and Cloudflare Integration

For incoming HTTP requests, `Tracer.NewSpanForHTTP()` in [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go) creates spans through a two-step process:

1. First attempts `newSpanForCloudflare()` to detect Cloudflare-specific headers (Ray ID, request timestamps)
2. Falls back to `newSpanWithStart()` for standard span initialization

When Cloudflare headers are present, the system creates a parent span recording the Ray ID and edge timestamps, then attaches a child span for actual request handling (implemented in [`pkg/tracing/cloudflare.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/cloudflare.go)).

### Outbound Request Injection

When forwarding requests to downstream services, `Span.InjectHTTP()` writes trace context into HTTP headers using the stored propagator. This ensures distributed traces remain connected across service boundaries regardless of whether downstream services run Easegress or other OpenTelemetry-compatible implementations.

### Lifecycle Management

The `Tracer.Close()` method calls `tp.Shutdown()` on the underlying `TracerProvider`, flushing pending spans to the exporter before component reload or shutdown. This prevents span loss during configuration updates or graceful shutdowns.

## Configuration Format

Tracing configuration is expressed through a `Spec` struct that supports JSON or YAML definitions. The `Spec.UnmarshalJSON` method fills defaults such as `headerFormatTraceContext` when not explicitly specified.

Example configuration structure:

```json
{
  "serviceName": "my-service",
  "tags": { "env": "prod" },
  "sampleRate": 1,
  "batchLimits": {
    "maxQueueSize": 2048,
    "batchTimeout": 5000,
    "exportTimeout": 30000,
    "maxExportBatchSize": 512
  },
  "exporter": {
    "jaeger": {
      "mode": "collector",
      "endpoint": "http://jaeger-collector:14268/api/traces"
    }
  },
  "headerFormat": "trace-context"
}

```

## HTTP Server Integration

In [`pkg/object/httpserver/mux.go`](https://github.com/megaease/easegress/blob/main/pkg/object/httpserver/mux.go), the HTTP server creates or reuses a `Tracer` instance during configuration reloads. The implementation checks if tracing specifications have changed using `reflect.DeepEqual` before instantiating a new tracer via `tracing.New()`.

For each incoming request, the server creates a span:

```go
span := mi.tracer.NewSpanForHTTP(stdr.Context(),
                               mi.superSpec.Name(),
                               stdr)
defer span.End()

```

This pattern ensures every request generates a complete trace while properly handling context cancellation and span closure.

## Propagating Traces to Downstream Services

When Easegress acts as a proxy, filters can propagate trace context to backend services using the span's injection capability:

```go
outReq, _ := http.NewRequest("GET", target, nil)
span.InjectHTTP(outReq)   // adds Traceparent / B3 header

```

Downstream services that recognize the selected header format (configured via `headerFormat`) automatically participate in the same trace, creating a complete distributed trace graph.

## Implementation Examples

### Configuring Tracing in a Pipeline YAML

Define tracing at the pipeline level to enable automatic instrumentation for all traffic flowing through that pipeline:

```yaml
kind: Pipeline
metadata:
  name: my-pipeline
spec:
  tracing:
    serviceName: my-pipeline
    exporter:
      otlp:
        protocol: grpc
        endpoint: otel-collector:4317
        insecure: true
    sampleRate: 0.5

```

### Programmatic Tracer Creation

For custom objects or advanced use cases, create a tracer programmatically using the `tracing` package:

```go
import (
    "github.com/megaease/easegress/v2/pkg/tracing"
)

spec := &tracing.Spec{
    ServiceName: "custom-service",
    Exporter: &tracing.ExporterSpec{
        Jaeger: &tracing.JaegerSpec{
            Mode:     "collector",
            Endpoint: "http://jaeger:14268/api/traces",
        },
    },
    HeaderFormat: tracing.HeaderFormatTraceContext,
}

tracer, err := tracing.New(spec)
if err != nil {
    // handle error
}
defer tracer.Close()

```

### Instrumenting a Custom Filter

Custom filters can participate in distributed tracing by creating spans and propagating context:

```go
func (f *MyFilter) Handle(req *http.Request) {
    span := f.tracer.NewSpanForHTTP(req.Context(), "my-filter", req)
    defer span.End()

    // add custom attributes
    span.SetAttributes(attribute.String("my-attr", "value"))

    // forward request …
    outReq, _ := http.NewRequest(req.Method, f.target, req.Body)
    span.InjectHTTP(outReq)
    // ...
}

```

## Summary

- Easegress implements distributed tracing with OpenTelemetry through a centralized [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go) module that wraps the OpenTelemetry SDK.
- The system supports multiple export protocols (Jaeger, Zipkin, OTLP) configured via `ExporterSpec` and instantiated in `newExporters()`.
- Context propagation supports both W3C Trace-Context and B3 formats through a configurable `TextMapPropagator`.
- HTTP servers in [`pkg/object/httpserver/mux.go`](https://github.com/megaease/easegress/blob/main/pkg/object/httpserver/mux.go) automatically create spans for incoming requests using `NewSpanForHTTP()`, with special handling for Cloudflare-traffic via [`pkg/tracing/cloudflare.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/cloudflare.go).
- Spans can inject trace context into outbound requests using `InjectHTTP()`, enabling end-to-end distributed tracing across service meshes.
- The `Tracer.Close()` method ensures graceful shutdown and span flushing via the underlying `TracerProvider.Shutdown()`.

## Frequently Asked Questions

### What OpenTelemetry exporters does Easegress support?

Easegress supports Jaeger (`jaeger.New`), Zipkin (`zipkin.New`), and OTLP (`otlptracegrpc.New` or `otlptracehttp.New`) exporters. These are configured through the `ExporterSpec` struct and instantiated in the `newExporters()` function within [`pkg/tracing/tracing.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/tracing.go).

### How does Easegress handle trace context propagation across services?

The system uses a stored `propagation.TextMapPropagator` created by `newPropagator()` to handle context injection and extraction. By default, it uses the W3C Trace-Context format, but can be configured to use B3 single-header format for legacy Zipkin compatibility via the `headerFormat` configuration option.

### Can Easegress trace requests coming through Cloudflare?

Yes. The `NewSpanForHTTP()` method first attempts to detect Cloudflare-specific headers via `newSpanForCloudflare()` in [`pkg/tracing/cloudflare.go`](https://github.com/megaease/easegress/blob/main/pkg/tracing/cloudflare.go). When detected, it creates a parent span containing the Ray ID and edge request timestamps, then attaches a child span for the actual request processing, providing visibility into the CDN layer.

### How do I ensure spans are flushed when Easegress reloads configuration?

Call `Tracer.Close()` during component shutdown or reload. This method invokes `tp.Shutdown()` on the underlying `sdktrace.TracerProvider`, which flushes any pending spans in the batch processor to the configured exporter before the tracer is destroyed.