How Easegress Implements Distributed Tracing with OpenTelemetry

Easegress implements distributed tracing with OpenTelemetry by wrapping the OpenTelemetry SDK into a configurable Tracer abstraction that handles span creation, context propagation, and multi-protocol export (Jaeger, Zipkin, OTLP) through centralized configuration in pkg/tracing/tracing.go.

Easegress is a cloud-native traffic orchestration system that provides built-in observability features for modern microservices. The platform implements distributed tracing with OpenTelemetry through a unified tracing layer defined in pkg/tracing/tracing.go, enabling seamless integration with popular backends like Jaeger and Zipkin. This implementation follows the standard OpenTelemetry architecture while adding Easegress-specific conveniences such as Cloudflare request tracing and flexible header format propagation.

Architecture Components

The tracing system in megaease/easegress follows the standard OpenTelemetry pipeline architecture, with six core responsibilities implemented as discrete layers.

Trace Exporters

The ExporterSpec configuration selects the appropriate backend protocol. In pkg/tracing/tracing.go, the newExporters() function instantiates concrete exporter objects based on the specification:

• Jaeger: Uses jaeger.New for direct collector submission
• Zipkin: Uses zipkin.New for Zipkin-compatible endpoints
• OTLP: Uses otlptracegrpc.New or otlptracehttp.New for OpenTelemetry Protocol transmission

Tracer Provider Configuration

The New(spec *Spec) function in pkg/tracing/tracing.go constructs an sdktrace.TracerProvider with pipeline options derived from the configuration spec. This includes:

• Span limits: Maximum attributes per span
• Sampler: Rate-based sampling via sampleRate
• Batch processor: Configurable via batchLimits (maxQueueSize, batchTimeout, exportTimeout, maxExportBatchSize)
• Resource attributes: Service name and custom tags

Context Propagation

Propagation is handled by a stored propagation.TextMapPropagator created in newPropagator(). The implementation supports:

• W3C Trace-Context (default): Standards-compliant traceparent/tracestate headers
• B3 single-header: Legacy Zipkin format activated when using legacy Zipkin configuration

The propagator is stored in the Tracer struct and used for both extraction (incoming) and injection (outgoing) operations.

Span Creation and Cloudflare Integration

For incoming HTTP requests, Tracer.NewSpanForHTTP() in pkg/tracing/tracing.go creates spans through a two-step process:

1. First attempts newSpanForCloudflare() to detect Cloudflare-specific headers (Ray ID, request timestamps)
2. Falls back to newSpanWithStart() for standard span initialization

When Cloudflare headers are present, the system creates a parent span recording the Ray ID and edge timestamps, then attaches a child span for actual request handling (implemented in pkg/tracing/cloudflare.go).

Outbound Request Injection

When forwarding requests to downstream services, Span.InjectHTTP() writes trace context into HTTP headers using the stored propagator. This ensures distributed traces remain connected across service boundaries regardless of whether downstream services run Easegress or other OpenTelemetry-compatible implementations.

Lifecycle Management

The Tracer.Close() method calls tp.Shutdown() on the underlying TracerProvider, flushing pending spans to the exporter before component reload or shutdown. This prevents span loss during configuration updates or graceful shutdowns.

Configuration Format

Tracing configuration is expressed through a Spec struct that supports JSON or YAML definitions. The Spec.UnmarshalJSON method fills defaults such as headerFormatTraceContext when not explicitly specified.

Example configuration structure:

{
  "serviceName": "my-service",
  "tags": { "env": "prod" },
  "sampleRate": 1,
  "batchLimits": {
    "maxQueueSize": 2048,
    "batchTimeout": 5000,
    "exportTimeout": 30000,
    "maxExportBatchSize": 512
  },
  "exporter": {
    "jaeger": {
      "mode": "collector",
      "endpoint": "http://jaeger-collector:14268/api/traces"
    }
  },
  "headerFormat": "trace-context"
}

HTTP Server Integration

In pkg/object/httpserver/mux.go, the HTTP server creates or reuses a Tracer instance during configuration reloads. The implementation checks if tracing specifications have changed using reflect.DeepEqual before instantiating a new tracer via tracing.New().

For each incoming request, the server creates a span:

span := mi.tracer.NewSpanForHTTP(stdr.Context(),
                               mi.superSpec.Name(),
                               stdr)
defer span.End()

This pattern ensures every request generates a complete trace while properly handling context cancellation and span closure.

Propagating Traces to Downstream Services

When Easegress acts as a proxy, filters can propagate trace context to backend services using the span's injection capability:

outReq, _ := http.NewRequest("GET", target, nil)
span.InjectHTTP(outReq)   // adds Traceparent / B3 header

Downstream services that recognize the selected header format (configured via headerFormat) automatically participate in the same trace, creating a complete distributed trace graph.

Implementation Examples

Configuring Tracing in a Pipeline YAML

Define tracing at the pipeline level to enable automatic instrumentation for all traffic flowing through that pipeline:

kind: Pipeline
metadata:
  name: my-pipeline
spec:
  tracing:
    serviceName: my-pipeline
    exporter:
      otlp:
        protocol: grpc
        endpoint: otel-collector:4317
        insecure: true
    sampleRate: 0.5

Programmatic Tracer Creation

For custom objects or advanced use cases, create a tracer programmatically using the tracing package:

import (
    "github.com/megaease/easegress/v2/pkg/tracing"
)

spec := &tracing.Spec{
    ServiceName: "custom-service",
    Exporter: &tracing.ExporterSpec{
        Jaeger: &tracing.JaegerSpec{
            Mode:     "collector",
            Endpoint: "http://jaeger:14268/api/traces",
        },
    },
    HeaderFormat: tracing.HeaderFormatTraceContext,
}

tracer, err := tracing.New(spec)
if err != nil {
    // handle error
}
defer tracer.Close()

Instrumenting a Custom Filter

Custom filters can participate in distributed tracing by creating spans and propagating context:

func (f *MyFilter) Handle(req *http.Request) {
    span := f.tracer.NewSpanForHTTP(req.Context(), "my-filter", req)
    defer span.End()

    // add custom attributes
    span.SetAttributes(attribute.String("my-attr", "value"))

    // forward request …
    outReq, _ := http.NewRequest(req.Method, f.target, req.Body)
    span.InjectHTTP(outReq)
    // ...
}

Summary

• Easegress implements distributed tracing with OpenTelemetry through a centralized pkg/tracing/tracing.go module that wraps the OpenTelemetry SDK.
• The system supports multiple export protocols (Jaeger, Zipkin, OTLP) configured via ExporterSpec and instantiated in newExporters().
• Context propagation supports both W3C Trace-Context and B3 formats through a configurable TextMapPropagator.
• HTTP servers in pkg/object/httpserver/mux.go automatically create spans for incoming requests using NewSpanForHTTP(), with special handling for Cloudflare-traffic via pkg/tracing/cloudflare.go.
• Spans can inject trace context into outbound requests using InjectHTTP(), enabling end-to-end distributed tracing across service meshes.
• The Tracer.Close() method ensures graceful shutdown and span flushing via the underlying TracerProvider.Shutdown().

Frequently Asked Questions

What OpenTelemetry exporters does Easegress support?

Easegress supports Jaeger (jaeger.New), Zipkin (zipkin.New), and OTLP (otlptracegrpc.New or otlptracehttp.New) exporters. These are configured through the ExporterSpec struct and instantiated in the newExporters() function within pkg/tracing/tracing.go.

How does Easegress handle trace context propagation across services?

The system uses a stored propagation.TextMapPropagator created by newPropagator() to handle context injection and extraction. By default, it uses the W3C Trace-Context format, but can be configured to use B3 single-header format for legacy Zipkin compatibility via the headerFormat configuration option.

Can Easegress trace requests coming through Cloudflare?

Yes. The NewSpanForHTTP() method first attempts to detect Cloudflare-specific headers via newSpanForCloudflare() in pkg/tracing/cloudflare.go. When detected, it creates a parent span containing the Ray ID and edge request timestamps, then attaches a child span for the actual request processing, providing visibility into the CDN layer.

How do I ensure spans are flushed when Easegress reloads configuration?

Call Tracer.Close() during component shutdown or reload. This method invokes tp.Shutdown() on the underlying sdktrace.TracerProvider, which flushes any pending spans in the batch processor to the configured exporter before the tracer is destroyed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}