How to Implement Distributed Tracing with OpenTelemetry in Kratos

Kratos provides first-class middleware in middleware/tracing that wraps OpenTelemetry APIs to automatically create spans, propagate distributed context across services, and inject trace IDs into logs for both server and client operations.

The go-kratos/kratos framework includes built-in support for distributed tracing via OpenTelemetry, enabling microservices to track requests across network boundaries without invasive code changes. By leveraging the tracing middleware, developers can capture detailed latency data and error information while maintaining compatibility with Jaeger, Zipkin, OTLP, and other backends.

Architecture of Kratos Distributed Tracing

Kratos abstracts OpenTelemetry concepts into transport-aware middleware that hooks into HTTP and gRPC handlers automatically.

Core Components

The implementation spans three primary source files under middleware/tracing/:

• Option and middleware constructors (tracing.go): Defines Server() and Client() middleware factory functions along with configuration options like WithTracerProvider and WithPropagator. Also exports TraceID() and SpanID() log valuers.

• Tracer wrapper (tracer.go): Encapsulates an otel/trace.Tracer instance and handles span lifecycle management—starting spans with proper kind designation (trace.SpanKindServer or trace.SpanKindClient), injecting/extracting carriers, recording errors, and finalizing spans via Tracer.End.

• Metadata propagator (metadata.go): A custom TextMapPropagator that injects the service name into the carrier as x-md-service-name, allowing downstream services to identify the origin of requests.

Server Request Flow

When a request enters a Kratos server:

1. The transport layer creates a transport.Transporter (HTTP or gRPC) that holds request headers.
2. The Server middleware extracts incoming propagation headers via transport.FromServerContext and calls Tracer.Start.
3. A server span (trace.SpanKindServer) is created and stored in the context.
4. Downstream handlers can fetch TraceID() or SpanID() or create child spans.
5. After the handler returns, Tracer.End records status, captures error details, and finishes the span.

Client Call Flow

When a Kratos client initiates an outbound call:

1. The client creates a transport.Transporter with an empty header carrier.
2. The Client middleware starts a client span (trace.SpanKindClient) via Tracer.Start and injects propagation headers into the carrier.
3. The underlying RPC or HTTP client sends the request; the downstream service extracts the headers as described above.
4. Upon completion, Tracer.End finalizes the span with duration and status codes.

Both flows rely on a TracerProvider. If none is supplied via WithTracerProvider, the middleware falls back to the globally registered provider (otel.SetTracerProvider), making it trivial to plug in any OpenTelemetry exporter.

Configuring OpenTelemetry Tracing in Kratos

Initialize your tracer provider and pass it to the Kratos application builder:

import (
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/propagation"
    "github.com/go-kratos/kratos/v2"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
)

func main() {
    // Initialize a Jaeger exporter
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint())
    if err != nil {
        panic(err)
    }
    tp := trace.NewTracerProvider(trace.WithBatcher(exp))

    // Build the Kratos application with tracing middleware
    app := kratos.New(
        kratos.Server(/* your HTTP or gRPC server */),
        kratos.Middleware(
            tracing.Server(
                tracing.WithTracerProvider(tp),
                tracing.WithPropagator(
                    propagation.NewCompositeTextMapPropagator(
                        propagation.Baggage{},
                        propagation.TraceContext{},
                    ),
                ),
            ),
        ),
    )
}

• WithTracerProvider injects your configured provider (Jaeger-backed in this example).
• WithPropagator customizes context propagation; the default includes W3C TraceContext, Baggage, and the Kratos metadata propagator.

Logging Trace IDs for Correlation

Correlate logs with traces by embedding trace identifiers into your logger using the provided valuers:

import (
    "os"
    "github.com/go-kratos/kratos/v2/log"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
)

func initLogger() log.Logger {
    logger := log.NewStdLogger(os.Stdout)
    logger = log.With(logger, "trace_id", tracing.TraceID())
    logger = log.With(logger, "span_id", tracing.SpanID())
    return logger
}

Inside any handler, log.WithContext(ctx, logger).Log("msg", "processing") automatically outputs the current trace and span IDs, enabling seamless correlation between distributed traces and application logs.

Using Alternative Tracing Backends

Because the middleware depends only on the OpenTelemetry API (trace.TracerProvider interface), you can replace the backend without changing application code. Supply any provider implementation—Zipkin, OTLP, OpenCensus bridge, or a proprietary custom provider—to tracing.WithTracerProvider(). This abstraction ensures your instrumentation remains vendor-neutral while supporting observability platform migrations.

Practical Code Examples

Server-Side HTTP Setup

package main

import (
    "log"
    "github.com/go-kratos/kratos/v2"
    "github.com/go-kratos/kratos/v2/transport/http"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/propagation"
)

func main() {
    exp, _ := jaeger.New(jaeger.WithCollectorEndpoint())
    tp := trace.NewTracerProvider(trace.WithBatcher(exp))

    srv := http.NewServer(http.Address(":8000"))

    app := kratos.New(
        kratos.Server(srv),
        kratos.Middleware(
            tracing.Server(
                tracing.WithTracerProvider(tp),
                tracing.WithPropagator(
                    propagation.NewCompositeTextMapPropagator(
                        propagation.Baggage{},
                        propagation.TraceContext{},
                    ),
                ),
            ),
        ),
    )

    if err := app.Run(); err != nil {
        log.Fatal(err)
    }
}

Client-Side gRPC Setup

package client

import (
    "context"
    "github.com/go-kratos/kratos/v2/transport/grpc"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/propagation"
)

func NewGRPCConn() (*grpc.ClientConn, error) {
    exp, _ := jaeger.New(jaeger.WithCollectorEndpoint())
    tp := trace.NewTracerProvider(trace.WithBatcher(exp))

    conn, err := grpc.DialInsecure(
        context.Background(),
        grpc.Endpoint("localhost:9000"),
        grpc.WithMiddleware(
            tracing.Client(
                tracing.WithTracerProvider(tp),
                tracing.WithPropagator(
                    propagation.NewCompositeTextMapPropagator(
                        propagation.Baggage{},
                        propagation.TraceContext{},
                    ),
                ),
            ),
        ),
    )
    return conn, err
}

Accessing Trace IDs in Business Logic

func (s *MyService) Hello(ctx context.Context, req *pb.HelloRequest) (*pb.HelloReply, error) {
    // Extract IDs for manual logging or response headers
    tid := tracing.TraceID()(ctx).(string)
    sid := tracing.SpanID()(ctx).(string)

    log.Infof("handling request trace_id=%s span_id=%s", tid, sid)
    
    return &pb.HelloReply{Message: "Hello"}, nil
}

Key Source Files

Reference these files when extending or debugging tracing behavior:

• middleware/tracing/tracing.go: Middleware entry points (Server, Client) and log valuers (TraceID, SpanID).
• middleware/tracing/tracer.go: Tracer struct implementation handling span creation, carrier injection/extraction, and finalization.
• middleware/tracing/metadata.go: Custom propagator for service name metadata (x-md-service-name).
• middleware/tracing/tracing_test.go: End-to-end tests demonstrating client and server middleware behavior.

Summary

• Built-in Middleware: Kratos provides tracing.Server and tracing.Client in middleware/tracing/tracing.go that wrap OpenTelemetry without transport-specific code.
• Automatic Propagation: The middleware extracts headers on ingress and injects them on egress using W3C standards and custom metadata carriers.
• Flexible Backends: Any OpenTelemetry-compatible TracerProvider works; default fallback uses the global provider.
• Log Correlation: Use tracing.TraceID() and tracing.SpanID() valuers to embed identifiers in structured logs.

Frequently Asked Questions

Can I use Zipkin or OTLP instead of Jaeger with Kratos tracing?

Yes. The middleware accepts any implementation of the OpenTelemetry trace.TracerProvider interface. Initialize a Zipkin exporter or OTLP exporter from the OpenTelemetry SDK, create a provider with trace.NewTracerProvider(), and pass it via tracing.WithTracerProvider().

How do I retrieve the current trace ID inside a service handler?

Use the tracing.TraceID() function as a log.Valuer or call it directly with the request context: tid := tracing.TraceID()(ctx).(string). The same applies to tracing.SpanID() for the span identifier. These functions extract the current span from the context managed by the middleware.

Does Kratos automatically propagate trace context between microservices?

Yes. The Client middleware injects propagation headers (W3C TraceContext and Baggage by default) into outgoing requests via the Tracer wrapper. The Server middleware extracts these headers from incoming transports. The custom Metadata propagator also adds the service name to aid request lineage debugging.

What happens if I don't configure a TracerProvider?

The middleware falls back to the globally registered OpenTelemetry tracer provider. If no provider has been registered globally, it uses the noop provider, meaning spans are created but discarded. This ensures your application runs without errors even when tracing is not configured, following a safe "instrumentation off" pattern.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}