# How to Implement Distributed Tracing with OpenTelemetry in Kratos

> Learn how to implement distributed tracing with OpenTelemetry in Kratos. Kratos middleware automatically creates spans, propagates context, and injects trace IDs into logs for seamless observability.

- Repository: [Kratos/kratos](https://github.com/go-kratos/kratos)
- Tags: how-to-guide
- Published: 2026-03-02

---

**Kratos provides first-class middleware in `middleware/tracing` that wraps OpenTelemetry APIs to automatically create spans, propagate distributed context across services, and inject trace IDs into logs for both server and client operations.**

The go-kratos/kratos framework includes built-in support for distributed tracing via OpenTelemetry, enabling microservices to track requests across network boundaries without invasive code changes. By leveraging the tracing middleware, developers can capture detailed latency data and error information while maintaining compatibility with Jaeger, Zipkin, OTLP, and other backends.

## Architecture of Kratos Distributed Tracing

Kratos abstracts OpenTelemetry concepts into transport-aware middleware that hooks into HTTP and gRPC handlers automatically.

### Core Components

The implementation spans three primary source files under `middleware/tracing/`:

- **`Option` and middleware constructors** ([`tracing.go`](https://github.com/go-kratos/kratos/blob/main/tracing.go)): Defines `Server()` and `Client()` middleware factory functions along with configuration options like `WithTracerProvider` and `WithPropagator`. Also exports `TraceID()` and `SpanID()` log valuers.

- **`Tracer` wrapper** ([`tracer.go`](https://github.com/go-kratos/kratos/blob/main/tracer.go)): Encapsulates an `otel/trace.Tracer` instance and handles span lifecycle management—starting spans with proper kind designation (`trace.SpanKindServer` or `trace.SpanKindClient`), injecting/extracting carriers, recording errors, and finalizing spans via `Tracer.End`.

- **`Metadata` propagator** ([`metadata.go`](https://github.com/go-kratos/kratos/blob/main/metadata.go)): A custom `TextMapPropagator` that injects the service name into the carrier as `x-md-service-name`, allowing downstream services to identify the origin of requests.

### Server Request Flow

When a request enters a Kratos server:

1. The transport layer creates a `transport.Transporter` (HTTP or gRPC) that holds request headers.
2. The `Server` middleware extracts incoming propagation headers via `transport.FromServerContext` and calls `Tracer.Start`.
3. A **server span** (`trace.SpanKindServer`) is created and stored in the context.
4. Downstream handlers can fetch `TraceID()` or `SpanID()` or create child spans.
5. After the handler returns, `Tracer.End` records status, captures error details, and finishes the span.

### Client Call Flow

When a Kratos client initiates an outbound call:

1. The client creates a `transport.Transporter` with an empty header carrier.
2. The `Client` middleware starts a **client span** (`trace.SpanKindClient`) via `Tracer.Start` and injects propagation headers into the carrier.
3. The underlying RPC or HTTP client sends the request; the downstream service extracts the headers as described above.
4. Upon completion, `Tracer.End` finalizes the span with duration and status codes.

Both flows rely on a `TracerProvider`. If none is supplied via `WithTracerProvider`, the middleware falls back to the globally registered provider (`otel.SetTracerProvider`), making it trivial to plug in any OpenTelemetry exporter.

## Configuring OpenTelemetry Tracing in Kratos

Initialize your tracer provider and pass it to the Kratos application builder:

```go
import (
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/propagation"
    "github.com/go-kratos/kratos/v2"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
)

func main() {
    // Initialize a Jaeger exporter
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint())
    if err != nil {
        panic(err)
    }
    tp := trace.NewTracerProvider(trace.WithBatcher(exp))

    // Build the Kratos application with tracing middleware
    app := kratos.New(
        kratos.Server(/* your HTTP or gRPC server */),
        kratos.Middleware(
            tracing.Server(
                tracing.WithTracerProvider(tp),
                tracing.WithPropagator(
                    propagation.NewCompositeTextMapPropagator(
                        propagation.Baggage{},
                        propagation.TraceContext{},
                    ),
                ),
            ),
        ),
    )
}

```

- **`WithTracerProvider`** injects your configured provider (Jaeger-backed in this example).
- **`WithPropagator`** customizes context propagation; the default includes W3C TraceContext, Baggage, and the Kratos metadata propagator.

## Logging Trace IDs for Correlation

Correlate logs with traces by embedding trace identifiers into your logger using the provided valuers:

```go
import (
    "os"
    "github.com/go-kratos/kratos/v2/log"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
)

func initLogger() log.Logger {
    logger := log.NewStdLogger(os.Stdout)
    logger = log.With(logger, "trace_id", tracing.TraceID())
    logger = log.With(logger, "span_id", tracing.SpanID())
    return logger
}

```

Inside any handler, `log.WithContext(ctx, logger).Log("msg", "processing")` automatically outputs the current trace and span IDs, enabling seamless correlation between distributed traces and application logs.

## Using Alternative Tracing Backends

Because the middleware depends only on the OpenTelemetry API (`trace.TracerProvider` interface), you can replace the backend without changing application code. Supply any provider implementation—Zipkin, OTLP, OpenCensus bridge, or a proprietary custom provider—to `tracing.WithTracerProvider()`. This abstraction ensures your instrumentation remains vendor-neutral while supporting observability platform migrations.

## Practical Code Examples

### Server-Side HTTP Setup

```go
package main

import (
    "log"
    "github.com/go-kratos/kratos/v2"
    "github.com/go-kratos/kratos/v2/transport/http"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/propagation"
)

func main() {
    exp, _ := jaeger.New(jaeger.WithCollectorEndpoint())
    tp := trace.NewTracerProvider(trace.WithBatcher(exp))

    srv := http.NewServer(http.Address(":8000"))

    app := kratos.New(
        kratos.Server(srv),
        kratos.Middleware(
            tracing.Server(
                tracing.WithTracerProvider(tp),
                tracing.WithPropagator(
                    propagation.NewCompositeTextMapPropagator(
                        propagation.Baggage{},
                        propagation.TraceContext{},
                    ),
                ),
            ),
        ),
    )

    if err := app.Run(); err != nil {
        log.Fatal(err)
    }
}

```

### Client-Side gRPC Setup

```go
package client

import (
    "context"
    "github.com/go-kratos/kratos/v2/transport/grpc"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/propagation"
)

func NewGRPCConn() (*grpc.ClientConn, error) {
    exp, _ := jaeger.New(jaeger.WithCollectorEndpoint())
    tp := trace.NewTracerProvider(trace.WithBatcher(exp))

    conn, err := grpc.DialInsecure(
        context.Background(),
        grpc.Endpoint("localhost:9000"),
        grpc.WithMiddleware(
            tracing.Client(
                tracing.WithTracerProvider(tp),
                tracing.WithPropagator(
                    propagation.NewCompositeTextMapPropagator(
                        propagation.Baggage{},
                        propagation.TraceContext{},
                    ),
                ),
            ),
        ),
    )
    return conn, err
}

```

### Accessing Trace IDs in Business Logic

```go
func (s *MyService) Hello(ctx context.Context, req *pb.HelloRequest) (*pb.HelloReply, error) {
    // Extract IDs for manual logging or response headers
    tid := tracing.TraceID()(ctx).(string)
    sid := tracing.SpanID()(ctx).(string)

    log.Infof("handling request trace_id=%s span_id=%s", tid, sid)
    
    return &pb.HelloReply{Message: "Hello"}, nil
}

```

## Key Source Files

Reference these files when extending or debugging tracing behavior:

- **[`middleware/tracing/tracing.go`](https://github.com/go-kratos/kratos/blob/main/middleware/tracing/tracing.go)**: Middleware entry points (`Server`, `Client`) and log valuers (`TraceID`, `SpanID`).
- **[`middleware/tracing/tracer.go`](https://github.com/go-kratos/kratos/blob/main/middleware/tracing/tracer.go)**: `Tracer` struct implementation handling span creation, carrier injection/extraction, and finalization.
- **[`middleware/tracing/metadata.go`](https://github.com/go-kratos/kratos/blob/main/middleware/tracing/metadata.go)**: Custom propagator for service name metadata (`x-md-service-name`).
- **[`middleware/tracing/tracing_test.go`](https://github.com/go-kratos/kratos/blob/main/middleware/tracing/tracing_test.go)**: End-to-end tests demonstrating client and server middleware behavior.

## Summary

- **Built-in Middleware**: Kratos provides `tracing.Server` and `tracing.Client` in [`middleware/tracing/tracing.go`](https://github.com/go-kratos/kratos/blob/main/middleware/tracing/tracing.go) that wrap OpenTelemetry without transport-specific code.
- **Automatic Propagation**: The middleware extracts headers on ingress and injects them on egress using W3C standards and custom metadata carriers.
- **Flexible Backends**: Any OpenTelemetry-compatible `TracerProvider` works; default fallback uses the global provider.
- **Log Correlation**: Use `tracing.TraceID()` and `tracing.SpanID()` valuers to embed identifiers in structured logs.

## Frequently Asked Questions

### Can I use Zipkin or OTLP instead of Jaeger with Kratos tracing?

Yes. The middleware accepts any implementation of the OpenTelemetry `trace.TracerProvider` interface. Initialize a Zipkin exporter or OTLP exporter from the OpenTelemetry SDK, create a provider with `trace.NewTracerProvider()`, and pass it via `tracing.WithTracerProvider()`.

### How do I retrieve the current trace ID inside a service handler?

Use the `tracing.TraceID()` function as a `log.Valuer` or call it directly with the request context: `tid := tracing.TraceID()(ctx).(string)`. The same applies to `tracing.SpanID()` for the span identifier. These functions extract the current span from the context managed by the middleware.

### Does Kratos automatically propagate trace context between microservices?

Yes. The `Client` middleware injects propagation headers (W3C TraceContext and Baggage by default) into outgoing requests via the `Tracer` wrapper. The `Server` middleware extracts these headers from incoming transports. The custom `Metadata` propagator also adds the service name to aid request lineage debugging.

### What happens if I don't configure a TracerProvider?

The middleware falls back to the globally registered OpenTelemetry tracer provider. If no provider has been registered globally, it uses the noop provider, meaning spans are created but discarded. This ensures your application runs without errors even when tracing is not configured, following a safe "instrumentation off" pattern.