# Paperclip AI OpenTelemetry Integration and Tracing Configuration: Complete Guide

> Integrate Paperclip AI with OpenTelemetry for distributed tracing. Learn how to configure tracing for your control plane by setting OTEL_EXPORTER_OTLP_ENDPOINT.

- Repository: [Paperclip/paperclip](https://github.com/paperclipai/paperclip)
- Tags: how-to-guide
- Published: 2026-08-12

---

**Paperclip AI provides opt-in OpenTelemetry auto-instrumentation that adds distributed tracing to the server-side control plane, activated only when `OTEL_EXPORTER_OTLP_ENDPOINT` is defined.**

This guide explains how Paperclip's observability layer works, how to configure tracing, and how to emit custom spans from your services. The implementation follows standard OpenTelemetry conventions while keeping overhead at zero when telemetry is disabled.

## How Paperclip's Telemetry System Works

The architecture separates concerns across three layers: **instrumentation bootstrap**, **telemetry client**, and **domain-specific event tracking**.

### Core Components

| Component | Source File | Purpose |
|-----------|-------------|---------|
| **Instrumentation Bootstrap** | [`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts) | Initializes the OpenTelemetry SDK, selects exporter protocol, creates startup tracer |
| **Telemetry Client** | [`server/src/telemetry.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/telemetry.ts) | Singleton wrapper exposing `getTelemetryClient()` and event helpers |
| **Shared Telemetry Library** | `packages/shared/src/telemetry/*` | Contract definitions and lightweight client for sandboxed plugins |
| **Service Emitters** | `server/src/services/*.ts` | Domain helpers like `trackRoutineRun()`, `trackAgentCreated()` |

### Bootstrap Flow

1. Server startup checks `process.env.OTEL_EXPORTER_OTLP_ENDPOINT`
2. If set, [[`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/instrumentation.ts) calls `bootstrapOtel(endpoint)` which dynamically imports the appropriate exporter package
3. The exporter resolves based on `OTEL_EXPORTER_OTLP_PROTOCOL` (lines 341–357)
4. A tracer object returns and stores in the singleton [`TelemetryClient`](https://github.com/paperclipai/paperclip/blob/master/server/src/telemetry.ts)
5. Services call domain-specific helpers that create spans with attached dimensions
6. On shutdown, `shutdownInstrumentation()` (line 287) flushes pending spans

### Exporter Resolution Logic

The bootstrap code handles three protocol variants:

- **`grpc`** — imports `@opentelemetry/exporter-trace-otlp-grpc`
- **`http/protobuf`** — imports `@opentelemetry/exporter-trace-otlp-proto`
- **`http/json`** — imports `@opentelemetry/exporter-trace-otlp-http`

If the package is missing for the requested protocol, the system logs a warning and falls back to a no-op tracer.

## Environment Variables for Configuration

Paperclip follows the OpenTelemetry specification for environment-based configuration:

| Variable | Required | Description |
|----------|----------|-------------|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Yes | Enables tracing and sets the collector URL (e.g., `http://localhost:4317`) |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | No | Transport protocol: `grpc` (default), `http/protobuf`, or `http/json` |
| `OTEL_RESOURCE_ATTRIBUTES` | No | Comma-separated key=value pairs added to every span (e.g., `service.version=1.2.3,deployment.env=production`) |

When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, `getTelemetryClient()` returns `undefined` and all tracing calls become no-ops.

## Key Source Files Reference

- **[[`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/instrumentation.ts)** — SDK initialization, exporter resolution, graceful shutdown
- **[[`server/src/telemetry.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/telemetry.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/telemetry.ts)** — Singleton client and `getTelemetryClient()` export
- **[[`packages/shared/src/telemetry/events.ts`](https://github.com/paperclipai/paperclip/blob/main/packages/shared/src/telemetry/events.ts)](https://github.com/paperclipai/paperclip/blob/master/packages/shared/src/telemetry/events.ts)** — Standardized event names and dimension types
- **[[`packages/shared/src/telemetry/client.ts`](https://github.com/paperclipai/paperclip/blob/main/packages/shared/src/telemetry/client.ts)](https://github.com/paperclipai/paperclip/blob/master/packages/shared/src/telemetry/client.ts)** — Lightweight client for plugin sandboxes
- **[[`server/src/services/routines.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/routines.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/services/routines.ts)** — Example usage with `trackRoutineRun`
- **[[`server/src/services/heartbeat.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/heartbeat.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/services/heartbeat.ts)** — `trackAgentFirstHeartbeat` implementation

## Configuration Examples

### Enabling Tracing in Development

```bash

# Point to local OpenTelemetry Collector

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

# Optional: use HTTP/JSON for easier debugging

export OTEL_EXPORTER_OTLP_PROTOCOL="http/json"

export OTEL_RESOURCE_ATTRIBUTES="service.name=paperclip-dev,deployment.environment=local"

pnpm dev

```

### Docker Compose with Jaeger

```yaml
services:
  paperclip:
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - OTEL_RESOURCE_ATTRIBUTES=service.name=paperclip,service.version=1.0.0
    depends_on:
      - jaeger

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # Jaeger UI

      - "4317:4317"    # OTLP gRPC

```

### Kubernetes with OpenTelemetry Operator

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: paperclip-collector
spec:
  mode: sidecar
  config: |
    exporters:
      otlp/jaeger:
        endpoint: jaeger-collector:4317
    service:
      pipelines:
        traces:
          exporters: [otlp/jaeger]

```

Apply the collector and set `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317` in the Paperclip deployment.

## Emitting Custom Spans

The telemetry client provides two patterns: **domain event helpers** for standard operations and **raw span creation** for custom instrumentation.

### Using Domain Event Helpers

```ts
// server/src/services/myService.ts
import { getTelemetryClient } from "../telemetry.js";
import { 
  trackRoutineRun, 
  trackAgentCreated,
  trackGoalResolved 
} from "@paperclipai/shared/telemetry";

export async function processRoutine(routineId: string) {
  const telemetry = getTelemetryClient();
  if (!telemetry) return;  // Tracing disabled, no overhead

  // Emits a "routine.run" span with standard dimensions
  await trackRoutineRun(telemetry, {
    routineId,
    agentId: "agent-123",
    startTime: Date.now(),
  });
}

```

### Creating Custom Spans Directly

```ts
import { getTelemetryClient } from "../telemetry.js";

export async function performComplexOperation(params: OperationParams) {
  const telemetry = getTelemetryClient();
  if (!telemetry) return;

  // Start a parent span
  await telemetry.tracer.startActiveSpan(
    "complex.operation",
    async (span) => {
      try {
        span.setAttribute("operation.params.count", params.items.length);
        
        // Child spans for sub-operations
        const results = await Promise.all(
          params.items.map((item, idx) =>
            telemetry.tracer.startActiveSpan(
              "complex.operation.process_item",
              async (childSpan) => {
                childSpan.setAttribute("item.index", idx);
                childSpan.setAttribute("item.id", item.id);
                
                const result = await process(item);
                
                childSpan.setAttribute("item.success", result.ok);
                childSpan.end();
                return result;
              }
            )
          )
        );
        
        span.setAttribute("operation.results.count", results.length);
        return results;
        
      } catch (err) {
        span.recordException(err);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        span.end();  // Always end the span
      }
    }
  );
}

```

### Using the Lightweight Client in Plugins

Sandboxes avoid pulling the full OpenTelemetry stack by using the shared client:

```ts
// Inside a plugin worker
import { makeTelemetryClient } from "@paperclipai/shared/telemetry/client";

export default async function pluginMain(ctx: { 
  tracer: any  // Injected by host
}) {
  const client = makeTelemetryClient(ctx.tracer);
  
  // Same API surface, minimal dependencies
  await client.tracer.startActiveSpan("plugin.initialize", async (span) => {
    span.setAttribute("plugin.name", "my-analyzer");
    
    const analysis = await analyzeDocument();
    
    span.setAttribute("analysis.findings.count", analysis.findings.length);
    span.end();
  });
}

```

## Graceful Shutdown Handling

Ensure pending spans flush before process exit:

```ts
import { shutdownInstrumentation } from "./instrumentation.js";

async function shutdown(signal: string) {
  console.log(`Received ${signal}, starting graceful shutdown...`);
  
  // Flush telemetry
  await shutdownInstrumentation();  // From line 287 in instrumentation.ts
  
  // Close database connections, etc.
  await closeConnections();
  
  process.exit(0);
}

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));

```

## Troubleshooting

| Symptom | Cause | Resolution |
|---------|-------|------------|
| No traces in collector | `OTEL_EXPORTER_OTLP_ENDPOINT` undefined | Verify environment variable is set and accessible to Node process |
| `MODULE_NOT_FOUND` on startup | Exporter package missing for protocol | Install matching package: `@opentelemetry/exporter-trace-otlp-grpc` or http/proto variant |
| High memory usage | Default instrumentations enabled | Check [`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts) — `fs`, `dns`, `net` are explicitly disabled in the bootstrap configuration |
| Missing custom attributes | Span ended before attributes set | Ensure `setAttribute()` calls occur before `span.end()` |

## Summary

- Paperclip's **OpenTelemetry integration is opt-in** via `OTEL_EXPORTER_OTLP_ENDPOINT` — zero overhead when disabled
- **[[`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/instrumentation.ts)** bootstraps the SDK and resolves exporters dynamically based on `OTEL_EXPORTER_OTLP_PROTOCOL`
- **[[`server/src/telemetry.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/telemetry.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/telemetry.ts)** provides the singleton client accessed via `getTelemetryClient()`
- The **[`packages/shared/src/telemetry/`](https://github.com/paperclipai/paperclip/tree/master/packages/shared/src/telemetry)** package isolates sandboxed plugins from heavy OpenTelemetry dependencies
- Domain helpers like `trackRoutineRun()` and `trackAgentCreated()` wrap raw span creation with consistent naming and dimensions
- Call `shutdownInstrumentation()` on graceful shutdown to flush pending traces

## Frequently Asked Questions

### How do I disable tracing without removing environment variables?

Set `OTEL_SDK_DISABLED=true`. This is the standard OpenTelemetry environment variable that prevents SDK initialization entirely, even when `OTEL_EXPORTER_OTLP_ENDPOINT` is present.

### Can I use a different collector protocol than gRPC?

Yes. Set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` or `http/json`. The bootstrap code in [[`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/instrumentation.ts) dynamically imports the matching exporter package. If the package isn't installed, it logs a warning and falls back to no-op.

### What happens if I call `getTelemetryClient()` before the server finishes starting?

The function returns `undefined` if the bootstrap hasn't completed or if `OTEL_EXPORTER_OTLP_ENDPOINT` is unset. Always check the return value before using the client — this pattern appears throughout service files like [[`server/src/services/routines.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/services/routines.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/services/routines.ts).

### How do I add custom dimensions to all spans automatically?

Use `OTEL_RESOURCE_ATTRIBUTES` for static metadata (service name, version, environment). For dynamic dimensions computed at runtime, wrap `getTelemetryClient()` and inject attributes via a span processor registered in [[`server/src/instrumentation.ts`](https://github.com/paperclipai/paperclip/blob/main/server/src/instrumentation.ts)](https://github.com/paperclipai/paperclip/blob/master/server/src/instrumentation.ts).