# How to Implement Observability with Cloud Trace and Logging in agents-cli

> Implement observability in agents-cli with Cloud Trace and logging. Automatically wire OpenTelemetry and send logs to GCS/BigQuery using environment variables.

- Repository: [Google/agents-cli](https://github.com/google/agents-cli)
- Tags: how-to-guide
- Published: 2026-07-01

---

**The `google/agents-cli` automatically wires OpenTelemetry into every scaffolded project, enabling Cloud Trace and prompt-response logging to GCS/BigQuery simply by configuring standard environment variables.**

The `google/agents-cli` tool streamlines Google Cloud agent development by embedding observability directly into the scaffolded project template. This provides out-of-the-box integration with Cloud Trace and structured logging without requiring manual instrumentation code. By leveraging the bootstrap logic found in the scaffolded entry point and a few environment variables, developers gain end-to-end visibility into LLM calls, tool executions, and distributed traces across local development and production deployments.

## How the Observability Stack Works

### OpenTelemetry SDK Initialization

The scaffolded [`main.go`](https://github.com/google/agents-cli/blob/main/main.go) located at [`src/google/agents/cli/scaffold/base_templates/go/main.go`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/scaffold/base_templates/go/main.go) initializes the OpenTelemetry SDK during application startup. It constructs a `TracerProvider` using `sdktrace.NewTracerProvider()` configured with a batch span processor and a resource descriptor that tags every span with the service name via `semconv.ServiceName(projectName)`.

### Cloud Trace Exporter

Spans are exported to Google Cloud Trace using the `cloudtrace.New()` exporter from `github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace`. The exporter initialization first attempts **Application Default Credentials**; if authentication fails, it automatically retries using the project ID specified in the `GOOGLE_CLOUD_PROJECT` environment variable.

### Context Propagation Across Boundaries

To ensure trace continuity across HTTP and gRPC calls, the bootstrap code installs a composite propagator using `otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))`. This injects the trace context into outgoing requests so that downstream services, including the ADK launcher, participate in the same trace.

### Prompt-Response Logging Pipeline

When `LOGS_BUCKET_NAME` is set, the GenAI instrumentation layer captures model inputs and outputs as NDJSON files in the specified GCS bucket. The Terraform configuration in `deployment/terraform/single-project/telemetry.tf` (generated when using the `--bq-analytics` scaffold flag) provisions a log sink that streams these files into a BigQuery table named `gen_ai_client_inference_operation_details`, with a standardized view `completions_view` for analytics queries.

## Required Environment Variables

| Variable | Purpose | Recommended Values |
|----------|---------|-------------------|
| `LOGS_BUCKET_NAME` | GCS bucket URI where prompt-response NDJSON files are stored | `gs://my-project-logs` |
| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | Controls payload granularity | `false` (disable), `NO_CONTENT` (metadata only), `true` (full content) |
| `GOOGLE_CLOUD_PROJECT` | Fallback project ID for the Cloud Trace exporter | `my-gcp-project-id` |
| `GENAI_TELEMETRY_PATH` | Subdirectory within the bucket for log files | `completions` (default) |

## Enabling Observability in Local Development

To activate Cloud Trace and logging when using `agents-cli playground`, export the required variables before launching the command:

```bash
export LOGS_BUCKET_NAME="gs://my-project-my-agent-logs"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="NO_CONTENT"
export GOOGLE_CLOUD_PROJECT="my-dev-project"
agents-cli playground

```

The scaffolded [`main.go`](https://github.com/google/agents-cli/blob/main/main.go) automatically detects these variables, initializes the `cloudtrace.New()` exporter, and configures the global `TracerProvider` via `otel.SetTracerProvider(tp)`.

## Disabling Logging in Production

To disable prompt-response logging in deployed environments for compliance or cost reasons, set the instrumentation variable to `false` in your Terraform service configuration:

```hcl
env {
  name  = "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"
  value = "false"
}

```

Apply the configuration:

```bash
cd deployment/terraform/single-project
terraform apply -var-file=vars/env.tfvars

```

Note that disabling this variable stops NDJSON generation; existing sinks and BigQuery tables remain but receive no new data.

## Verifying the Integration

### Cloud Trace Verification

After deploying or running locally, navigate to **Cloud Console → Trace → Trace Explorer** to view the distributed trace tree. Each agent execution generates spans for LLM calls, tool invocations, and custom logic, all tagged with the service name defined in the `semconv.ServiceName` resource attribute.

### BigQuery and GCS Verification

Confirm that logs are reaching the storage layer:

```bash
PROJECT_ID="your-project-id"
PROJECT_NAME="my-agent"

# List NDJSON files in the telemetry bucket

gsutil ls gs://${PROJECT_ID}-${PROJECT_NAME}-logs/completions/

# Query the raw inference table

bq query --use_legacy_sql=false \
  "SELECT COUNT(*) FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.gen_ai_client_inference_operation_details\`"

# Query the analytics view

bq query --use_legacy_sql=false \
  "SELECT * FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.completions_view\` LIMIT 10"

```

If data is missing, verify that the service account has `storage.objectCreator` permissions on the bucket and that the application logs contain no telemetry-setup warnings.

## Summary

- The `google/agents-cli` scaffolds OpenTelemetry automatically in [`src/google/agents/cli/scaffold/base_templates/go/main.go`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/scaffold/base_templates/go/main.go), requiring zero manual instrumentation for basic observability.
- Cloud Trace export is handled by `cloudtrace.New()` with automatic fallback to the `GOOGLE_CLOUD_PROJECT` environment variable.
- Prompt-response logging activates when `LOGS_BUCKET_NAME` is set, writing NDJSON to GCS and syncing to BigQuery via the Terraform-managed log sinks in `telemetry.tf`.
- Control data capture granularity using `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`—set to `false` to disable, `NO_CONTENT` for metadata only, or `true` for full payloads.
- Distributed traces propagate automatically across service boundaries using W3C TraceContext and Baggage propagators configured via `propagation.NewCompositeTextMapPropagator`.

## Frequently Asked Questions

### Does agents-cli require code changes to enable Cloud Trace?

No. The scaffolded [`main.go`](https://github.com/google/agents-cli/blob/main/main.go) automatically initializes the OpenTelemetry SDK and Cloud Trace exporter when valid credentials are present. You only need to ensure that `GOOGLE_CLOUD_PROJECT` is set or Application Default Credentials are configured, and the `TracerProvider` will be established via `otel.SetTracerProvider(tp)`.

### How do I disable prompt-response logging for compliance reasons?

Set the environment variable `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `false` in your deployment configuration. In Terraform-managed deployments, add this to the `env` block in `service.tf` and run `terraform apply` to stop the GenAI instrumentation from capturing message content.

### Where are the log files physically stored before reaching BigQuery?

NDJSON log files are written to the GCS bucket specified by `LOGS_BUCKET_NAME` under the path defined by `GENAI_TELEMETRY_PATH` (defaulting to `completions/`). The Terraform configuration referenced in the observability documentation creates a log sink that exports these files to the BigQuery table `gen_ai_client_inference_operation_details`.

### Can I use Cloud Trace without enabling the logging bucket?

Yes. Cloud Trace operates independently of the prompt-response logging pipeline. Trace spans are exported as long as the `cloudtrace.New()` exporter successfully initializes with valid GCP credentials, regardless of whether `LOGS_BUCKET_NAME` is configured or the BigQuery sink exists.