How to Implement Observability with Cloud Trace and Logging in agents-cli

The google/agents-cli automatically wires OpenTelemetry into every scaffolded project, enabling Cloud Trace and prompt-response logging to GCS/BigQuery simply by configuring standard environment variables.

The google/agents-cli tool streamlines Google Cloud agent development by embedding observability directly into the scaffolded project template. This provides out-of-the-box integration with Cloud Trace and structured logging without requiring manual instrumentation code. By leveraging the bootstrap logic found in the scaffolded entry point and a few environment variables, developers gain end-to-end visibility into LLM calls, tool executions, and distributed traces across local development and production deployments.

How the Observability Stack Works

OpenTelemetry SDK Initialization

The scaffolded main.go located at src/google/agents/cli/scaffold/base_templates/go/main.go initializes the OpenTelemetry SDK during application startup. It constructs a TracerProvider using sdktrace.NewTracerProvider() configured with a batch span processor and a resource descriptor that tags every span with the service name via semconv.ServiceName(projectName).

Cloud Trace Exporter

Spans are exported to Google Cloud Trace using the cloudtrace.New() exporter from github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace. The exporter initialization first attempts Application Default Credentials; if authentication fails, it automatically retries using the project ID specified in the GOOGLE_CLOUD_PROJECT environment variable.

Context Propagation Across Boundaries

To ensure trace continuity across HTTP and gRPC calls, the bootstrap code installs a composite propagator using otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})). This injects the trace context into outgoing requests so that downstream services, including the ADK launcher, participate in the same trace.

Prompt-Response Logging Pipeline

When LOGS_BUCKET_NAME is set, the GenAI instrumentation layer captures model inputs and outputs as NDJSON files in the specified GCS bucket. The Terraform configuration in deployment/terraform/single-project/telemetry.tf (generated when using the --bq-analytics scaffold flag) provisions a log sink that streams these files into a BigQuery table named gen_ai_client_inference_operation_details, with a standardized view completions_view for analytics queries.

Required Environment Variables

Variable	Purpose	Recommended Values
LOGS_BUCKET_NAME	GCS bucket URI where prompt-response NDJSON files are stored	gs://my-project-logs
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT	Controls payload granularity	false (disable), NO_CONTENT (metadata only), true (full content)
GOOGLE_CLOUD_PROJECT	Fallback project ID for the Cloud Trace exporter	my-gcp-project-id
GENAI_TELEMETRY_PATH	Subdirectory within the bucket for log files	completions (default)

Enabling Observability in Local Development

To activate Cloud Trace and logging when using agents-cli playground, export the required variables before launching the command:

export LOGS_BUCKET_NAME="gs://my-project-my-agent-logs"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="NO_CONTENT"
export GOOGLE_CLOUD_PROJECT="my-dev-project"
agents-cli playground

The scaffolded main.go automatically detects these variables, initializes the cloudtrace.New() exporter, and configures the global TracerProvider via otel.SetTracerProvider(tp).

Disabling Logging in Production

To disable prompt-response logging in deployed environments for compliance or cost reasons, set the instrumentation variable to false in your Terraform service configuration:

env {
  name  = "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"
  value = "false"
}

Apply the configuration:

cd deployment/terraform/single-project
terraform apply -var-file=vars/env.tfvars

Note that disabling this variable stops NDJSON generation; existing sinks and BigQuery tables remain but receive no new data.

Verifying the Integration

Cloud Trace Verification

After deploying or running locally, navigate to Cloud Console → Trace → Trace Explorer to view the distributed trace tree. Each agent execution generates spans for LLM calls, tool invocations, and custom logic, all tagged with the service name defined in the semconv.ServiceName resource attribute.

BigQuery and GCS Verification

Confirm that logs are reaching the storage layer:

PROJECT_ID="your-project-id"
PROJECT_NAME="my-agent"

# List NDJSON files in the telemetry bucket

gsutil ls gs://${PROJECT_ID}-${PROJECT_NAME}-logs/completions/

# Query the raw inference table

bq query --use_legacy_sql=false \
  "SELECT COUNT(*) FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.gen_ai_client_inference_operation_details\`"

# Query the analytics view

bq query --use_legacy_sql=false \
  "SELECT * FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.completions_view\` LIMIT 10"

If data is missing, verify that the service account has storage.objectCreator permissions on the bucket and that the application logs contain no telemetry-setup warnings.

Summary

• The google/agents-cli scaffolds OpenTelemetry automatically in src/google/agents/cli/scaffold/base_templates/go/main.go, requiring zero manual instrumentation for basic observability.
• Cloud Trace export is handled by cloudtrace.New() with automatic fallback to the GOOGLE_CLOUD_PROJECT environment variable.
• Prompt-response logging activates when LOGS_BUCKET_NAME is set, writing NDJSON to GCS and syncing to BigQuery via the Terraform-managed log sinks in telemetry.tf.
• Control data capture granularity using OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT—set to false to disable, NO_CONTENT for metadata only, or true for full payloads.
• Distributed traces propagate automatically across service boundaries using W3C TraceContext and Baggage propagators configured via propagation.NewCompositeTextMapPropagator.

Frequently Asked Questions

Does agents-cli require code changes to enable Cloud Trace?

No. The scaffolded main.go automatically initializes the OpenTelemetry SDK and Cloud Trace exporter when valid credentials are present. You only need to ensure that GOOGLE_CLOUD_PROJECT is set or Application Default Credentials are configured, and the TracerProvider will be established via otel.SetTracerProvider(tp).

How do I disable prompt-response logging for compliance reasons?

Set the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to false in your deployment configuration. In Terraform-managed deployments, add this to the env block in service.tf and run terraform apply to stop the GenAI instrumentation from capturing message content.

Where are the log files physically stored before reaching BigQuery?

NDJSON log files are written to the GCS bucket specified by LOGS_BUCKET_NAME under the path defined by GENAI_TELEMETRY_PATH (defaulting to completions/). The Terraform configuration referenced in the observability documentation creates a log sink that exports these files to the BigQuery table gen_ai_client_inference_operation_details.

Can I use Cloud Trace without enabling the logging bucket?

Yes. Cloud Trace operates independently of the prompt-response logging pipeline. Trace spans are exported as long as the cloudtrace.New() exporter successfully initializes with valid GCP credentials, regardless of whether LOGS_BUCKET_NAME is configured or the BigQuery sink exists.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}