How to Connect OpenSRE to Datadog for Observability: Complete Integration Guide

You can connect OpenSRE to Datadog by storing your API and application keys in ~/.tracer/integrations.json or environment variables, then running opensre integrations verify datadog to confirm the connection.

OpenSRE, an open-source root-cause analysis platform from Tracer-Cloud, connects directly to Datadog to query logs, monitors, and Kubernetes telemetry. This integration allows you to connect OpenSRE to Datadog for observability without writing custom API wrappers, leveraging the built-in DatadogClient and configuration models.

Prerequisites and Credential Storage

Before querying Datadog data, you must provide valid credentials. OpenSRE supports two storage methods for the Datadog API key, application key, and optional site specification.

File-based storage persists credentials in ~/.tracer/integrations.json. The CLI command python -m app.integrations setup datadog interactively prompts for these values and writes them to this file.

Environment variable fallback reads from DD_API_KEY, DD_APP_KEY, and DD_SITE if the file is missing. Refer to .env.example lines 82-85 in the Tracer-Cloud/opensre repository for the exact variable names.

Configuration Validation and Client Initialization

Once credentials are stored, OpenSRE validates and initializes the Datadog client through a typed configuration layer.

Validating the DatadogIntegrationConfig Model

When the CLI or web UI loads the integration, the raw JSON is validated against DatadogIntegrationConfig in app/integrations/models.py (lines 38-45). This Pydantic model normalizes the site parameter and guarantees both the API key and application key are present before any network calls occur.

Creating the Client with make_client

The tool-level helpers in app/tools/DataDogLogsTool/_client.py call make_client or make_async_client to instantiate the HTTP wrapper. The factory injects the supplied keys and optional site, defaulting to _DEFAULT_SITE = "datadoghq.com".

# app/tools/DataDogLogsTool/_client.py

def make_client(api_key: str | None, app_key: str | None,
                site: str = _DEFAULT_SITE) -> DatadogClient | None:
    if not api_key or not app_key:
        return None
    return DatadogClient(_config(api_key, app_key, site))

Core Datadog API Operations in OpenSRE

The DatadogClient in app/services/datadog/client.py implements three core endpoints used by OpenSRE for observability data.

search_logs calls the Log Search API v2 (POST /api/v2/logs/events/search) to retrieve log entries matching a query string and time range.

list_monitors calls the Monitor List API (GET /api/v1/monitor) to fetch the complete catalogue of Datadog monitors and their statuses.

get_pods_on_node reuses search_logs to discover Kubernetes pod telemetry for a given node IP, enabling node-level RCA investigations.

The client builds the base URL from the site (https://api.{site}) and injects the required headers DD-API-KEY and DD-APPLICATION-KEY (see lines 35-38 and 64-71 of the client file).

Verifying Your OpenSRE Datadog Connection

Before running any RCA workflows, validate the configuration using the built-in verification routine.

Run the CLI command:

opensre integrations verify datadog

This executes app/integrations/verify.py::_verify_datadog (lines 33-57), which validates the configuration, creates a DatadogClient, and performs a cheap list_monitors call to ensure the keys have the correct scope. A success message confirms connectivity to api.datadoghq.com (or your specified site).

Using Datadog Tools in RCA Workflows

Once verified, OpenSRE nodes can invoke Datadog-specific tools to pull observability data into investigations.

Querying Logs with query_datadog_logs

The query_datadog_logs function in app/tools/DataDogLogsTool/__init__.py forwards a user query to DatadogClient.search_logs.

from app.tools.DataDogLogsTool import query_datadog_logs

logs = query_datadog_logs(
    api_key="YOUR_DD_API_KEY",
    app_key="YOUR_DD_APP_KEY",
    query="env:prod service:checkout status:error",
    time_range_minutes=30,
    limit=20,
)
print(logs)

Accessing Monitors and Node-Pod Mappings

• query_datadog_monitors in app/tools/DataDogMonitorsTool/__init__.py returns the monitor catalogue.
• query_datadog_node_pods in app/tools/DataDogNodePodsTool/__init__.py surfaces pods running on a specific node based on log telemetry.

Each tool checks is_configured and returns a standard unavailable response when keys are missing.

Generating Direct Datadog Console URLs

Utility functions in app/aws_urls.py such as build_datadog_logs_url generate Datadog UI links for a given query. This allows you to jump directly from an RCA investigation in OpenSRE to the corresponding query in the Datadog console.

Summary

• Store Datadog credentials in ~/.tracer/integrations.json via the CLI or use DD_API_KEY, DD_APP_KEY, and DD_SITE environment variables.
• The DatadogIntegrationConfig model in app/integrations/models.py validates keys and normalizes the site parameter.
• Initialize the client using make_client from app/tools/DataDogLogsTool/_client.py, which wraps the low-level DatadogClient in app/services/datadog/client.py.
• Verify connectivity with opensre integrations verify datadog before running RCA workflows.
• Use query_datadog_logs, query_datadog_monitors, and query_datadog_node_pods to pull observability data into your investigations.

Frequently Asked Questions

Where does OpenSRE store Datadog credentials?

OpenSRE stores Datadog credentials in the per-user file ~/.tracer/integrations.json when configured via the CLI. Alternatively, it reads from the environment variables DD_API_KEY, DD_APP_KEY, and DD_SITE as defined in .env.example lines 82-85.

Which Datadog API endpoints does OpenSRE use?

OpenSRE uses three primary endpoints: the Log Search API v2 (POST /api/v2/logs/events/search) via search_logs, the Monitor List API (GET /api/v1/monitor) via list_monitors, and a log-based discovery method via get_pods_on_node.

How do I troubleshoot a failed Datadog connection in OpenSRE?

Run opensre integrations verify datadog to execute the verification routine in app/integrations/verify.py. This performs a cheap list_monitors call that validates your API keys have the correct scope and confirms connectivity to api.datadoghq.com or your specified site.

Can I use Datadog EU sites with OpenSRE?

Yes. When configuring the integration, specify the site parameter as datadoghq.eu (or your specific Datadog site). The DatadogIntegrationConfig model in app/integrations/models.py normalizes this value, and the client factory in app/tools/DataDogLogsTool/_client.py uses it to build the correct base URL https://api.{site}.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}