# How to Connect OpenSRE to Datadog for Observability: Complete Integration Guide

> Easily connect OpenSRE to Datadog for enhanced observability. Follow our guide to store keys and verify integration for seamless monitoring and insights.

- Repository: [Tracer/opensre](https://github.com/Tracer-Cloud/opensre)
- Tags: how-to-guide
- Published: 2026-04-18

---

**You can connect OpenSRE to Datadog by storing your API and application keys in `~/.tracer/integrations.json` or environment variables, then running `opensre integrations verify datadog` to confirm the connection.**

OpenSRE, an open-source root-cause analysis platform from Tracer-Cloud, connects directly to Datadog to query logs, monitors, and Kubernetes telemetry. This integration allows you to connect OpenSRE to Datadog for observability without writing custom API wrappers, leveraging the built-in `DatadogClient` and configuration models.

## Prerequisites and Credential Storage

Before querying Datadog data, you must provide valid credentials. OpenSRE supports two storage methods for the Datadog API key, application key, and optional site specification.

**File-based storage** persists credentials in `~/.tracer/integrations.json`. The CLI command `python -m app.integrations setup datadog` interactively prompts for these values and writes them to this file.

**Environment variable fallback** reads from `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` if the file is missing. Refer to `.env.example` lines 82-85 in the Tracer-Cloud/opensre repository for the exact variable names.

## Configuration Validation and Client Initialization

Once credentials are stored, OpenSRE validates and initializes the Datadog client through a typed configuration layer.

### Validating the DatadogIntegrationConfig Model

When the CLI or web UI loads the integration, the raw JSON is validated against `DatadogIntegrationConfig` in [`app/integrations/models.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/integrations/models.py) (lines 38-45). This Pydantic model normalizes the site parameter and guarantees both the API key and application key are present before any network calls occur.

### Creating the Client with make_client

The tool-level helpers in [`app/tools/DataDogLogsTool/_client.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/tools/DataDogLogsTool/_client.py) call `make_client` or `make_async_client` to instantiate the HTTP wrapper. The factory injects the supplied keys and optional site, defaulting to `_DEFAULT_SITE = "datadoghq.com"`.

```python

# app/tools/DataDogLogsTool/_client.py

def make_client(api_key: str | None, app_key: str | None,
                site: str = _DEFAULT_SITE) -> DatadogClient | None:
    if not api_key or not app_key:
        return None
    return DatadogClient(_config(api_key, app_key, site))

```

## Core Datadog API Operations in OpenSRE

The `DatadogClient` in [`app/services/datadog/client.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/services/datadog/client.py) implements three core endpoints used by OpenSRE for observability data.

**`search_logs`** calls the **Log Search API v2** (`POST /api/v2/logs/events/search`) to retrieve log entries matching a query string and time range.

**`list_monitors`** calls the **Monitor List API** (`GET /api/v1/monitor`) to fetch the complete catalogue of Datadog monitors and their statuses.

**`get_pods_on_node`** reuses `search_logs` to discover Kubernetes pod telemetry for a given node IP, enabling node-level RCA investigations.

The client builds the base URL from the site (`https://api.{site}`) and injects the required headers `DD-API-KEY` and `DD-APPLICATION-KEY` (see lines 35-38 and 64-71 of the client file).

## Verifying Your OpenSRE Datadog Connection

Before running any RCA workflows, validate the configuration using the built-in verification routine.

Run the CLI command:

```bash
opensre integrations verify datadog

```

This executes `app/integrations/verify.py::_verify_datadog` (lines 33-57), which validates the configuration, creates a `DatadogClient`, and performs a cheap `list_monitors` call to ensure the keys have the correct scope. A success message confirms connectivity to `api.datadoghq.com` (or your specified site).

## Using Datadog Tools in RCA Workflows

Once verified, OpenSRE nodes can invoke Datadog-specific tools to pull observability data into investigations.

### Querying Logs with query_datadog_logs

The `query_datadog_logs` function in [`app/tools/DataDogLogsTool/__init__.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/tools/DataDogLogsTool/__init__.py) forwards a user query to `DatadogClient.search_logs`.

```python
from app.tools.DataDogLogsTool import query_datadog_logs

logs = query_datadog_logs(
    api_key="YOUR_DD_API_KEY",
    app_key="YOUR_DD_APP_KEY",
    query="env:prod service:checkout status:error",
    time_range_minutes=30,
    limit=20,
)
print(logs)

```

### Accessing Monitors and Node-Pod Mappings

- **`query_datadog_monitors`** in [`app/tools/DataDogMonitorsTool/__init__.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/tools/DataDogMonitorsTool/__init__.py) returns the monitor catalogue.
- **`query_datadog_node_pods`** in [`app/tools/DataDogNodePodsTool/__init__.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/tools/DataDogNodePodsTool/__init__.py) surfaces pods running on a specific node based on log telemetry.

Each tool checks `is_configured` and returns a standard *unavailable* response when keys are missing.

## Generating Direct Datadog Console URLs

Utility functions in [`app/aws_urls.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/aws_urls.py) such as `build_datadog_logs_url` generate Datadog UI links for a given query. This allows you to jump directly from an RCA investigation in OpenSRE to the corresponding query in the Datadog console.

## Summary

- Store Datadog credentials in `~/.tracer/integrations.json` via the CLI or use `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` environment variables.
- The `DatadogIntegrationConfig` model in [`app/integrations/models.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/integrations/models.py) validates keys and normalizes the site parameter.
- Initialize the client using `make_client` from [`app/tools/DataDogLogsTool/_client.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/tools/DataDogLogsTool/_client.py), which wraps the low-level `DatadogClient` in [`app/services/datadog/client.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/services/datadog/client.py).
- Verify connectivity with `opensre integrations verify datadog` before running RCA workflows.
- Use `query_datadog_logs`, `query_datadog_monitors`, and `query_datadog_node_pods` to pull observability data into your investigations.

## Frequently Asked Questions

### Where does OpenSRE store Datadog credentials?

OpenSRE stores Datadog credentials in the per-user file `~/.tracer/integrations.json` when configured via the CLI. Alternatively, it reads from the environment variables `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as defined in `.env.example` lines 82-85.

### Which Datadog API endpoints does OpenSRE use?

OpenSRE uses three primary endpoints: the **Log Search API v2** (`POST /api/v2/logs/events/search`) via `search_logs`, the **Monitor List API** (`GET /api/v1/monitor`) via `list_monitors`, and a log-based discovery method via `get_pods_on_node`.

### How do I troubleshoot a failed Datadog connection in OpenSRE?

Run `opensre integrations verify datadog` to execute the verification routine in [`app/integrations/verify.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/integrations/verify.py). This performs a cheap `list_monitors` call that validates your API keys have the correct scope and confirms connectivity to `api.datadoghq.com` or your specified site.

### Can I use Datadog EU sites with OpenSRE?

Yes. When configuring the integration, specify the site parameter as `datadoghq.eu` (or your specific Datadog site). The `DatadogIntegrationConfig` model in [`app/integrations/models.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/integrations/models.py) normalizes this value, and the client factory in [`app/tools/DataDogLogsTool/_client.py`](https://github.com/Tracer-Cloud/opensre/blob/main/app/tools/DataDogLogsTool/_client.py) uses it to build the correct base URL `https://api.{site}`.