How to Define a Datadog Monitor Using Terraform for Metric-Based Alerting

Use the datadog_monitor resource in your Terraform configuration to declaratively define metric-based alerts, allowing Terraform to automatically provision and synchronize monitoring rules with your Datadog account through the provider plugin protocol.

When you define a Datadog monitor using Terraform, you transform manual alerting configuration into version-controlled infrastructure code. The hashicorp/terraform repository implements a provider plugin protocol that enables the Datadog provider to translate your HCL definitions into API calls, creating monitors that automatically stay synchronized with your configuration.

How Terraform Provisions Datadog Monitors

Provider Protocol and Resource Lifecycle

Terraform manages Datadog monitors through a standardized plugin protocol defined in docs/plugin-protocol/README.md. When you run terraform apply, the core engine invokes the Datadog provider binary and negotiates the resource lifecycle described in docs/resource-instance-change-lifecycle.md.

The provisioning workflow follows four distinct phases:

1. Initialize – terraform init downloads the terraform-provider-datadog binary and registers it with the plugin protocol.
2. Plan – Terraform calls PlanResourceChange for the datadog_monitor resource, calculating the diff between your HCL definition and the current state.
3. Apply – The provider's ApplyResourceChange function translates your configuration into a Datadog API request to create or modify the monitor.
4. Read – After creation, ReadResource pulls the monitor's current configuration back into the Terraform state file, enabling drift detection on subsequent runs.

This lifecycle ensures that when you define a Datadog monitor using Terraform, the resulting infrastructure remains declarative and self-healing.

Defining a Datadog Monitor in Terraform

Provider Configuration

Before creating monitors, configure the Datadog provider with your API and application keys. The provider binary uses these credentials to authenticate with Datadog's API endpoints.

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

Monitor Resource Definition

Use the datadog_monitor resource to define metric-based alerting rules. The following example creates a high CPU alert for production web servers:

resource "datadog_monitor" "high_cpu" {
  name               = "High CPU usage on web servers"
  type               = "metric alert"
  query              = "avg(last_5m):avg:system.cpu.user{role:web,env:prod} > 75"
  message            = <<EOT
    🚨 CPU usage exceeded 75% in the last 5 minutes.
    @slack-datadog-alerts
    Current value: {{value}}%
  EOT
  tags               = ["team:ops", "env:prod"]
  priority           = 1
  renotify_interval  = 10
  evaluation_delay   = 300
  notify_no_data     = false
  no_data_timeframe  = 10
}

Key Configuration Attributes

When you define a Datadog monitor using Terraform, these arguments control the alerting behavior:

• type – Must be "metric alert" for metric-based thresholds. Other valid types include "log alert", "trace-analytics alert", and "synthetics alert".
• query – The Datadog query string using the query syntax. The example uses avg(last_5m):avg:system.cpu.user{role:web,env:prod} > 75 to trigger when the 5-minute average exceeds 75%.
• message – The notification payload supporting template variables like {{value}} and @mentions for Slack, PagerDuty, or email destinations.
• renotify_interval – Minutes between renotifications while the alert remains triggered.
• evaluation_delay – Seconds to wait before evaluating the query, useful for avoiding false positives during metric reporting delays.

Deploying the Monitor

After defining your configuration, use the standard Terraform workflow to provision the monitor:

terraform init          # Download the Datadog provider binary

terraform fmt           # Format HCL syntax

terraform validate      # Validate configuration syntax

terraform plan -out=plan.out   # Preview changes

terraform apply plan.out        # Create the monitor in Datadog

Once applied, Terraform stores the monitor's ID and configuration in the state file. Subsequent terraform plan commands will detect any manual changes made through the Datadog UI and propose corrections to match your HCL definition.

Summary

• Define a Datadog monitor using Terraform by declaring a datadog_monitor resource with metric queries, thresholds, and notification settings.
• The Datadog provider implements the Terraform plugin protocol documented in docs/plugin-protocol/README.md to translate HCL into API calls.
• Terraform's resource lifecycle, detailed in docs/resource-instance-change-lifecycle.md, ensures monitors remain synchronized through plan, apply, and read operations.
• Store monitor definitions in version control to enable code review, automated testing, and drift detection for your alerting infrastructure.

Frequently Asked Questions

How does Terraform authenticate with the Datadog API?

Terraform authenticates using the api_key and app_key arguments configured in the datadog provider block. These credentials are passed to the provider binary, which includes them in HTTP headers when calling Datadog's REST API endpoints. Store these keys in environment variables or a secrets manager rather than committing them to version control.

What happens if someone modifies the monitor in the Datadog UI after Terraform creates it?

Terraform detects the drift during the next terraform plan execution. The provider's ReadResource function, implemented according to the lifecycle in docs/resource-instance-change-lifecycle.md, queries the current state from Datadog and compares it to the HCL configuration. Terraform will propose a plan to revert the manual changes and restore the monitor to the declared configuration.

Can I use Terraform to manage multiple Datadog monitors with similar configurations?

Yes. Use Terraform's for_each or count meta-arguments to iterate over a map or list of monitor definitions. Define the common parameters in a locals block or a Terraform module, then instantiate multiple datadog_monitor resources with specific queries and thresholds. This approach reduces duplication and ensures consistent tagging and notification settings across your alerting fleet.

Why does my monitor show "No Data" immediately after Terraform creates it?

Datadog monitors may report "No Data" if the metric query evaluates before data points arrive from your agents or integrations. Set the evaluation_delay attribute on the datadog_monitor resource to specify a delay in seconds before Datadog begins evaluating the query. For example, evaluation_delay = 300 waits five minutes, allowing metrics to populate and preventing false "No Data" alerts during provisioning.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}