How Prometheus Uses Service Discovery to Dynamically Monitor Ephemeral Kubernetes Services

Prometheus leverages Kubernetes service discovery to automatically detect and monitor short-lived pods and services by querying the Kubernetes API for real-time metadata, eliminating manual configuration updates when containers scale or restart.

Prometheus implements a pull-based scraping architecture that requires up-to-date target lists to collect metrics from ephemeral Kubernetes workloads. According to the bregman-arie/devops-exercises repository, Prometheus achieves this through native integration with the Kubernetes control plane, treating the API server as the single source of truth for dynamic target discovery.

Kubernetes API Integration: The Source of Truth

Prometheus maintains current awareness of cluster state by establishing a watch on the Kubernetes API server. Instead of relying on static IP lists or external registries, Prometheus queries the API for resources including Pods, Endpoints, Services, and Ingresses to build its scrape target list.

When controllers such as Deployments, DaemonSets, or Jobs create, scale, or terminate pods, the API server broadcasts these changes. Prometheus consumes these events in real time, immediately adding new pods to its scrape pool and removing terminated instances. This mechanism ensures that monitoring coverage matches the actual running workload without operator intervention.

Native Service Discovery for Container Environments

The repository's containers documentation explicitly identifies "Native service discovery" as a core capability of modern container runtimes. As noted in topics/containers/README.md at line 1131, Prometheus leverages this principle by bypassing external service-registry tools. The Kubernetes control plane already exposes the required metadata—including labels, annotations, and port specifications—making third-party discovery mechanisms unnecessary.

This native approach allows Prometheus to resolve service addresses through DNS or direct pod IP lookup using the internal cluster networking layer, ensuring that scraped targets are always reachable within the cluster network.

Label-Driven Target Selection and Filtering

Prometheus implements granular, per-team monitoring through label selectors and annotations rather than broad IP range scanning. Users annotate pods or services with specific Prometheus metadata to opt-in to scraping.

Annotation-Based Opt-In

The standard convention uses the prometheus.io/scrape: "true" annotation to indicate scrapeable targets. Additional annotations define the metrics endpoint:

• prometheus.io/port: Specifies the container port exposing /metrics
• prometheus.io/path: Defines custom metrics paths when /metrics is not used

Relabeling Configuration

Within the Prometheus configuration file, relabel_configs transform Kubernetes metadata into scrape parameters. The configuration references __meta_kubernetes_pod_annotation_* labels populated by the service discovery mechanism. For example, __meta_kubernetes_pod_annotation_prometheus_io_port replaces the default scrape port, while namespace and pod name labels enable metric attribution to specific teams or applications.

Dynamic Configuration Reloading

Prometheus applies scrape configuration changes without requiring a process restart. When the discovered set of targets changes due to pod lifecycle events, Prometheus updates its internal target list on the fly. This hot-reloading capability guarantees that newly created pods begin emitting metrics immediately while terminated pods stop being scraped, preventing failed scrape attempts against dead endpoints.

High Availability and Scalability

Each Prometheus instance independently discovers targets from the Kubernetes API, allowing multiple replicas to run in high-availability mode. As documented in README.md lines 925-967, each replica maintains an identical dynamic view of the cluster, ensuring that metric collection continues uninterrupted even during individual Prometheus server failures or rolling updates.

This architecture scales horizontally: additional Prometheus instances can shard scrape targets by namespace or job while each continues to discover its assigned subset through the same API mechanisms.

Practical Implementation

The following configuration demonstrates a complete Prometheus Kubernetes service discovery setup.

Configure the Prometheus server to discover pods based on annotations:

# prometheus.yaml – Prometheus server configuration

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # -------------------------------------------------

  # Kubernetes service discovery

  # -------------------------------------------------

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only pods that expose Prometheus metrics

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

      # Use the pod's port annotation as the target port

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: $1

      # Set the metrics path (default /metrics)

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
        replacement: $1

Annotate application pods to enable scraping:

# Example pod manifest that registers itself for scraping

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
  annotations:
    prometheus.io/scrape: "true"        # <-- enable scraping

    prometheus.io/port: "8080"          # <-- port exposing /metrics

    prometheus.io/path: "/metrics"      # <-- optional custom path

spec:
  containers:
  - name: my-app
    image: my-app:latest
    ports:
    - containerPort: 8080

Verify discovery status using the Prometheus API:

# Verify that Prometheus has discovered the pod

curl http://<prometheus-server>:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="kubernetes-pods") | .discoveredLabels'

As shown in topics/argo/README.md lines 464-465, ecosystem tools like Argo CD reference these Prometheus endpoints to trigger deployments based on metric thresholds, demonstrating the integration between service discovery and the broader DevOps toolchain.

Summary

• Prometheus uses the Kubernetes API as the source of truth to discover ephemeral pods, services, and endpoints without manual configuration updates.
• Native service discovery eliminates external registry dependencies by leveraging the Kubernetes control plane's built-in metadata, as documented in topics/containers/README.md.
• Label-based selection via annotations like prometheus.io/scrape enables granular, opt-in monitoring that scales with team-specific namespaces and applications.
• Dynamic reloading ensures immediate scrape target updates when pods scale or restart, while independent discovery across HA replicas guarantees continuous monitoring coverage.

Frequently Asked Questions

How does Prometheus handle pods that scale up or down rapidly?

Prometheus watches the Kubernetes API server for pod lifecycle events through its service discovery mechanism. When the Deployment or ReplicaSet controller creates new pods, Prometheus immediately adds them to the active scrape target list. When pods terminate, Prometheus removes them within seconds, preventing failed scrape attempts against non-existent endpoints.

What is the difference between role: pod and role: service in kubernetes_sd_configs?

The role: pod configuration discovers individual pod IP addresses directly, providing granular per-instance metrics. The role: service configuration discovers service endpoints (the IP:port combinations backing a Kubernetes Service), which aggregate metrics across all ready pods behind a service. Use role: pod for detailed instance-level monitoring and role: service for cluster-level aggregated metrics.

Do I need to restart Prometheus when adding new namespaces to monitor?

No. Prometheus applies configuration changes dynamically without requiring a restart. When you update the namespace selector in kubernetes_sd_configs or add new scrape jobs, Prometheus hot-reloads the configuration and immediately begins discovering targets in the newly specified namespaces through the Kubernetes API.

Can multiple Prometheus instances discover the same targets without conflicts?

Yes. According to the repository documentation in README.md lines 925-967, each Prometheus instance independently queries the Kubernetes API and maintains its own target list. Multiple replicas can scrape the same targets for high availability, or they can shard targets by namespace or job label to distribute the scrape load while each instance continues to discover its assigned subset dynamically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}