Prometheus Metric Collection Strategies: Comparing Pushgateway, Node Exporter, and cAdvisor

Prometheus Pushgateway uses a push-based model for short-lived batch jobs, while Node Exporter and cAdvisor employ pull-based scraping for host-level and container-level metrics respectively.

In modern cloud-native observability stacks, selecting the appropriate metric collection strategy is critical for comprehensive system visibility. The bregman-arie/devops-exercises repository highlights how these three Prometheus ecosystem components serve fundamentally different monitoring requirements across diverse infrastructure layers. This article examines the architectural patterns, deployment configurations, and optimal use cases for each collector based on their underlying implementations.

Core Collection Models

The fundamental distinction between these components lies in their data transmission direction, lifecycle management, and target resource types.

Prometheus Pushgateway: Push-Based Intermediary

Pushgateway acts as a bridge for ephemeral or batch jobs that cannot wait for Prometheus to scrape them. Instead of exposing an endpoint for polling, applications actively push metrics via HTTP POST requests to the gateway.

Jobs send metrics to the gateway's /metrics/job/<job_name> endpoint using the Prometheus exposition format. The gateway stores only the latest value for each metric series until overwritten or cleared. Prometheus then scrapes the gateway's /metrics endpoint on its standard interval (default 15 seconds), treating the gateway as a persistent intermediary.

This pattern is essential for CI pipelines, backup scripts, and cron jobs that terminate before a scrape cycle completes. However, the gateway retains only the most recent push; it is not a time-series database and does not aggregate historical data between pushes.

Node Exporter: Native Pull-Based Host Monitoring

Node Exporter represents the classic pull-based Prometheus model, running as a stateless daemon on every target node. It reads kernel statistics from the host's /proc and /sys filesystems on each scrape request, exposing pre-defined metrics like node_cpu_seconds_total and node_memory_MemAvailable_bytes.

Deployed as a binary or Docker container (prom/node-exporter), it exposes data on port 9100 without requiring write access from external systems. Prometheus initiates all connections, making firewall rules and security boundaries straightforward to manage. Each scrape yields a fresh snapshot of host resources.

cAdvisor: Container-Aware Pull Integration

cAdvisor (Container Advisor) provides granular container-level resource metrics within Kubernetes environments. Unlike standalone exporters, cAdvisor runs embedded within the kubelet on each node, continuously monitoring cgroups and kernel metrics for all containers on that host.

As noted in [topics/kubernetes/README.md](https://github.com/bregman-arie/devops-exercises/blob/master/topics/kubernetes/README.md) within the bregman-arie/devops-exercises repository, the metrics-server leverages the cAdvisor component of the kubelet to collect node-level container statistics. Prometheus scrapes the kubelet's read-only port (typically 10255) or secure port to retrieve per-container CPU, memory, filesystem, and network usage metrics like container_cpu_user_seconds_total.

Deployment and Configuration Patterns

Each component requires distinct deployment strategies aligned with its collection model and target environment.

Configuring Pushgateway for Batch Workloads

Run the gateway as a persistent HTTP service:

docker run -d -p 9091:9091 prom/pushgateway

Client jobs push metrics via HTTP POST before terminating:

cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/backup_job/instance/batch-01

# TYPE job_duration_seconds gauge

job_duration_seconds 45.2

# TYPE job_records_processed counter

job_records_processed 892
EOF

Configure Prometheus to scrape the gateway:

scrape_configs:
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['pushgateway-host:9091']

Deploying Node Exporter on Linux Hosts

Deploy with full host namespace access to read system metrics:

docker run -d --net="host" --pid="host" \
  -v "/:/host:ro,rslave" \
  -v "/sys:/sys:ro" \
  -v "/proc:/proc:ro" \
  prom/node-exporter \
  --path.rootfs=/host

Prometheus configuration targets each node's exporter individually:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node1:9100', 'node2:9100', 'node3:9100']

Scraping cAdvisor Metrics in Kubernetes

cAdvisor requires no separate binary installation. Enable scraping through the kubelet by configuring Prometheus with Kubernetes service discovery:

scrape_configs:
  - job_name: 'kubelet-cadvisor'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*):10250
        replacement: $1:10255
        target_label: __address__

For RBAC-enabled clusters, ensure the Prometheus service account has permission to access node metrics:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-cadvisor
rules:
- apiGroups: [""]
  resources: ["nodes/metrics"]
  verbs: ["get"]

Strategic Selection Criteria

Choose your collector based on workload characteristics and infrastructure layer:

• Pushgateway: Select when monitoring short-lived processes that exit before scrape intervals complete, or when aggregating results from serverless functions that cannot expose persistent endpoints.
• Node Exporter: Deploy for infrastructure health monitoring, capacity planning, and host-level resource exhaustion alerts across bare-metal or virtual machine fleets.
• cAdvisor: Implement for Kubernetes-native observability, per-container autoscaling metrics, and pod resource quota enforcement within container orchestration platforms.

Summary

• Pushgateway bridges push-based batch jobs to Prometheus's pull model, storing only the latest metric values until overwritten by subsequent pushes.
• Node Exporter provides stateless, pull-based access to host kernel statistics via /proc and /sys filesystems, exposing data at :9100/metrics.
• cAdvisor exposes container-level metrics through the Kubernetes kubelet's embedded instance, as referenced in the bregman-arie/devops-exercises repository's Kubernetes documentation.
• Pushgateway requires write access from client jobs and maintains state between scrapes, while Node Exporter and cAdvisor expose read-only, stateless endpoints scraped by Prometheus.
• Node Exporter runs as a standalone binary per node, whereas cAdvisor is embedded within the kubelet without requiring separate deployment.

Frequently Asked Questions

When should I use Pushgateway instead of a direct exporter?

Use Pushgateway for short-lived batch processes that terminate before Prometheus can scrape them, such as CI/CD pipelines or nightly backup scripts. Direct exporters suit long-running services with persistent HTTP endpoints. Avoid Pushgateway for regular service monitoring, as it creates single points of failure and complicates metric lifecycle management when jobs push conflicting labels.

Does cAdvisor run as a separate container in Kubernetes?

No, cAdvisor runs embedded within the kubelet on each Kubernetes node. It is not deployed as a standalone container or DaemonSet. Prometheus scrapes container metrics through the kubelet's API endpoints, typically port 10255 (read-only) or the secure kubelet port with authentication, as implemented in the metrics-server architecture described in the bregman-arie/devops-exercises repository.

Can Node Exporter monitor container resource usage?

No, Node Exporter exposes host-level metrics only, reading from the node's /proc and /sys filesystems. It cannot see per-container cgroup statistics or individual pod resource consumption. For container metrics, use cAdvisor via the kubelet, or query the kubelet's native resource metrics endpoint.

How does metric retention differ between these components?

Pushgateway retains only the latest pushed value for each metric series, making it unsuitable for historical trend analysis. Node Exporter and cAdvisor are stateless collectors that generate fresh snapshots on each scrape request; Prometheus itself stores the time-series history in its TSDB. Never treat Pushgateway as a long-term storage solution or metric aggregation buffer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}