# Understanding the Scale-Down Behavior of GKE DaemonSets

> Learn how GKE DaemonSets scale down. Understand that DaemonSet pods are ignored by the Cluster Autoscaler, allowing nodes to be removed even when only DaemonSets are present.

- Repository: [Google/skills](https://github.com/google/skills)
- Tags: deep-dive
- Published: 2026-08-09

---

**In Google Kubernetes Engine (GKE), the Cluster Autoscaler explicitly ignores DaemonSet pods when evaluating nodes for scale-down, meaning DaemonSets do not block node removal even if they are the only workloads running on a node.**

The scale-down behavior of GKE DaemonSets often confuses operators who expect these system-critical pods to pin nodes indefinitely. According to the `google/skills` repository documentation, the GKE Cluster Autoscaler (CA) treats DaemonSets as non-blocking by design, allowing infrastructure costs to optimize automatically even when every node runs daemon workloads.

## How GKE Cluster Autoscaler Handles DaemonSets During Scale-Down

The GKE Cluster Autoscaler does not consider DaemonSets as blockers for node scale-down. When evaluating a candidate node for removal, the autoscaler checks for pods that must remain on the node—such as those with `NoExecute` eviction policies, Pod Disruption Budgets (PDBs) with zero disruptions allowed, or pods using local storage—while explicitly filtering out DaemonSet pods.

This behavior is documented in [`skills/cloud/gke-cluster-autoscaler/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-cluster-autoscaler/SKILL.md), where the authors label the belief that DaemonSets prevent scale-down as the **"DAEMONSET MYTH"** (line 16). The code logic assumes DaemonSets run on every node by definition, so their presence provides no signal about whether a specific node is excess capacity.

## Common Misconceptions vs. Real Scale-Down Blockers

### The DaemonSet Myth

Many operators assume that because DaemonSet pods run on every node, they act as anchors preventing the Cluster Autoscaler from reclaiming resources. This is incorrect. As implemented in the GKE autoscaler logic, DaemonSet pods are ignored entirely during the scale-down simulation because they will simply be recreated on whatever nodes remain after consolidation.

### Actual Blockers That Prevent Node Removal

When a node refuses to scale down despite appearing empty, investigate these actual blockers instead:

- **Bare pods** without controller references
- Pods annotated with `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"`
- Pods using **local storage** (EmptyDir volumes)
- **System pods** in `kube-system` that are **not** managed by a DaemonSet
- PDBs configured with `maxUnavailable: 0` or `minAvailable: 100%`

## Managing System Pods to Enable Scale-Down

While DaemonSets themselves are ignored, non-DaemonSet pods residing in `kube-system` can still block scale-down. The `google/skills` repository recommends segregating these system pods onto dedicated infrastructure to prevent them from pinning expensive compute nodes.

According to [`skills/cloud/gke-compute-classes/references/compute-class-lifecycle.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-compute-classes/references/compute-class-lifecycle.md) (lines 13-15), you should label the `kube-system` namespace so that non-DaemonSet system pods are scheduled onto a cheap "system" ComputeClass. This pattern ensures that only user workloads block scale-down on premium nodes.

## Practical Commands for Node Drain and Troubleshooting

When manually draining nodes or troubleshooting scale-down failures, use these commands derived from the repository's operational scripts.

To drain a node while explicitly ignoring DaemonSet pods—the standard preparation for node removal:

```bash
kubectl drain <NODE_NAME> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

```

To label the `kube-system` namespace for ComputeClass segregation, preventing non-DaemonSet system pods from blocking scale-down on expensive pools:

```bash
kubectl label ns kube-system \
  cloud.google.com/default-compute-class-non-daemonset=system-pool

```

To identify all active scale-down blockers in your cluster, including verification that DaemonSets are correctly ignored, run the diagnostic script provided in the repository:

```bash
./assets/find-scale-down-blockers.sh -n kube-system

```

This script, located at [`skills/cloud/gke-cluster-autoscaler/assets/find-scale-down-blockers.sh`](https://github.com/google/skills/blob/main/skills/cloud/gke-cluster-autoscaler/assets/find-scale-down-blockers.sh), scans for the actual conditions preventing node reclamation.

## Summary

- **DaemonSets are ignored**: The GKE Cluster Autoscaler does not treat DaemonSet pods as blockers for node scale-down, as documented in [`skills/cloud/gke-cluster-autoscaler/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-cluster-autoscaler/SKILL.md).
- **Check real blockers**: When nodes fail to scale down, investigate bare pods, local storage usage, restrictive PDBs, and non-DaemonSet system pods instead.
- **Segregate system workloads**: Use namespace labeling and ComputeClasses to keep non-DaemonSet `kube-system` pods off expensive node pools, as described in [`skills/cloud/gke-compute-classes/references/compute-class-lifecycle.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-compute-classes/references/compute-class-lifecycle.md).
- **Use --ignore-daemonsets**: When manually draining nodes, always include this flag to allow clean node removal.

## Frequently Asked Questions

### Do DaemonSets prevent GKE Cluster Autoscaler from removing nodes?

No. According to the `google/skills` repository, the GKE Cluster Autoscaler explicitly ignores DaemonSet pods during scale-down evaluations. The documentation refers to this misconception as the "DAEMONSET MYTH" in [`skills/cloud/gke-cluster-autoscaler/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-cluster-autoscaler/SKILL.md).

### Why does my node with only DaemonSet pods fail to scale down?

If a node running only DaemonSet pods refuses to scale down, another blocker is present. Check for pods using local storage, bare pods without controllers, restrictive Pod Disruption Budgets, or non-DaemonSet pods in the `kube-system` namespace. Run the [`find-scale-down-blockers.sh`](https://github.com/google/skills/blob/main/find-scale-down-blockers.sh) script to identify the specific obstruction.

### How can I prevent system pods from blocking scale-down?

Label the `kube-system` namespace with `cloud.google.com/default-compute-class-non-daemonset=system-pool` to schedule non-DaemonSet system pods onto a dedicated, inexpensive ComputeClass. This segregation pattern, documented in [`skills/cloud/gke-compute-classes/references/compute-class-lifecycle.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-compute-classes/references/compute-class-lifecycle.md), ensures these pods do not pin expensive nodes.

### What is the --ignore-daemonsets flag used for?

The `--ignore-daemonsets` flag in `kubectl drain` instructs Kubernetes to proceed with node evacuation even if DaemonSet pods are present. Since DaemonSets are designed to run on every node, they are recreated automatically on remaining nodes after the drain completes, making this flag essential for manual node maintenance and scale-down preparation.