Questions and answers about node group autoscaling in Managed Service for Kubernetes
Why are there N nodes in my cluster now, but it's not getting downsized?
Autoscaling doesn't stop nodes with pods that can't be evicted. Scaling barriers:
- Pods whose eviction is limited with PodDisruptionBudget.
- Pods in the
kube-system
namespace:- That haven't been created under the DaemonSet
controller. - That don't have
PodDisruptionBudget
or whose eviction is limited byPodDisruptionBudget
.
- That haven't been created under the DaemonSet
- Pods that haven't been created under a replication controller (ReplicaSet
, Deployment , or StatefulSet ). - Pods with a
local storage
. - Pods that can't be evicted anywhere due to restrictions. For example, due to lack of resources or lack of nodes matching the affinity or anti-affinity
selectors. - Pods with an annotation that disables eviction:
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
.
Note
Pods kube-system
, pods with local-storage
, and pods without a replication controller can be evicted. To do this, set the annotation "safe-to-evict": "true"
:
kubectl annotate pod <pod name> cluster-autoscaler.kubernetes.io/safe-to-evict=true
Other possible causes:
-
The node group has already reached its minimum size.
-
The node is idle for less than 10 minutes.
-
During the last 10 minutes, the node group has been scaled up.
-
During the last 3 minutes, there was an unsuccessful attempt to scale down the node group.
-
There was an unsuccessful attempt to stop a certain node. In this case, the next attempt occurs in 5 minutes.
-
The node has an annotation that prohibits stopping it on scale-down:
"cluster-autoscaler.kubernetes.io/scale-down-disabled": "true"
. You can add or remove an annotation usingkubectl
.Check for annotation on the node:
kubectl describe node <node name> | grep scale-down-disabled
Result:
Annotations: cluster-autoscaler.kubernetes.io/scale-down-disabled: true
Set the annotation:
kubectl annotate node <node name> cluster-autoscaler.kubernetes.io/scale-down-disabled=true
Remove the annotation by running the
kubectl
command with-
:kubectl annotate node <node name> cluster-autoscaler.kubernetes.io/scale-down-disabled-
Why has the pod been deleted, but the node group fails to downsize?
If the node is underloaded, it's removed in 10 minutes.
Why isn't autoscaling performed even when the number of nodes gets less than the minimum or greater than the maximum?
Autoscaling won't violate preset limits, but Managed Service for Kubernetes isn't explicitly controlling the limits. Scaling up will only trigger if there are pods in the unschedulable
status.
To get answers to other questions about autoscaling, see the documentation Kubernetes