Safely Drain a Node in Kubernetes

How to safely drain a node, optionally respecting the PodDisruptionBudget you have defined.

You have a cluster with a few nodes and pods serving applications. What happens when one of these nodes go down. The pods on them are not accessible. If the node was down for more than 5 minutes, then the pods are terminated from that node. Also, If the pods were part of a ReplicaSet then they are recreated on other nodes. The time it waits for a pod to come back online is known as the pod eviction timeout and is set on the controller manager with a default value of five minutes. So whenever a node goes offline, the master node waits for upto 5 minutes before considering the node dead.

$ kubectl get nodes
$ kubectl drain <node name>

you can drain the node of all the workloads so that the workloads are moved to other nodes in the cluster. when you drain the node the pods are gracefully terminated from the node that they are on and recreated on another. The node is also marked as unschedulable. Meaning no pods can be scheduled on the node until you specifically remove the restriction.

$ kubectl uncordon <node name>

you need to uncordon it, so that pods can be scheduled on it again.

Leave a Reply

Your email address will not be published.