Node Maintenance Takes Down Services That Have Enough Replicas
Key takeaway: Replica count describes capacity, not availability. Distribution across failure domains and a disruption budget are what keep a service running during voluntary maintenance.
The Gap Between Replicas and Availability
A deployment specifies three replicas. The scheduler, absent any constraint, may place all three on the same node because that node had capacity.
A cluster upgrade drains that node. All three pods are evicted at once. The service is entirely unavailable until replacements schedule elsewhere and pass readiness checks — thirty seconds at best, several minutes if images need pulling.
Nothing was misconfigured in an obvious sense. The replica count was reasonable and the drain was a routine operation.
The Two Controls That Matter
Topology spread constraints tell the scheduler to distribute pods across a domain — nodes, availability zones, or any node label. Setting a maximum skew of one across zones means an even distribution, so losing a zone loses a predictable fraction rather than everything.
Pod disruption budgets constrain voluntary eviction. A budget stating minAvailable: 2 means the eviction API refuses to remove a third pod while only two remain ready. A drain then proceeds pod by pod, waiting for replacements to become ready between each.
| Mechanism | Protects against |
|---|---|
| Replica count | Insufficient capacity |
| Topology spread | Correlated failure in one domain |
| Pod disruption budget | Too-fast voluntary eviction |
| Anti-affinity | Co-location on one node |
All four are needed, and each addresses something the others do not. A budget without spread constraints slows down a drain that still ends up removing everything.
What a Budget Does Not Do
Disruption budgets only apply to voluntary disruptions — drains, upgrades, autoscaler scale-down. They do not apply when a node fails, when the kernel kills a process, or when a pod is evicted for exceeding its memory limit.
That distinction matters because it means a budget is not a reliability guarantee. It is a guarantee that your own maintenance will not cause an outage, which is a narrower and still valuable property.
The other common surprise is a budget that blocks maintenance entirely. Setting minAvailable equal to the replica count means no pod can ever be evicted voluntarily, so a cluster upgrade stalls indefinitely waiting for a budget that can never be satisfied. Cluster operators then either override it or the upgrade fails, and neither outcome was intended.
Getting the Numbers Right
Express the budget as maxUnavailable: 1 rather than minAvailable: N where possible. The former scales correctly when replica count changes; the latter becomes wrong after a scale-up or scale-down and nobody remembers to adjust it.
Confirm that the replica count exceeds the minimum available by at least one, otherwise the budget is unsatisfiable by construction.
Single-replica workloads cannot have a meaningful budget. If a service genuinely cannot tolerate any disruption, it needs more than one replica — the budget cannot manufacture availability that the deployment does not have.
Verifying It Works
The test is straightforward and rarely performed: drain a node in a non-production cluster and observe whether the service stays available. That exercise reveals unsatisfiable budgets, missing spread constraints and slow readiness probes together.
Readiness probe timing matters here more than elsewhere. A drain waits for replacements to be ready, so a probe with a long initial delay makes every node drain slow, and one that reports ready before the application can serve traffic makes the budget meaningless.
The Bottom Line
Combine topology spread constraints with a maxUnavailable: 1 disruption budget and a readiness probe that reflects actual serving capability. Then drain a node deliberately to confirm the arrangement works, because that is the only way to discover an unsatisfiable budget before it blocks an upgrade.
