↓ Skip to main content
Alhassan Ibrahim
  1. Posts/

Designing Resilient Kubernetes Deployments

·2 mins· loading · loading · ·
Kubernetes Devops Reliability
Alhassan Ibrahim
Author
Alhassan Ibrahim
Passionate about software reliability, availability, resilience, security, and automation.
Table of Contents

Most Kubernetes outages I’ve debugged in production weren’t caused by Kubernetes itself — they were caused by a deployment manifest that never told Kubernetes what “healthy” and “safe to disrupt” actually meant. A handful of fields close most of that gap.

PodDisruptionBudgets
#

Node drains, cluster autoscaler scale-downs, and rolling node upgrades all evict pods. Without a PodDisruptionBudget, Kubernetes is free to evict every replica of a Deployment at once during a voluntary disruption.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api

minAvailable: 2 tells the eviction API to keep at least two pods running no matter how aggressively the cluster wants to drain nodes.

Readiness and liveness probes
#

A liveness probe restarts a stuck container; a readiness probe removes a not-yet-ready pod from the Service endpoints list. Conflating the two is a common cause of cascading failures — a slow dependency during startup will get the pod killed and restarted in a loop if the liveness probe is too aggressive, instead of just being held out of traffic by the readiness probe.

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 15
  failureThreshold: 3

Topology spread constraints
#

Three replicas scheduled onto the same node or availability zone provide the illusion of redundancy without the substance of it.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

Resource requests and limits
#

Requests drive scheduling and are what the PodDisruptionBudget and topology spread constraints reason about; limits (especially CPU limits) can throttle a healthy pod into failing its own liveness probe under load. Set requests close to real steady-state usage, and be deliberate about whether a CPU limit is even worth the throttling risk.

None of these four things are exotic. What matters is treating them as a checklist for every workload that has an SLA attached to it, not just the ones that have already caused an incident.

Related