Kubernetes operations

Liveness, readiness, and startup probes: what each does

Three probes, three different failure actions, and mixing them up is the classic mistake.

• Liveness answers "is the process wedged?" On failure the kubelet kills and restarts the container. Use it only for a hung or deadlocked process that will not recover on its own.
• Readiness answers "can this pod serve right now?" On failure the pod is marked NotReady and pulled out of the Service's endpoints, so it stops receiving traffic. The container is not killed, and traffic resumes once the probe passes again.
• Startup protects a slow-booting app. While it runs, liveness and readiness are disabled; once it passes, they take over. Size it with failureThreshold x periodSeconds (30 x 10s allows 300s to boot).

Wrong "Readiness restarts the pod." Readiness never kills anything; it only adds or removes the pod from the Service endpoints. Liveness is the one that restarts.

Defaults worth memorizing: periodSeconds 10, timeoutSeconds 1, failureThreshold 3, successThreshold 1, initialDelaySeconds 0.

Requests vs limits, and why a CPU limit can throttle idle CPU

A request is what the scheduler reserves, the guaranteed floor that drives where the pod lands. A limit is the runtime ceiling. They behave differently by resource:

• CPU is compressible. A CPU limit is a per-100ms CFS quota, so a container is throttled (slowed) when it bursts past that quota, never killed. It can be throttled even when the node has spare CPU.
• Memory is incompressible. Cross the memory limit and the kernel OOMKills the container (exit 137), which then restarts.

Wrong "Exceeding the CPU limit kills the pod." CPU over its limit only throttles. It is memory over its limit that triggers an OOMKill.

Config vs Secret, and what base64 does not buy you

Use a ConfigMap for non-secret config, a Secret for anything confidential. A Secret's values are base64-encoded, which is encoding, not encryption: base64 -d reverses it in one command. By default Secrets sit unencrypted in etcd, so anyone with API or etcd access can read them. For real protection you enable encryption at rest and lock down RBAC.

One update gotcha for both objects: mounted as a volume, a ConfigMap or Secret updates on its own eventually, but values consumed as environment variables need a pod restart to change.

Rolling updates and rollback

Changing a Deployment's pod template (a new image or env) triggers a rolling update: a new ReplicaSet scales up while the old one scales down. Scaling replicas alone does not trigger a rollout. Two knobs bound the disruption, both defaulting to 25%: maxSurge (pods allowed above desired) and maxUnavailable (pods allowed missing below desired). The old ReplicaSet stays scaled to zero, so kubectl rollout undo deployment/api reverts the template fast. That rollback reverts only the template, never data or a schema migration, and a rolling update assumes old and new versions can run side by side; when they cannot, use Recreate or blue/green.

How a liveness probe that checks a dependency cascades

The dangerous liveness probe reaches past the local process. If /health returns 500 whenever the database is unreachable, a database outage makes every replica fail liveness at once. The kubelet kills and restarts them together, a dependency blip becomes a fleet-wide restart storm, and pods slide into CrashLoopBackOff whose exponential backoff outlasts the outage. The official docs call this out: incorrect liveness probes cause cascading failures.

Liveness must answer one narrow question, whether this process itself is wedged or deadlocked. Absorb transient dependency errors in the app with timeouts, retries, and a circuit breaker. Do not reflexively move the same check into readiness: readiness stops the restarts, but a shared dependency there flips every pod to NotReady at once, so the Service drains to zero endpoints and you have traded a restart storm for a total blackout. If the process crashes on an unrecoverable error, restartPolicy already restarts it, so a local-only liveness probe is sometimes optional.

Why a CPU limit throttles a pod with idle CPU

A request sets the CFS cpu.shares (relative weight under contention) and tells the scheduler how much to reserve. A CPU limit sets a CFS quota: a hard cap on CPU-time per roughly 100ms period. A bursty container that wants a lot of CPU for a short slice hits that per-period ceiling and is throttled, adding latency, while the node sits idle.

Wrong "Set the CPU limit equal to the request so the pod gets exactly what it reserved." Equal request and limit caps the container at its reserved slice every period, so a bursty, latency-sensitive service is throttled even with spare CPU. Right set the request (it drives scheduling and the share weight) and be cautious about a CPU limit on latency-sensitive work. Keep a memory limit regardless, because memory is incompressible and a runaway pod can OOM its neighbors. QoS falls out of the numbers: equal requests and limits on every resource is Guaranteed (evicted last); no requests or limits is BestEffort (evicted first under memory pressure).

The traffic path: Service, EndpointSlices, readiness

A Service is a stable virtual IP and DNS name over a changing pod set. Its selector is watched by a controller that writes matching pod IPs into EndpointSlices, and only Ready pods are listed, which is the mechanism a readiness probe drives. kube-proxy programs the node dataplane to DNAT the ClusterIP to a backend pod IP. An Ingress adds L7 routing by host and path plus TLS termination in front of Services, but the resource does nothing on its own: it needs an Ingress controller to satisfy it. End to end: client to Ingress controller (terminates TLS, routes by host or path) to Service to EndpointSlices to a Ready pod.

When Kubernetes or HPA is the wrong tool

An HPA scales replicas from a metric: desiredReplicas = ceil(current x currentMetric / targetMetric), with a 10% tolerance band and a 300s scale-down stabilization window so it does not flap. For CPU or memory utilization targets the pods must set the matching resource request, because utilization is a percentage of the request; with no request set, the HPA does nothing for that metric.

It is the wrong tool in several cases. It cannot scale a DaemonSet, which runs one pod per node by design. It fights a fixed replica count or a VPA aimed at the same metric. It cannot help when the bottleneck is a non-scaled dependency: adding pods to hammer a maxed-out database only moves the queue. Its loop plus metric lag plus pod startup means it reacts in tens of seconds, so a sub-15s spike needs pre-provisioning. For a single small service run by a small team, the operational cost of the control plane, networking, and on-call for the cluster can outweigh what Kubernetes buys you, and a managed PaaS or a plain VM is sometimes cheaper.