Kubernetes networking

What a Service gives you

A pod's IP is disposable. Pods die, roll, and reschedule, and each new pod gets a fresh IP, so nothing should ever hardcode a pod address. A Service is the stable thing in front of a changing set of pods: it has a fixed virtual IP and a DNS name for as long as the Service exists, and it forwards to whichever pods currently match its selector. The set of backing pod IPs lives in EndpointSlices that a controller keeps in sync as pods come and go, and only Ready pods receive traffic.

Four types cover almost everything, and they stack:

• ClusterIP (the default): a virtual IP reachable only inside the cluster. This is what one service uses to call another.
• NodePort: everything ClusterIP does, plus the Service is opened on a fixed port (default range 30000 to 32767) on every node's IP.
• LoadBalancer: everything NodePort does, plus it asks the cloud provider for an external load balancer with a public address pointing at those node ports.
• Headless (clusterIP: None): no virtual IP at all. DNS returns the pod IPs directly, so the caller talks to individual pods. Used for StatefulSets and anything that needs to address specific replicas.

Wrong "I'll use NodePort (or LoadBalancer) so my API can reach the payments service." Those expose the service outside the cluster and, for LoadBalancer, cost money per service. Internal calls want ClusterIP. Reach for NodePort or LoadBalancer only when traffic from outside the cluster has to arrive.

Finding a service by name

Every ClusterIP Service gets a DNS record. The full name is service.namespace.svc.cluster.local. Inside a pod you rarely type all of that, because kubelet writes a search list into the pod's /etc/resolv.conf:

search payments.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10          # the cluster DNS service (CoreDNS)
options ndots:5

So a pod in the payments namespace can say db and it expands to db.payments.svc.cluster.local. That shortcut is the thing that bites you across namespaces.

Wrong "db worked in one namespace so it works everywhere." The bare name only resolves through the search list, which starts with the pod's own namespace. From another namespace, db first tries db.<that-namespace>.svc.cluster.local and misses. Cross-namespace calls work fine, you just qualify the name: db.payments, or the full db.payments.svc.cluster.local. Services are reachable cluster-wide; the name is what is namespace-scoped.

One entry point for HTTP: Ingress

A LoadBalancer Service per public app means one cloud load balancer per app, which gets expensive fast. For HTTP and HTTPS, the usual pattern is a single Ingress: it routes by hostname and URL path to different backend Services, and it can terminate TLS, all behind one load balancer.

rules:
  - host: shop.example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend: { service: { name: storefront, port: { number: 80 } } }
  - host: api.example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend: { service: { name: api, port: { number: 80 } } }

Two things people miss. An Ingress does nothing on its own: you must run an Ingress controller (ingress-nginx, a cloud controller, Envoy-based, and so on) to satisfy it, otherwise the object just sits there. And Ingress is HTTP/HTTPS only, so a raw TCP or UDP service still needs NodePort or LoadBalancer. (The newer Gateway API replaces Ingress for teams that want more than host/path routing; the senior notes cover the split.)

Why every pod has its own IP

Kubernetes gives every pod a unique, cluster-wide IP, and every pod can reach every other pod on any node without NAT. That is the whole pod network model, and it is why you address a pod by its IP directly and why Services can just forward to pod IPs. The IPs are assigned by a CNI (Container Network Interface) plugin such as Calico, Cilium, or a cloud VPC plugin; Kubernetes itself ships no network implementation, so the plugin is what makes pod-to-pod traffic flow.

Containers inside one pod are the exception to "one IP each": they share the pod's network namespace, so they reach each other over localhost and cannot both bind the same port.

Pods are open by default

A fresh pod accepts connections from anything that can route to it, in and out. There is no implicit firewall between namespaces. You restrict traffic with a NetworkPolicy, and the key rule is that a policy only affects the pods it selects: a pod stays wide open until some policy selects it, and then only the sources that policy lists can reach it. So "we added a NetworkPolicy" does not mean "the namespace is locked down." It locks down exactly the pods the policy's selector matches, in exactly the direction (ingress or egress) it names. The senior notes cover the default-deny pattern and the cross-namespace selector trap.

kube-proxy is not in the packet's path

A ClusterIP is a virtual IP with no process listening on it. When a pod connects to 10.96.0.10:80, no proxy accepts that connection and re-dials a backend. Instead kube-proxy runs on every node, watches the API server for Services and EndpointSlices, and programs the node's kernel dataplane so that packets destined for a Service IP are rewritten (destination NAT) to one of the Ready backend pod IPs. The pick is effectively random per new connection, which is where the "load balancing" comes from, and the client's source IP is preserved through the rewrite.

The mode is which kernel machinery it programs:

• iptables: the long-standing default. Correct and everywhere, but the ruleset grows roughly linearly with the number of Services and endpoints, so very large clusters see slow rule sync and per-packet cost.
• nftables: went GA in Kubernetes 1.33. Uses nftables maps for close to O(1) lookup regardless of Service count, so it scales far better on modern kernels (5.13+). It is the recommended mode for new large clusters, though iptables is still the shipped default for compatibility.
• ipvs: an older kernel load-balancer mode, deprecated as of v1.35.

Wrong "kube-proxy proxies my Service traffic, so it's a bottleneck / a single hop." In these modes there is no userspace hop for data. kube-proxy is a control-plane agent that writes forwarding rules; the packets are moved by the kernel. What kube-proxy can be slow at is reprogramming rules when endpoints churn, not forwarding.

Session affinity is available (sessionAffinity: ClientIP) if you need a client pinned to one backend, but the default is per-connection spraying, so do not assume stickiness.

The ndots:5 tax on external names

That options ndots:5 line in every pod's resolv.conf is the single most common Kubernetes DNS surprise. The rule: if the name you look up has fewer than 5 dots, the resolver treats it as relative and tries it against each search domain first, only falling back to the name as absolute after those miss.

Count the dots on a name like api.stripe.com: two. Under ndots:5 it is "relative," so a pod in namespace web issues, in order:

api.stripe.com.web.svc.cluster.local   -> NXDOMAIN
api.stripe.com.svc.cluster.local       -> NXDOMAIN
api.stripe.com.cluster.local           -> NXDOMAIN
api.stripe.com                         -> the real answer

Each failing candidate is usually queried twice (A and AAAA), so one external call can cost six wasted lookups before the good one. That is the burst of failed DNS you see under load, and it shows up as latency and as pressure on CoreDNS.

Wrong "External DNS is slow, so scale CoreDNS." Adding CoreDNS replicas treats the symptom. The fix is to stop generating the wasted queries: make external names fully qualified with a trailing dot (api.stripe.com.) so the search list is skipped, or set a lower ndots for that workload via dnsConfig, or deploy NodeLocal DNSCache so the NXDOMAINs are answered on-node. ndots:5 is not a mistake, it exists so that short in-cluster names (db, db.web) resolve through the search list; the cost lands on external names.

Ingress is frozen; Gateway API is the successor

Ingress solved one thing: L7 host and path routing to Services behind a shared load balancer, with TLS termination. It has real limits. It is HTTP/HTTPS only, most non-trivial behavior (rewrites, timeouts, canary weighting) lives in controller-specific annotations, and one Ingress mixes concerns that different teams own. The Ingress API is now feature-frozen: generally available and not going away, but receiving no new features.

The Gateway API is the GA replacement (GatewayClass, Gateway, and HTTPRoute are all v1). Two properties separate it from Ingress. It is role-oriented: a Gateway (the listener and its address, owned by cluster operators) is separate from the HTTPRoutes (the routing rules, owned by app teams), so a platform team can hand out routing without handing out the whole load balancer. And expressive routing (header matching, traffic splitting, cross-namespace routing gated by allowedRoutes) is first-class in the spec instead of being annotation soup. Migration is a one-time conversion; Gateway API does not include an Ingress kind.

For non-HTTP traffic, neither is the tool: a raw TCP/UDP service still terminates at a NodePort or a LoadBalancer Service (Gateway API is extending into other protocols, but the L7 HTTP story is the mature part).

NetworkPolicy: default-allow, additive, and only as strong as the CNI

Three properties decide every NetworkPolicy question.

First, pods are default-allow. A pod accepts all ingress and all egress until some policy selects it. The moment one policy selects it for a direction, that pod becomes isolated in that direction and only the union of matching policies' allowed sources gets through. So the way to lock a namespace down is an explicit default-deny that selects everything, then poke holes:

# default-deny ingress for the whole namespace
spec:
  podSelector: {}          # {} selects every pod in the namespace
  policyTypes: [Ingress]   # no ingress rules => nothing allowed in

Second, policies are additive, never subtractive. You cannot write a policy that denies a specific source; you write allows, and everything not allowed by any policy (once the pod is isolated) is denied. Order does not matter because the result is a union.

Third, the cross-namespace selector trap. A bare podSelector inside a from or to block is scoped to the policy's own namespace. To allow a source in another namespace you need a namespaceSelector (optionally combined with a podSelector):

ingress:
  - from:
      - namespaceSelector:
          matchLabels: { kubernetes.io/metadata.name: analytics }
        podSelector:
          matchLabels: { app: reporter }   # reporter pods, in analytics only

Wrong "We created a NetworkPolicy in the namespace, so the namespace is isolated." It isolates only the pods its selector matches, only in the direction it lists, and only if the cluster's CNI plugin implements NetworkPolicy. On a network plugin that does not enforce policy, the API server still accepts the object and nothing changes, which is how a team ends up believing they are segmented when they are wide open. NetworkPolicy is L3/L4 (IPs, ports, namespaces); L7 rules (methods, paths, identities) belong to a service mesh, not to this resource.

Debugging connectivity in order

When "pod A can't reach service B," resolve it top-down: name, then endpoints, then policy, then the pod itself. Skipping to the exotic cause wastes the most time.

• DNS: nslookup B or dig B.ns.svc.cluster.local from inside a pod. No answer points at name scoping (the ndots/search-domain issue above) or CoreDNS, not at the app.
• Endpoints: kubectl get endpointslices -l kubernetes.io/service-name=B. An empty slice means no pod is Ready or the Service selector matches nothing, so the Service resolves but forwards to no one and connections hang or refuse.
• Policy: if DNS and endpoints are fine but traffic is dropped, look for a NetworkPolicy selecting either side. A drop with no RST (a hang that times out) is the classic policy signature, versus an application refusal.
• Pod: curl a backend pod IP directly, bypassing the Service. If that works but the ClusterIP does not, the problem is in the Service or kube-proxy layer, not the app.