The Kubernetes Network Model

Kubernetes sets the rules for the cluster's network and implements none of them itself. A network plug-in keeps them, and every cluster keeps the same rules, whichever plug-in it runs: Calico, Cilium, Flannel, or a cloud's own, such as the Amazon VPC CNI on EKS. What differs from one plug-in to the next is how it keeps them.

• The rules are on this page, grouped by subject.
• The plug-in's choices are listed at the end: where the addresses come from, how a packet crosses between nodes, and how policies are enforced.
• Each viewer shows one plug-in's answer. Kubernetes: the Pod Network shows Calico's, and marks its choices as such.

Pod addresses

The first three rules are Kubernetes's network model, the requirement every plug-in must meet:

• Every pod has its own IP address, unique in the cluster.
• Every pod reaches every other pod at that address, on the same node or another, with no address translation (NAT). A pod receiving a connection sees the sender's own pod address.
• The agents on a node, such as the kubelet, reach every pod on that node.

So a pod behaves like a small machine on one flat network. Three more facts follow from how pods are built:

• A pod's containers share its address. They share one network namespace, so they have the same interfaces and ports, and reach each other on localhost. Two pods can both listen on port 80, because each has its own address.
• A pod's address lasts as long as the pod. A pod that replaces it gets a new one, so nothing should depend on a pod's address; that is what Services are for.
• The Pod object records both addresses: its own in status.podIP, its node's in status.hostIP. kubectl get pods -o wide shows the pod's address and its node.

The address a pod sees on its own interface is the one other pods connect to, with no ports to publish:

hostNetwork: pods on the node's network

A pod with hostNetwork: true has no network of its own: its containers use the node's interfaces and the node's address, and the plug-in is never called for it.

• The cluster's own parts run that way, because they have to work before the pod network does:
  • the control plane (kube-apiserver, etcd, the scheduler, the controller manager) starts before any plug-in is installed;
  • kube-proxy writes rules into the node's own kernel;
  • the plug-in's agent is the plug-in, and cannot wait for the network it builds.
• An ordinary application never needs it. A host-network pod takes the node's ports, so two copies of it cannot run on one node.
• kubectl get pods -A -o wide shows both kinds: most pods with a pod address, the host-network ones with their node's address.

Node addresses

• Each node is a computer with its own address on a network Kubernetes does not build: a subnet in a cloud, or a LAN in a data centre. The cluster assumes the nodes can reach each other on it.
• The Node object records the node's addresses in status.addresses: its InternalIP, and in a cloud often an ExternalIP and host names too. kubectl get nodes -o wide shows them.
• Pod addresses, node addresses and Service addresses must not overlap, so that every address means one thing to every node.

The parts that build the network

Kubernetes writes the rules, and each part of the network is someone else's job:

• Pod addresses and pod-to-pod traffic: the network plug-in, through the CNI.
• Network policies: the same plug-in, when it supports them.
• Names: the cluster's DNS server, usually CoreDNS.
• Services: kube-proxy, or a plug-in that replaces it.

Neither the kubelet nor the API server configures the network. The plug-in is chosen when the cluster is built, and a node stays NotReady until the plug-in has installed itself there: the kubelet reports the node's network as not ready until the runtime finds a CNI configuration.

How a pod gets its network

The pod's network is made before any of its containers start, by the container runtime and the plug-in:

• The kubelet asks the container runtime (containerd or CRI-O) for the pod's sandbox, the part of the pod that holds its namespaces.
• The runtime creates the pod's network namespace: the kernel's way of giving a group of processes their own interfaces, addresses, routes and ports.
• It runs the plug-in's CNI program with the command ADD. The runtime finds the program through its configuration: by convention, a file in /etc/cni/net.d names it and /opt/cni/bin holds it.
  • the program gives the pod an interface, usually eth0, with its address and a route out
  • it connects the interface to the node, usually through a veth pair: two interfaces joined like the ends of a cable, one inside the pod and one on the node
  • it prints the result, the address included, and exits; it is a program the runtime runs, not a process that stays
• The runtime starts the pod's pause process in the namespace. Its only job is to keep the namespace in use for as long as the pod lives.
• The containers start and join the namespace, so the address is theirs, and it survives a container's restart: the sandbox stays, and so do the namespace and the address.
• Deleting the pod runs the program once more, with DEL, which releases the address and removes the interface.

ps -e on a node shows each pod as a small tree: the runtime's shim for the pod, with /pause and the containers' own processes under it.

The CNI call

CNI, the Container Network Interface, is a short specification for one exchange: a runtime asks a program to connect a network namespace to the network.

• The runtime passes the request in environment variables: CNI_COMMAND (ADD or DEL), CNI_NETNS (the namespace's path) and CNI_IFNAME (the interface to create, eth0), with the network's configuration on standard input.
• The program answers in JSON on standard output, the pod's address included. The runtime reports that address to the kubelet, and it becomes the pod's status.podIP.
• How the program moves packets between nodes is its own business, and plug-ins differ most there.

 1kubelet
 2  │  CRI: "run a pod sandbox"
 3  ▼
 4container runtime
 5  │  new network namespace
 6  │  exec /opt/cni/bin/<plug-in>
 7  │    CNI_COMMAND=ADD  CNI_NETNS=…  CNI_IFNAME=eth0
 8  │    stdin: the configuration from /etc/cni/net.d
 9  ▼
10plug-in  ──►  IPAM plug-in   "an address, please"
11  │
12  ▼
13{"ips": [{"address": "10.244.1.5/…"}], …}

IPAM and plug-in chains

• Handing out addresses is a job of its own, IPAM (IP address management). The network plug-in calls an IPAM plug-in for it, named in the configuration's ipam field. IPAM must never give two pods the same address: host-local keeps its records on the node and hands out the node's own range; Calico's IPAM keeps them as objects in the cluster.
• A configuration file can list several plug-ins, in a chain. The runtime calls them in order, and each adds to the result of the one before. portmap, which implements hostPort, is a common link.

A failed ADD

• The runtime cannot start a pod's containers until the plug-in has given the sandbox a network. If the ADD call fails, the pod waits in ContainerCreating.
• kubectl describe pod shows why, in an event with the reason FailedCreatePodSandBox and the plug-in's own error message after (add):.
• The kubelet retries, so a plug-in that recovers lets the pod start with no action from you. Pods already running are not affected: the call happens once, when the sandbox is created.

Network policies

A NetworkPolicy narrows down, for the pods it selects, who may connect to them and whom they may connect to:

 1apiVersion: networking.k8s.io/v1
 2kind: NetworkPolicy
 3metadata:
 4  name: allow-client-policy
 5  namespace: other-ns        # applies inside this namespace
 6spec:
 7  podSelector:               # the pods this policy is about
 8    matchLabels:
 9      app: server
10  policyTypes:               # the directions it controls
11  - Ingress
12  ingress:                   # what may come in
13  - from:
14    - podSelector:           # from these pods
15        matchLabels:
16          app: client
17    ports:                   # on these ports
18    - protocol: TCP
19      port: 80

• NetworkPolicy is one API on every cluster, and the plug-in enforces it. The API server only stores the object. With a plug-in that has no policy support, the policy is stored, kubectl get networkpolicy lists it, and nothing is filtered, with no error anywhere.
• With no policy, all traffic is allowed.
• A pod that a policy selects for a direction (ingress, egress, or both) is isolated in it: only the traffic some policy allows gets through. The two directions are independent: a pod isolated for ingress still sends whatever it likes.
• Policies only allow, and they add up. A pod's allowed traffic is the union of every policy that selects it, so a new policy can only allow more. A policy that selects pods and allows nothing is a default deny.
• Policies name peers by label (pods, namespaces) or by address range (ipBlock). The kernel filters addresses, so the plug-in keeps a list of the addresses that match each label, and rewrites it as pods come and go.
• A policy decides whether a connection may start. The node tracks each connection in the kernel's connection-tracking table, and the replies of an allowed connection always get back. So a server isolated for egress still answers its clients; it cannot open a connection of its own.

The plug-in's choices

Everything above holds on every cluster. These differ from one plug-in to the next, and often with its configuration:

• The pod's range on a node: kube-controller-manager can give each node a range in its spec.podCIDR, but a plug-in need not use it. Calico's own address management ignores it by default.
• The node's end of a pod's interface: a veth pair is usual, not required; some plug-ins use other kinds of interface.
• Traffic leaving the cluster: most plug-ins translate a pod's address to its node's (masquerading) when a pod connects outside the pod network, and most let you turn it off.

Pods on one node: routed or bridged

• Routed: the node is a router with one interface per pod. Each pod's packets go to the node, and the node's routing table holds one route per local pod. Calico works this way.
• Bridged: every pod's veth joins a Linux bridge on the node, as Docker's containers join docker0. The pods share one subnet and reach each other directly. Flannel and the plain bridge plug-in work this way.

Pods on two nodes: routing or a tunnel

A packet for a pod on another node leaves with a pod address as its destination, and the network between the nodes has to carry it there:

• Plain routing: the node sends the packet to the other node as its next hop, unchanged. It is the fastest way, and works only where the network between the nodes delivers packets for pod addresses: the nodes share one link, or the routers between them have learned the pod routes.
• An overlay tunnel: the node wraps the pod's packet in a second one, addressed from its own node address to the other node's. The network in between sees only node addresses; the other node unwraps the packet and delivers the original. The cost is a few bytes of header on every packet.

The extra header takes room from the pod's own packet. A link carries packets up to a size limit, its MTU, so the plug-in sets the pod's eth0 to an MTU smaller by the header's size, and the pod's kernel makes its packets small enough before sending them. A wrong MTU is a classic overlay fault: small packets such as ping get through, and large transfers stall.

The course Kubernetes Networking checks each rule on your own cluster, starting from the same model in its first chapter.