How pods in a Kubernetes cluster get their addresses and reach each other — node addresses, each node's pod range, the network plug-in, a packet from pod to pod on one node and across two, and a NetworkPolicy that filters it.
Every node in a Kubernetes cluster has one address on the network the nodes share, and every pod has an
address of its own, taken from its node's range. The simulator uses the cluster of the
Top View
: first the two kinds of address, then a pod getting
its address, then a packet from pod to pod on one node and across two, and finally a network policy that
filters it.
Kubernetes sets the rules for this network, and a plug-in keeps them: the rules are the same on every
cluster, and listed in
The Kubernetes Network Model
.
This cluster's plug-in is Calico, and the page marks the parts that are Calico's choice, which another
plug-in makes differently.
The simulator's layout
- Each node's footer holds what the node has outside any pod: containerd, the CNI program (dashed:
a file, run only while a pod is created or deleted), and the computer with its
eth0 address. - The five nodes are in the middle, as in the Top View: manage-1 and manage-2 on top, three workers
under them. The system pods are drawn small, since this page is not about them.
- The node network runs along the bottom of the cluster, with a line down to it from each node's
eth0 and its address. - Each node's pod range is in its card's header, labelled
pod range. Each pod shows its network
namespace, a dashed box holding the pause process and eth0 with the pod's address, and under it the
container's process. - Each node card ends with its kernel's routes, as
ip route prints them, and later the network
policy's filter rules. Every packet's path can be read there. - The networks card at the top right holds the cluster's address plan: the node network, the pod
network, and the network policy once it exists.
- Packets fly with their addresses on them. A packet crossing between nodes shows its outer header,
node address to node address, above the pod addresses inside.
- Calico's choices carry a tag,
Calico's choice: the pod network's range per node (in the networks
card), the routes to the other nodes through tunl0, and the wrapped header. Everything untagged
holds on every cluster. - The events are on the right, grouped by scene, with the current step's explanation open.
Node addresses and the node network
Each node is a computer with one address on the node network, here 10.0.1.0/24: the network the
computers are plugged into, a subnet in a cloud or a LAN in a data centre.
- Its routers and switches know these addresses and carry a packet between any two nodes.
- The address sits on the node's interface,
eth0, and the Node object records it as its
InternalIP.
Pod ranges and the network plug-in
The pods' addresses come from a second network, 10.244.0.0/16, which only the nodes know:
- Each node owns a range of 256 addresses, a
/24: kube-controller-manager wrote it into the Node's
spec.podCIDR when the node joined. The ranges do not overlap, so two nodes never hand out the same
address. Using a range per node is Calico's choice here: other plug-ins take pod addresses from the
cloud's own network instead. - Kubernetes does not wire pods itself. It calls a network plug-in through the Container Network
Interface (CNI); this cluster runs Calico, which has two parts on every node:
- the CNI program,
/opt/cni/bin/calico: a file on the node's disk, not a running process, so ps
on the node never lists it. containerd finds it through /etc/cni/net.d/10-calico.conflist- while creating a pod, containerd runs it with the command
ADD - it sets up the pod's interface and address, and exits
- deleting the pod runs it once more, with
DEL
- the agent,
calico-node, one pod per node, which writes the routes to the other nodes and the network
policies' rules
- Each node's kernel has one route per other node:
10.244.2.0/24 via 10.0.1.22 dev tunl0 on worker-1
sends anything for a pod on worker-2 to worker-2's node address, through the tunnel interface. These
routes are Calico's choice, and so is the tunnel.
A pod's address: the veth pair
When the kubelet has containerd start a pod, the pod's network is made before its containers:
- containerd creates the pod's network namespace: a network stack of its own, with its own interfaces
and routes.
- The CNI program, run with the command
ADD, creates a veth pair, two interfaces joined like the ends
of a cable: eth0 inside the pod, cali3f1a9c20b41 on the node. - It takes a free address from the node's range for
eth0, and adds one route on the node:
10.244.1.5 dev cali3f1a9c20b41. - containerd starts the pod's pause process,
/pause, in the namespace: a tiny program whose only
job is to keep the namespace in use for as long as the pod lives. - Each container starts last and joins the same namespace, so
eth0 and its address belong to all of
the pod's containers, and they survive a container's restart.
ps -e on the node lists each pod as a small tree: containerd-shim-runc-v2, the shim containerd
starts for the pod, with /pause and the containers' own processes under it.
The address lasts as long as the pod: a pod that replaces it gets a new one.
Pod to pod on one node
A packet between two pods on one node never leaves the computer. It leaves the client pod at the node's end
of its veth pair, the node's kernel finds the route for the destination pod's address, and sends it into
that pod's pair. kubectl exec client -- curl … runs curl inside the pod, so the request carries the pod's
address, not your workstation's.
Calico makes the node a router between its pods, which is its choice; other plug-ins put the pods on a
bridge instead (
routed or bridged
).
The simulator leaves out how the packet reaches the node, in three steps:
- A pod has no neighbours. Its
eth0 holds its address as a /32, a network of one, so every packet
it sends, even to a pod on the same node, goes to its default gateway, 169.254.1.1. - No interface holds
169.254.1.1. The node's end of the veth pair has proxy ARP turned on, so it
answers the pod's ARP request for the gateway with its own MAC address, ee:ee:ee:ee:ee:ee. - The pod sends the frame to that MAC, the packet arrives in the node, and the node's route for the
other pod takes it on.
So ip neigh in a pod lists one neighbour only, 169.254.1.1, and a traceroute between two pods on one
node shows two hops: the node, then the other pod.
Pod to pod across nodes: the tunnel
A packet for a pod on another node goes through the other node's address:
- worker-1's route for worker-2's range sends the packet to
tunl0. tunl0 wraps it in a new IP header from 10.0.1.21 to 10.0.1.22 (IP-in-IP), since the node
network knows only node addresses. Other plug-ins wrap it in VXLAN instead, or send it unwrapped where
the node network has been given the pod ranges.- worker-2 unwraps it and routes the original packet to the web pod by its own local route.
The web pod sees the request come from 10.244.1.6, the client pod's own address. That is one of the
rules of
the network model
, which every plug-in
keeps: every pod reaches every other pod at its own address, with no address translation (NAT) on the
way. How the packet crosses, here the wrapping, is Calico's choice.
- Calico wraps every packet between nodes when its pool says
ipipMode: Always, as here. With
ipipMode: CrossSubnet it wraps only between nodes on different subnets, and routes plainly between
nodes that share one; it can use VXLAN instead, or no tunnel at all
(
routing or a tunnel
). tunl0 is where the wrapping happens: a packet routed into it leaves by eth0 wrapped, and one
arriving wrapped comes out of it unwrapped. tcpdump on the node's eth0 shows both headers on one
line, IP 10.0.1.21 > 10.0.1.22: IP 10.244.1.6 > 10.244.2.6: ….- The pods' MTU is 1480, 20 bytes under the node network's 1500, so a pod's packet still fits once
wrapped.
- A
traceroute across shows three hops: the pod's own node, the other node's tunl0, and the pod.
The network between the nodes does not appear: to the packet, the tunnel is one hop.
Network policies: filter rules by address
With no policy, every pod reaches every other pod. A NetworkPolicy picks pods by label and lists who may
connect to them:
1apiVersion: networking.k8s.io/v1
2kind: NetworkPolicy
3metadata:
4 name: web-allow-frontend
5spec:
6 podSelector:
7 matchLabels:
8 app: web
9 ingress:
10 - from:
11 - podSelector:
12 matchLabels:
13 role: frontend
14 ports:
15 - protocol: TCP
16 port: 80
- Once a policy picks a pod, whatever no policy allows is dropped.
- The plug-in enforces it, not the API server. calico-node on each node with a web pod writes filter
rules in front of that pod's interface. With a plug-in that has no policy support, such as Flannel alone,
the policy is stored and nothing is filtered.
- The kernel filters addresses, not labels, so the agent looks up the pods labelled
role=frontend
and allows their addresses. Labelling another pod role=frontend adds its address to every list, with
no change to the policy. - A dropped packet gets no answer: curl waits out its
-m 3 limit and exits with code 28, which
kubectl exec reports as command terminated with exit code 28. - The reply needs no rule of its own: the kernel tracks each connection and lets the answers of an
allowed one back.
These behaviours are the same with every plug-in that supports policies; the form of the rules on each
node, drawn here as one line per decision, is Calico's.
Simplifications in the simulator
- Each node's range is one
/24, its podCIDR. Calico's own address management, its default, gives
out blocks of 64 addresses (/26) instead, and a node takes more blocks as it fills; Flannel, and Calico
set to use the node's podCIDR, give out addresses as drawn. - The routes and rules are shortened.
ip route prints a few more words per line (proto bird,
onlink, scope link), and Calico's filter rules are iptables or nftables chains, drawn here as one
line per decision. - Some system pods are left out:
calico-kube-controllers and the cluster's DNS server, CoreDNS. - Steps that happen within milliseconds of each other are drawn one at a time.
The symbols for the node, the kubelet and the pod come from the
Kubernetes Icons Set
, licensed under
Apache-2.0 or CC-BY-4.0.
The course
Kubernetes Networking
works through the
same mechanisms on your own two-node cluster.