How pods in a Kubernetes cluster get their addresses and reach each other — node addresses, each node's pod range, the network plug-in, a packet from pod to pod on one node and across two, and a NetworkPolicy that filters it.
Every node in a Kubernetes cluster has one address on the network the nodes share, and every pod an
address of its own, from its node's range.
- The cluster is the
Top View's.
- We follow a packet from pod to pod: on one node, then across two.
- Then a network policy filters it.
- Kubernetes sets the rules for this network, the same on every cluster:
The Kubernetes Network Model.
- A network plug-in keeps them. This cluster's is Calico.
Reading the drawing
- The node network runs along the bottom of the cluster, with a line down to it from each node's
eth0. - Each node's pod range is in its card's header.
- A pod's dashed box is its network namespace: the pause process, and
eth0 with the pod's address. - Each node card ends with its Linux kernel, which every packet passes through. It shows the routes
in the step that explains them, and the policy's filter rules in the last scene.
- A packet between nodes shows its outer header, node address to node address, above the pod
addresses inside.
Calico's choice tags what another plug-in does differently. Everything untagged holds on every
cluster.
Node addresses and the node network
Each node is a computer with one address on the node network, here 10.0.1.0/24.
- The node network is the one the computers are plugged into: a subnet in a cloud, or a LAN in a data
centre.
- Its routers and switches know these addresses, and carry a packet between any two nodes.
- The address sits on the node's interface,
eth0. - The Node object records it as its
InternalIP.
Pod ranges and the network plug-in
The pods' addresses come from a second network, 10.244.0.0/16, which only the nodes know. It is one
flat network: every pod reaches every other pod at its own address, whichever node either is on.
- Each node hands out addresses from a range of its own, 256 of them, a
/24:- the range is a share of the pod network's addresses, not a network of its own;
- kube-controller-manager wrote it into the Node's
spec.podCIDR when the node joined; - the ranges do not overlap, so two nodes never hand out the same address;
- a range per node is Calico's choice: other plug-ins take pod addresses from the cloud's own
network.
- Kubernetes calls a network plug-in to wire each pod, through the Container Network Interface
(CNI). Calico has two parts on every node:
- the CNI program,
/opt/cni/bin/calico: a file on the node's disk, so ps never lists it.- containerd finds it through
/etc/cni/net.d/10-calico.conflist. - Creating a pod, containerd runs it with the command
ADD: it sets up the pod's interface and
address, and exits. - Deleting the pod runs it once more, with
DEL.
- the agent,
calico-node, one pod per node: it writes the routes to the other nodes, and the
network policies' rules.
- Each node's kernel has one route per other node:
10.244.2.0/24 via 10.0.1.22 dev tunl0 on
worker-1 sends anything for a pod on worker-2 to worker-2's node address, through the tunnel
interface. These routes and the tunnel are Calico's choice.
A pod's address: the veth pair
When the kubelet has containerd start a pod, the pod's network is made before its containers:
- containerd creates the pod's network namespace: a network stack of its own, with its own
interfaces and routes.
- The CNI program, run with
ADD, creates a veth pair, two interfaces joined like the ends of a
cable: eth0 inside the pod, cali3f1a9c20b41 on the node. - It takes a free address from the node's range for
eth0, and adds one route on the node:
10.244.1.5 dev cali3f1a9c20b41. - containerd starts the pod's pause process,
/pause, in the namespace. Its only job is to keep
the namespace in use for as long as the pod lives. - Each container starts last and joins the same namespace:
eth0 and its address belong to all of the pod's containers;- they survive a container's restart.
ps -e on the node lists each pod as a small tree: containerd-shim-runc-v2, the shim containerd
starts for the pod, with /pause and the containers' own processes under it.
The address lasts as long as the pod: a pod that replaces it gets a new one.
Pod to pod on one node
A packet between two pods on one node stays inside the computer.
- It leaves the client pod at the node's end of its veth pair.
- The node's kernel finds the route for the destination pod's address, and sends the packet into that
pod's pair.
kubectl exec client -- curl … runs curl inside the pod, so the request carries the pod's address,
not our workstation's.
The node is a router between its pods, which is Calico's choice; other plug-ins put the pods on a
bridge (
routed or bridged).
The packet reaches the node in three steps, which the drawing leaves out:
- A pod's
eth0 holds its address as a /32, a network of one, so every packet it sends, even to
a pod on the same node, goes to its default gateway, 169.254.1.1. - The node's end of the veth pair answers for
169.254.1.1: it has proxy ARP turned on, so it
answers the pod's ARP request with its own MAC address, ee:ee:ee:ee:ee:ee. - The pod sends the frame to that MAC, the packet arrives in the node, and the node's route for the
other pod takes it on.
So ip neigh in a pod lists one neighbour only, 169.254.1.1, and a traceroute between two pods on
one node shows two hops: the node, then the other pod.
Pod to pod across nodes: the tunnel
A packet for a pod on another node travels inside one addressed to the other node.
- worker-1's route for worker-2's range sends the packet to
tunl0. tunl0 wraps it in a new IP header from 10.0.1.21 to 10.0.1.22 (IP-in-IP).- worker-2 unwraps it and routes the original packet to the web pod by its own local route.
The web pod sees the request come from 10.244.1.6, the client pod's own address.
- That is a rule of
the network model, which
every plug-in keeps: a pod reaches every other pod at its own address, with no address translation
(NAT) on the way.
- How the packet crosses is the plug-in's choice. Calico's options:
ipipMode: Always, as here, wraps every packet between nodes;ipipMode: CrossSubnet wraps only between nodes on different subnets;- VXLAN, or no tunnel at all
(
routing or a tunnel).
What the tunnel looks like on the nodes:
tunl0 is where the wrapping happens: a packet routed into it leaves by eth0 wrapped, and one
arriving wrapped comes out of it unwrapped.tcpdump on the node's eth0 shows both headers on one line:
IP 10.0.1.21 > 10.0.1.22: IP 10.244.1.6 > 10.244.2.6: ….- The pods' MTU is 1480, 20 bytes under the node network's 1500, so a pod's packet still fits once
wrapped.
- A
traceroute across shows three hops: the pod's own node, the other node's tunl0, and the
pod. To the packet, the tunnel is one hop.
Network policies: filter rules by address
With no policy, every pod reaches every other pod. A NetworkPolicy picks pods by label and lists who may
connect to them:
1apiVersion: networking.k8s.io/v1
2kind: NetworkPolicy
3metadata:
4 name: web-allow-frontend
5spec:
6 podSelector:
7 matchLabels:
8 app: web
9 ingress:
10 - from:
11 - podSelector:
12 matchLabels:
13 role: frontend
14 ports:
15 - protocol: TCP
16 port: 80
- Once a policy picks a pod, whatever no policy allows is dropped.
- The plug-in enforces the policy: calico-node on each node with a web pod writes filter rules in
front of that pod's interface.
- A plug-in with no policy support, such as Flannel alone, leaves the policy stored and nothing
filtered.
- The kernel filters by address, so the agent looks up the pods labelled
role=frontend and allows
their addresses. Labelling another pod role=frontend adds its address to every list, with no change
to the policy. - A dropped packet gets no answer: curl waits out its
-m 3 limit and exits with code 28, which
kubectl exec reports as command terminated with exit code 28. - The reply needs no rule of its own: the kernel tracks each connection and lets the answers of an
allowed one back.
The form of the rules on each node is Calico's; everything else here holds for every plug-in that
supports policies.
The symbols for the node, the kubelet and the pod come from the
Kubernetes Icons Set, licensed under
Apache-2.0 or CC-BY-4.0.
The course
Kubernetes Networking works through the
same mechanisms on your own two-node cluster.