Pipeline Identities, Part 5: Deploying to EKS
In this part you'll let the pipeline deploy to an EKS cluster. The call you'll cover:
- call 6, job → EKS: the job deploys the image with the AWS credentials from Part 4.
What it shows:
- Two worlds check the job, so it is refused twice before it gets in:
- IAM decides what the role may do in AWS: read the cluster's address;
- the cluster's access entry decides what the role may do inside Kubernetes.
- The pipeline may change one namespace,
web, and nothing else. - The deployment ends with pods that cannot pull the private image: Part 6 gives the cluster a credential of its own.
Create the cluster
Create an EKS cluster with the Terraform EKS lab.
- It costs money from the moment it exists: a NAT gateway, the control plane and two nodes.
- Creating it takes 15 minutes or more.
- If AWS refuses its Kubernetes version, raise
cluster_versionin itseks_terraform_config.tf.
Point
kubectlon your machine at the cluster, with the cluster name and region of that lab:1$ aws eks update-kubeconfig --name my-learn-eks --region us-west-2 2$ kubectl get nodes
Add the manifest
Create k8s/deployment.yaml in your repository: two pods serving the image, and a Service in front of them. The
pipeline replaces IMAGE_PLACEHOLDER with the image it pushed.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: hello
5spec:
6 replicas: 2
7 selector:
8 matchLabels:
9 app: hello
10 template:
11 metadata:
12 labels:
13 app: hello
14 spec:
15 containers:
16 - name: web
17 image: IMAGE_PLACEHOLDER
18 ports:
19 - containerPort: 80
20---
21apiVersion: v1
22kind: Service
23metadata:
24 name: hello
25spec:
26 selector:
27 app: hello
28 ports:
29 - port: 80
30 targetPort: 80
Add the deploy steps
Add one more repository variable,
EKS_CLUSTER, with the cluster's name:my-learn-eks.In
.github/workflows/pipeline.yml, add three steps after Ask AWS who the job is:1 - name: Point kubectl at the cluster 2 run: aws eks update-kubeconfig --name "${{ vars.EKS_CLUSTER }}" --region "${{ vars.AWS_REGION }}" 3 4 - name: Ask Kubernetes who the job is 5 run: kubectl auth whoami 6 7 - name: Deploy to the web namespace 8 run: | 9 sed "s|IMAGE_PLACEHOLDER|$IMAGE|" k8s/deployment.yaml | kubectl apply -n web -f - 10 sleep 20 11 kubectl get pods -n web
aws eks update-kubeconfigasks AWS for the cluster's address and certificate, and writes the job's kubeconfig.kubectl auth whoamiasks the cluster who the caller is.- The deploy step applies the manifest to
web, then lists the pods.
Push, and read AWS's refusal
Commit and push. In the run, Point kubectl at the cluster fails (cut short here):
1An error occurred (AccessDeniedException) when calling the DescribeCluster operation:
2User: arn:aws:sts::<account-id>:assumed-role/pipeline-identities/GitHubActions is not authorized to perform:
3eks:DescribeCluster on resource: arn:aws:eks:us-west-2:<account-id>:cluster/my-learn-eks …
- The role was made in Part 4 with no permissions.
- IAM refuses every AWS action the role is not granted.
Grant the role a right in AWS
Create one more file in
pipeline-identities-aws/,eks.tf:1variable "cluster_name" { 2 description = "The cluster from the Terraform EKS lab" 3 type = string 4} 5 6data "aws_eks_cluster" "this" { 7 name = var.cluster_name 8} 9 10# In AWS: the role may read the cluster's address (aws eks update-kubeconfig). 11data "aws_iam_policy_document" "describe" { 12 statement { 13 actions = ["eks:DescribeCluster"] 14 resources = [data.aws_eks_cluster.this.arn] 15 } 16} 17 18resource "aws_iam_role_policy" "describe" { 19 name = "describe-cluster" 20 role = aws_iam_role.pipeline.id 21 policy = data.aws_iam_policy_document.describe.json 22}Add the cluster's name to
values.tfvars:1region = "us-west-2" 2github_repo = "your-github-user/pipeline-identities" 3cluster_name = "my-learn-eks"Apply it:
1$ terraform apply -var-file=values.tfvars
Run again, and read the cluster's refusal
Open the failed run, and Re-run all jobs. Point kubectl at the cluster passes now, and Ask Kubernetes who the job is fails:
1error: You must be logged in to the server (Unauthorized)
- The cluster learned which role is calling, and found no access entry for it.
- Being someone in AWS lets nobody into a cluster.
See the token kubectl sends
The cluster learned which role was calling, though the job sent it no AWS key: kubectl sends a request to STS,
signed and not yet sent.
- The kubeconfig runs
aws eks get-tokenbefore each request. - That command signs an STS
GetCallerIdentityrequest with the AWS credentials, and wraps it as a token startingk8s-aws-v1.. - EKS sends the signed request to STS, and STS answers with the caller's role.
- EKS looks for the role's access entry: with none, it answers
Unauthorized; with one, it checks the request against it.
Run the same command on your own machine. The token is k8s-aws-v1., then the signed URL in base64
(aHR0cHM6Ly9zdHM is https://sts):
1$ aws eks get-token --cluster-name my-learn-eks --region us-west-2 | jq -r .status.token | cut -c1-26
2k8s-aws-v1.aHR0cHM6Ly9zdHM
Grant the role access in the cluster
Create a second file in
pipeline-identities-aws/,eks-access.tf:1variable "namespace" { 2 description = "The one namespace the pipeline may change" 3 type = string 4 default = "web" 5} 6 7# In the cluster: the role may edit the namespace "web", and nothing else. 8resource "aws_eks_access_entry" "pipeline" { 9 cluster_name = var.cluster_name 10 principal_arn = aws_iam_role.pipeline.arn 11 type = "STANDARD" 12} 13 14resource "aws_eks_access_policy_association" "pipeline" { 15 cluster_name = var.cluster_name 16 principal_arn = aws_iam_role.pipeline.arn 17 policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy" 18 19 access_scope { 20 type = "namespace" 21 namespaces = [var.namespace] 22 } 23 24 depends_on = [aws_eks_access_entry.pipeline] 25}- The access entry lets the role into the cluster.
AmazonEKSEditPolicy, scoped toweb, gives it the rights of Kubernetes'editrole, in that namespace only.
Apply it, then create the namespace yourself: the pipeline may change
web, but not create it.1$ terraform apply -var-file=values.tfvars 2$ kubectl create namespace web 3namespace/web createdCheck the EKS console: your cluster → Access lists the role, with
AmazonEKSEditPolicyscoped toweb.
Run again, and read the pods' failure
Open the failed run, and Re-run all jobs.
Read Ask Kubernetes who the job is: it prints the role as a Kubernetes user.
1ATTRIBUTE VALUE 2Username arn:aws:sts::<account-id>:assumed-role/pipeline-identities/GitHubActions 3Groups [system:authenticated] 4…Read the deploy step: it lists two pods in
ErrImagePull, which becomeImagePullBackOff.Ask for their events, which say why (cut short here):
1$ kubectl get pods -n web 2NAME READY STATUS RESTARTS AGE 3hello-6d4c9f7b8-2xkqp 0/1 ImagePullBackOff 0 1m 4hello-6d4c9f7b8-h7v5n 0/1 ImagePullBackOff 0 1m 5$ kubectl describe pod -n web -l app=hello | grep -m1 Failed 6 Warning Failed … failed to pull and unpack image "docker.io/<your-docker-id>/pipeline-identities:…": 7 … pull access denied, repository does not exist or may require authorization …
- The job had the right to deploy, and it did.
- The node pulling the image has no Docker Hub credential, and the repository is private.
Try a namespace outside web
Add a step after Point kubectl at the cluster:
1 - name: Look outside the web namespace 2 run: kubectl get pods -n defaultCommit and push, and read the step's error:
1Error from server (Forbidden): pods is forbidden: User "arn:aws:sts::<account-id>:assumed-role/pipeline-identities/GitHubActions" 2cannot list resource "pods" in API group "" in the namespace "default"AWS let the job in; Kubernetes refuses outside
web.Remove the step, and commit and push.