Pipeline Identities, Part 5: Deploying to EKS

In this part you'll let the pipeline deploy to an EKS cluster. The call you'll cover:

  • call 6, job → EKS: the job deploys the image with the AWS credentials from Part 4.

What it shows:

  • Two worlds check the job, so it is refused twice before it gets in:
    • IAM decides what the role may do in AWS: read the cluster's address;
    • the cluster's access entry decides what the role may do inside Kubernetes.
  • The pipeline may change one namespace, web, and nothing else.
  • The deployment ends with pods that cannot pull the private image: Part 6 gives the cluster a credential of its own.

Create the cluster

  1. Create an EKS cluster with the Terraform EKS lab.

    • It costs money from the moment it exists: a NAT gateway, the control plane and two nodes.
    • Creating it takes 15 minutes or more.
    • If AWS refuses its Kubernetes version, raise cluster_version in its eks_terraform_config.tf.
  2. Point kubectl on your machine at the cluster, with the cluster name and region of that lab:

    1$ aws eks update-kubeconfig --name my-learn-eks --region us-west-2
    2$ kubectl get nodes
    

Add the manifest

Create k8s/deployment.yaml in your repository: two pods serving the image, and a Service in front of them. The pipeline replaces IMAGE_PLACEHOLDER with the image it pushed.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: hello
 5spec:
 6  replicas: 2
 7  selector:
 8    matchLabels:
 9      app: hello
10  template:
11    metadata:
12      labels:
13        app: hello
14    spec:
15      containers:
16        - name: web
17          image: IMAGE_PLACEHOLDER
18          ports:
19            - containerPort: 80
20---
21apiVersion: v1
22kind: Service
23metadata:
24  name: hello
25spec:
26  selector:
27    app: hello
28  ports:
29    - port: 80
30      targetPort: 80

Add the deploy steps

  1. Add one more repository variable, EKS_CLUSTER, with the cluster's name: my-learn-eks.

  2. In .github/workflows/pipeline.yml, add three steps after Ask AWS who the job is:

     1      - name: Point kubectl at the cluster
     2        run: aws eks update-kubeconfig --name "${{ vars.EKS_CLUSTER }}" --region "${{ vars.AWS_REGION }}"
     3
     4      - name: Ask Kubernetes who the job is
     5        run: kubectl auth whoami
     6
     7      - name: Deploy to the web namespace
     8        run: |
     9          sed "s|IMAGE_PLACEHOLDER|$IMAGE|" k8s/deployment.yaml | kubectl apply -n web -f -
    10          sleep 20
    11          kubectl get pods -n web
    
  • aws eks update-kubeconfig asks AWS for the cluster's address and certificate, and writes the job's kubeconfig.
  • kubectl auth whoami asks the cluster who the caller is.
  • The deploy step applies the manifest to web, then lists the pods.

Push, and read AWS's refusal

Commit and push. In the run, Point kubectl at the cluster fails (cut short here):

1An error occurred (AccessDeniedException) when calling the DescribeCluster operation:
2User: arn:aws:sts::<account-id>:assumed-role/pipeline-identities/GitHubActions is not authorized to perform:
3eks:DescribeCluster on resource: arn:aws:eks:us-west-2:<account-id>:cluster/my-learn-eks …
  • The role was made in Part 4 with no permissions.
  • IAM refuses every AWS action the role is not granted.

Grant the role a right in AWS

  1. Create one more file in pipeline-identities-aws/, eks.tf:

     1variable "cluster_name" {
     2  description = "The cluster from the Terraform EKS lab"
     3  type        = string
     4}
     5
     6data "aws_eks_cluster" "this" {
     7  name = var.cluster_name
     8}
     9
    10# In AWS: the role may read the cluster's address (aws eks update-kubeconfig).
    11data "aws_iam_policy_document" "describe" {
    12  statement {
    13    actions   = ["eks:DescribeCluster"]
    14    resources = [data.aws_eks_cluster.this.arn]
    15  }
    16}
    17
    18resource "aws_iam_role_policy" "describe" {
    19  name   = "describe-cluster"
    20  role   = aws_iam_role.pipeline.id
    21  policy = data.aws_iam_policy_document.describe.json
    22}
    
  2. Add the cluster's name to values.tfvars:

    1region       = "us-west-2"
    2github_repo  = "your-github-user/pipeline-identities"
    3cluster_name = "my-learn-eks"
    
  3. Apply it:

    1$ terraform apply -var-file=values.tfvars
    

Run again, and read the cluster's refusal

Open the failed run, and Re-run all jobs. Point kubectl at the cluster passes now, and Ask Kubernetes who the job is fails:

1error: You must be logged in to the server (Unauthorized)
  • The cluster learned which role is calling, and found no access entry for it.
  • Being someone in AWS lets nobody into a cluster.

See the token kubectl sends

The cluster learned which role was calling, though the job sent it no AWS key: kubectl sends a request to STS, signed and not yet sent.

  1. The kubeconfig runs aws eks get-token before each request.
  2. That command signs an STS GetCallerIdentity request with the AWS credentials, and wraps it as a token starting k8s-aws-v1..
  3. EKS sends the signed request to STS, and STS answers with the caller's role.
  4. EKS looks for the role's access entry: with none, it answers Unauthorized; with one, it checks the request against it.

Run the same command on your own machine. The token is k8s-aws-v1., then the signed URL in base64 (aHR0cHM6Ly9zdHM is https://sts):

1$ aws eks get-token --cluster-name my-learn-eks --region us-west-2 | jq -r .status.token | cut -c1-26
2k8s-aws-v1.aHR0cHM6Ly9zdHM

Grant the role access in the cluster

  1. Create a second file in pipeline-identities-aws/, eks-access.tf:

     1variable "namespace" {
     2  description = "The one namespace the pipeline may change"
     3  type        = string
     4  default     = "web"
     5}
     6
     7# In the cluster: the role may edit the namespace "web", and nothing else.
     8resource "aws_eks_access_entry" "pipeline" {
     9  cluster_name  = var.cluster_name
    10  principal_arn = aws_iam_role.pipeline.arn
    11  type          = "STANDARD"
    12}
    13
    14resource "aws_eks_access_policy_association" "pipeline" {
    15  cluster_name  = var.cluster_name
    16  principal_arn = aws_iam_role.pipeline.arn
    17  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy"
    18
    19  access_scope {
    20    type       = "namespace"
    21    namespaces = [var.namespace]
    22  }
    23
    24  depends_on = [aws_eks_access_entry.pipeline]
    25}
    
    • The access entry lets the role into the cluster.
    • AmazonEKSEditPolicy, scoped to web, gives it the rights of Kubernetes' edit role, in that namespace only.
  2. Apply it, then create the namespace yourself: the pipeline may change web, but not create it.

    1$ terraform apply -var-file=values.tfvars
    2$ kubectl create namespace web
    3namespace/web created
    
  3. Check the EKS console: your cluster → Access lists the role, with AmazonEKSEditPolicy scoped to web.

Run again, and read the pods' failure

  1. Open the failed run, and Re-run all jobs.

  2. Read Ask Kubernetes who the job is: it prints the role as a Kubernetes user.

    1ATTRIBUTE   VALUE
    2Username    arn:aws:sts::<account-id>:assumed-role/pipeline-identities/GitHubActions
    3Groups      [system:authenticated]
    4…
    
  3. Read the deploy step: it lists two pods in ErrImagePull, which become ImagePullBackOff.

  4. Ask for their events, which say why (cut short here):

    1$ kubectl get pods -n web
    2NAME                     READY   STATUS             RESTARTS   AGE
    3hello-6d4c9f7b8-2xkqp    0/1     ImagePullBackOff   0          1m
    4hello-6d4c9f7b8-h7v5n    0/1     ImagePullBackOff   0          1m
    5$ kubectl describe pod -n web -l app=hello | grep -m1 Failed
    6  Warning  Failed  …  failed to pull and unpack image "docker.io/<your-docker-id>/pipeline-identities:…":
    7  … pull access denied, repository does not exist or may require authorization …
    
  • The job had the right to deploy, and it did.
  • The node pulling the image has no Docker Hub credential, and the repository is private.

Try a namespace outside web

  1. Add a step after Point kubectl at the cluster:

    1      - name: Look outside the web namespace
    2        run: kubectl get pods -n default
    
  2. Commit and push, and read the step's error:

    1Error from server (Forbidden): pods is forbidden: User "arn:aws:sts::<account-id>:assumed-role/pipeline-identities/GitHubActions"
    2cannot list resource "pods" in API group "" in the namespace "default"
    

    AWS let the job in; Kubernetes refuses outside web.

  3. Remove the step, and commit and push.