Overview

Self-managed Kubernetes clusters can support the same identity and security features as managed services, but getting there requires some extra setup.

In this post we’ll build a bare-metal Kubernetes cluster and configure it to access AWS resources in the same way an EKS (or other AWS-hosted Kubernetes) cluster does, without relying on long-lived credentials like access keys. To accomplish this, we’ll use IAM Roles for Service Accounts (IRSA), allowing Kubernetes workloads to assume AWS IAM roles using short-lived, federated credentials.

What is IRSA & How does it work

The super short explanation of IRSA is it allows you to map Service Accounts in a Kubernetes Cluster to IAM Roles in AWS, using the Service Account Token (which is just a JWT) provided by Kubernetes.

To get IRSA working we need a few things setup, in AWS we need:

To configure a Federated Access provider
The AWS role(s) we want to assume need to have a trust policy that allows our federated access provider

In our Kubernetes cluster we need to configure:

The Service Account Issuer URL must be reachable from AWS
A projected volume with the serviceAccountToken we’re going to use needs to be configured on pods that want access to AWS

Configuring the Kubernetes API Server

Find your kube-apiserver pod in the kube-system namespace and describe it to get something like the following:

...
    Command:
      kube-apiserver
...
      --anonymous-auth=true
      --service-account-issuer=https://kubernetes.mydomain.lan
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
...

If your issuer is already public then you’re good to go, if it’s not you’ll need to edit the manifest located in /etc/kubernetes/manifests/kubernetes/kube-apiserver.yaml on your control-plane node(s). Note: Don’t use kubectl edit ... for this, it wont persist.

Add an additional --service-account-issuer=https://new.publicly.accessible.domain.com above your current issuer. This will allow your API server to start generating tokens with the new issuer, but also to accept tokens from the old issuer as tokens get rotated out for the newer issuer.

Note: This will cause downtime for the control-plane as some things are restarted so plan accordingly. Not as much of an issue in clusters with a highly available control-plane.

Ensuring public acess to your issuer

Run kubectl get --raw /.well-known/openid-configuration and get your jwks_uri from the output, this is what systems need to hit to get your signing keys. For me this is /openid/v1/jwks.

If you can’t access that url outside of your cluster with a regular HTTP(S) request, apply the following configuration to allow anonymous access to it.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: anonymous-access-jwks
rules:
- nonResourceURLs:
  - "/openid/v1/jwks"
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: anonymous-access-jwks
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: anonymous-access-jwks
subjects:
- kind: Group
  name: system:unauthenticated

Configuring a projected volume with short-lived tokens

The Kubernetes documentation is amazing so I’m not going to say much on this, you’re just going to create a projected volume for each pod that wants to access AWS: https://kubernetes.io/docs/concepts/storage/projected-volumes/#serviceaccounttoken. Make sure you set your token audience to whatever is in your IAM trust policy, standard practice is to use sts.amazonaws.com.

In EKS clusters this happens with a mutating webhook that mutates any pod with an attached service account that has an annotation eks.amazonaws.com/role-arn=role-here. For brevity I wont be setting that up in this blog, but will do another in the future with that functionality for a full feature parity with EKS.

You also need to set a couple environment variables:

AWS_WEB_IDENTITY_TOKEN_FILE and the value should be the path to your Service Accout Token. Standard practice is to use the path /var/run/secrets/<descriptor>/serviceaccount/token. In EKS the <descriptor> is eks.amazonaws.com. Anything descriptive related to AWS would be sufficient, or you can use what AWS does.
AWS_ROLE_ARN and the value should be the role in AWS you want to assume.

Configuring Federated Access Provider

We can quickly configure this with the below Terraform snippit, otherwise you can configure it via the AWS console in the IAM section under Identity Providers.

data "tls_certificate" "k8s_baremetal" {
    url = "new.publicly.accessible.domain.com"
}

resource "aws_iam_openid_connect_provider" "k8s_baremetal" {
    url = "new.publicly.accessible.domain.com"
    client_id_list = ["sts.amazonaws.com"]
    thumbprint_list = [
        for cert in data.tls_certificate.k8s_baremetal.certificates : cert.sha1_fingerprint
    ]
}

Configuring AWS Role Trust Policy

For every role we want to assume from within our cluster we have to configure a Trust Policy that defines which pods are able to assume the role.

An example would look like this:

{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:my-k8s-namespace:my-service-account",
                    "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Statement.Principal.Federated is our Federated Access Provider from the previous step.

Statement.Action is the permission we want the pod to have, in this case we want it to be able to assume any role this trust policy is attached to.

Statement.Condition.StringEquals.* are the two claims from our Kubernetes JWT that we want to restrict this policy with. The one ending in aud is short for Audience, and in this case our Audience is the Secure Token Service at AWS. The one ending in sub is the Subject, for Kubernetes the format is system:serviceaccount:<namespace>:<service account name>.

If you want a more general Trust Policy you can swap this StringEquals part out for a StringLike and use wildcards like system:serviceaccount:<namespace>:*, though generally not recommended.

End Result

Throwing all of this together we should now be able to federate access to AWS from our Kubernetes pods, as long as the applications were written using the AWS SDK and understand how to use the AWS_WEB_IDENTITY_TOKEN_FILE. Digging further into how to use that file is outside the scope of this post but in practice this “just works” for most applications.

In a future post, I plan to dive into setting up a mutating webhook so we can annotate service accounts in the same way we do in an EKS cluster. For now, this demonstrates the core functionality required to get workload identity federation working on a self-managed Kubernetes cluster.

While this walkthrough is AWS-focused, the approach itself is not. The same workload identity federation pattern can be applied to any platform that supports federated identity, such as Google Cloud, Azure, Snowflake, Trino, and others, making this a reusable model rather than a one-off AWS solution.

Helix's Blog

Workload Identity Federation in Baremetal Kubernetes Clusters