The Kubernetes API provides access to the Kubernetes objects in a cluster. However, authentication is required before any entity can interact with the objects in a Kubernetes cluster. The handy kubectl command or (k command as most of us ubiquitously set it in our terminal) requires a user to be set in the kubeconfig. This user’s credentials, combined with the cluster’s URL and a default namespace, are used to establish a context to tell kubectl how to connect to a cluster. This is the typical mechanism for interacting with a Kubernetes cluster in a terminal. However, sometimes a Kubernetes workload (for example, a pod) needs to communicate with the Kubernetes API of the cluster in which it is deployed. The kubeconfig is stored outside the cluster, so a pod cannot authenticate with a user in the kubeconfig; hence, the need for a different mode of authentication. Enter service accounts.
A service account is a distinct identity within a Kubernetes cluster. It is stored in the Kubernetes cluster (in etcd), and as such, it is Kubernetes-managed. Service accounts allow application pods or internal processes in a cluster to authenticate and interact with the Kubernetes API server.
Note: Service Accounts are namespaced. Every Kubernetes namespace gets a default service account upon creation.
This article will discuss two points:
Prerequisites:
Use cases of Service Accounts in Kubernetes
CI/CD pipelines
CI/CD workflows can be integrated with a Kubernetes cluster to spin up ephemeral services to run CI jobs. An example is Gitlab’s CI/CD integration with K8s. A service account is required to ensure restricted access for the CI jobs, and in some cases, to allow access to the cluster.
Monitoring and Logging
Monitoring tools like Prometheus need to collect logs, metrics, and traces across workloads in a cluster. At the base level, Prometheus needs to gather logs from nodes, ingresses, services, and pods, depending on the monitoring setup. For these log collections to happen, a service account is needed to grant Prometheus the required privileges in the Kubernetes cluster it exists in.
Dependent-startup pipeline
A service can be dependent on another service to function (for example, an API that needs a database). To solve this problem, the deployment of the dependent services that the application needs to function should be ordered. A reliable pattern to solve this problem in Kubernetes is to check the deployment status of the dependent services by interacting with the cluster’s API. A service account is used to authenticate against the cluster’s API to get the deployment status of the dependent services.
Example Usage of Service Accounts in a Dependent-Startup Pipeline
It is not uncommon for a deployment to be executed in order. For example, look at the infographic below:
All the services for this mythical application need to be deployed in this order. A tool that helps with this is k8s-wait-for. It is a simple script that allows waiting for a Kubernetes service, job, or pod to enter the desired state.
In this example, we will demonstrate the API deployment dependency of a Todo API by running migrations first before deploying the API. Referenced source code can be found here.
Here’s what the API deployment looks like:
For this implementation, the database migration will be run as a Kubernetes Job. To listen for when to trigger the API deployment after the DB migration job is successful with k8s-wait-for, we’ll use an init container on the API deployment manifest.
An initContainer is a container in a pod that runs to completion before the pod containers are started.
Database migration Job manifest:
apiVersion: batch/v1
kind: Job
metadata:
name: go-todo-pre-deploy
labels:
app: go-todo
tier: job
spec:
template:
spec:
containers:
- name: pre-deploy
image: docker.io/teniolafatunmbi/go-todo
command:
[
"sh",
"-c",
'migrate -source file:///app/internal/database/migrations -database "postgres://$DB_USER:$DB_PASS@$DB_HOST:$DB_PORT/$DB_NAME?sslmode=disable" up',
]
envFrom:
- configMapRef:
name: go-todo-config
restartPolicy: Never
API Deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-todo-deployment
namespace: go-todo
labels:
app: go-todo
tier: application
spec:
selector:
matchLabels:
app: go-todo
tier: application
template:
metadata:
name: go-todo-app
labels:
app: go-todo
tier: application
spec:
serviceAccountName: go-todo-sa
initContainers:
- name: wait-for-go-todo-pre-deploy
image: ghcr.io/groundnuty/k8s-wait-for:v2.0
imagePullPolicy: Always
args:
- "job"
- "go-todo-pre-deploy"
containers:
- name: go-todo
image: docker.io/teniolafatunmbi/go-todo
ports:
- containerPort: 9000
envFrom:
- configMapRef:
name: go-todo-config
readinessProbe:
httpGet:
path: /todos
port: 9000
initialDelaySeconds: 3
periodSeconds: 3
livenessProbe:
httpGet:
path: /
port: 9000
initialDelaySeconds: 3
periodSeconds: 3
The initContainers stanza in the go-todo-deployment defines a container that runs the k8s-wait-for script with the job go-todo-pre-deploy arguments. What that does is to wait for the go-todo-pre-deploy to be successful. If the job is successful, the wait-for-go-todo-pre-deploy init container runs successfully, and the API deployment proceeds.
There’s one field to note in this go-todo-deployment manifest - the serviceAccountName field. Remember that k8s-wait-for checks the status of a specified Kubernetes object. For that to happen, it needs to interact with the Kubernetes API server. And, if there’s no authentication identity, reading from the Kubernetes API server will fail. You’ll get an error that reads like this:
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list resource "<resource-name>" in API group "" in the namespace "<namespace-name>"
This is because every pod is assigned a default service account, but it has limited privileges. Hence, a need for a dedicated service account with only the required privileges for the init container to run.
Regarding the service account to use: A Pod is assigned a default service account with limited permissions. It typically lacks the ability to list or modify most cluster resources beyond basic API discovery. Whether or not to update the privileges in the default service account or use a dedicated service account is a matter of choice. I use a dedicated service account (go-todo-sa) in this case because I like to be explicit.
Service Account manifests:
Creating a service account for use requires three Kubernetes objects: the ServiceAccount, a Role, and a RoleBinding.
The Role defines the privileges to grant.
The ServiceAccount defines the name (and other metadata) of the service account.
The RoleBinding attaches a defined Role to a ServiceAccount.
Manifests:
apiVersion: v1
kind: ServiceAccount
metadata:
name: go-todo-sa
namespace: go-todo
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: go-todo
name: pod-reader
rules:
- apiGroups: ["", "batch"]
resources: ["pods", "jobs"]
verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: go-todo
subjects:
- kind: ServiceAccount
name: go-todo-sa
namespace: go-todo
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
With this setup - a job to run database migrations, a dedicated service account to read jobs and pods, and an init container to listen for the completion status of the database migration job, we have an API deployment pipeline that is executed in a defined order.
Conclusion
Service accounts are a critical component for authentication in Kubernetes. Understanding how they work is crucial to building and orchestrating Kubernetes workloads and components that interact with the Kubernetes API server securely.
