Workload identity is the best practice for authenticating as a service account when running on GKE.
See https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity for how this works
First enable workload identity on the cluster and all node-pools in the cluster:
enable-workload-identity.sh K8S_PROJECT ZONE CLUSTER
Next ensure the kubernetes service account exists and that it has an
iam.gke.io/gcp-service-account
annotation. This associates it with the desired
GCP service account.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
iam.gke.io/gcp-service-account: [email protected]
name: SOMETHING
namespace: SOMEWHERE
Once this service account exists in the cluster, then an owner of
[email protected]
-- typically a PROJECT
owner --
runs bind-service-accounts.sh
:
bind-service-accounts.sh \
K8S_PROJECT ZONE CLUSTER SOMEWHERE SOMETHING \
[email protected]
This script assumes the same person can access both K8S_PROJECT
and
PROJECT
. If that is not true then the PROJECT
owner can just run this
command directly:
# Note: K8S_PROJECT is the project owning the GKE cluster
# whereas PROJECT owns the service account (may be the same)
gcloud iam service-accounts add-iam-policy-binding \
--project=PROJECT \
--role=roles/iam.workloadIdentityUser \
--member=serviceAccount:K8S_PROJECT.svc.id.goog[SOMEWHERE/SOMETHING] \
[email protected]
This is what tells GCP that the SOMEWHERE/SOMETHING
service account in
K8S_PROJECT
is authorized to act as
[email protected]
.
These are all described in the how-to GKE doc link at the top.
At this point any pod that:
- Runs in a
K8S_PROJECT
GKE cluster - Inside the
SOMEWHERE
namespace - Using the
SOMETHING
service
will authenticate as [email protected]
to GCP. The
bind-service-accounts.sh
script will verify this (see the how-to doc above for
the manual command).
Whenever you want a pod to authenticate this way, just configure the
serviceAccountName
on the PodSpec
. See an example
pod
and
prowjob:
Here are minimal deployment and prow job that use an image and args to print the
authenticated user to STDOUT
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: foo
namespace: SOMEWHERE # from above
labels:
app: foo
spec:
replicas: 1
selector:
matchLabels:
app: foo
template:
metadata:
labels:
app: foo
spec:
serviceAccountName: SOMETHING # from above
containers:
- name: whatever
image: google/cloud-sdk:slim
args:
- gcloud
- auth
- list
periodics:
- name: foo
interval: 10m
decorate: true
spec:
serviceAccountName: SOMETHING # from above, note: namespace is chosen by prow
containers:
- image: google/cloud-sdk:slim
args:
- gcloud
- auth
- list
Note: Workload identity works best with pod utilities: Migrate to use pod utilities if there is no
decoration: true
on your job, and come back to this doc once that's done.
Prow jobs run in kubernetes pods, and each pod is responsible for uploading artifacts
to designated remote storage location(e.g. GCS), this upload is done by a
container called sidecar
. To be able to upload to GCS, sidecar
container
will need to way to authenticate to GCP, which was historically done by GCP
service account key(normally stored as service-account
secret in build
clusters), this key is mounted onto sidecar
container by prow, which it uses
for authenticating with GCP.
Workload identity is a keyless solution from GCP that appears to be more secure than storing keys, and Prow also supports this by running the entire pod with a service account that has GCS operation permission.
To migrate from using GCP service account keys to workload identity, there are two different scenarios, and the steps are different. The differentiator is whether the prow job itself directly interacts with GCP or not, if any of the following config is in the job then it's very likely that the answer is yes:
volumes:
- name: <DOES_NOT_MATTER>
secret:
secretName: service-account
labels:
preset-service-account: "true"
Add the following sections on the prow job config:
decoration_config:
gcs_credentials_secret: "" # Use workload identity for uploading artifacts
spec:
# This account exists in "default" build cluster for now, for any other build cluster it
# can be set up by following the steps from the top of this script.
serviceAccountName: prowjob-default-sa
-
Inspect the test process/script invoked by the prow job, remove any logic that assumes the existence of GCP service account key file, or the environment variable of
GOOGLE_APPLICATION_CREDENTIALS
before migrating. -
Creating new GCP service account with necessary GCP IAM permissions, and set up workload identity with the new service account by following Kubernetes service account and GCP service account above.
-
Modify prow job config by adding the sections similar to above, replacing the service account name with the new service account name.