[FEATURE]: Allow custom annotations for image builders so that IRSA can be used #2931
Comments
|
Hi @ohookins! Thanks for the detailed suggestion here. Would you be interested in implementing this functionality and submitting a PR? We'd be happy to hop on a call and do some pair-programming on this, and to point you to the right places in the codebase to get started (and answer any questions that may come up). |
|
This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product! |
stale
bot
added
the
stale
vvagaytsev
removed
the
stale
|
Just so nobody else duplicates the effort: I am actively working on this :) |
|
One thing I noticed during the investigation is that a service account would also be useful for the tasks. First implementation will only target in-cluster-building, but I will at least create another feature request issue for allowing for IRSA (or injecting credentials through service accounts in general) also for tasks. |
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
|
We should make sure, when implementing this, that it also can be used to enable using GCP's workload identity: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity |
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931
#3384) * feat(k8s): add service account and irsa support for in-cluster-builder This commit introduces a new service account, and makes the annotations configurable in kaniko and buildkit in-cluster-builders. This change enables the use of IRSA for in-cluter-building which makes it more secure. Fixes #2931 * docs(irsa): add docs for using in-cluster-building with IRSA I decided to replace the existing docs, because the existing approach doesn't provide defense-in-depth. Even in dev clusters, the access to the Docker registries should be restricted to as few places as possible, to make it harder for attackers to push bad images. * chore: fix test * Update core/src/plugins/kubernetes/container/build/common.ts * chore: resolve conflicts on rebase * chore: resolve more conflicts on rebase * revert in-cluster-building.md * docs: add irsa separately * chore: add gcr credential helper to builder and util images * chore: fix serviceAccount for kaniko and support using different kaniko namespace * chore: fix configschema * chore: update reference docs * chore: fix typo in kubernetes config.ts * chore: remove var used for debugging * chore: fix test-framework * chore: add namespace to serviceAccount manifest * chore: add integ tests * chore: remove namespace from util manifest since it is always in project namepsace * chore: use kaniko and buildkit envs for integ tests * chore: add docs for google workload identity * chore: move tests to their own suite * chore: fix docs * chore: remove unused kube api in test * chore: add assertions and use isEqual * Update core/test/integ/src/plugins/kubernetes/container/build/build.ts Co-authored-by: Steffen Neubauer <[email protected]> * chore: use constant * chore: compare resources in test for better output --------- Co-authored-by: Anna Mager <[email protected]> Co-authored-by: Anna Mager <[email protected]>
|
IRSA for Runs is already possible in Garden 0.13 via the kubernetes-pod |
Feature Request
Background / Motivation
According to the AWS EKS Best Practices Guide, when using IRSA you should set the instance metadata service to IMDSv2 and the hop count to 1. This means that pods will no longer be able to talk to the regular instance metadata endpoint and inherit the role of the node.
Under normal conditions, kaniko or cluster-buildkit will just use the default Service Account, and not have any specific IAM role that grants it ECR push permissions, so it will rely on the instance role and ECR push permissions associated with it. If we want to enforce the above security settings, any ECR pushes will fail. Since there's no opportunity to add annotations to these pods, we can't tell IRSA which IAM role the builder should assume, and hence can't give it the right permissions. This will break effectively all modules using in-cluster built images.
What should the user be able to do?
Add a configuration in the Garden project configuration associated with the chosen builder service, that adds the K8s annotation to the builder's service account. This probably also means that a build service account needs to be created by Garden.
Why do they want to do this? What problem does it solve?
See the description above.
Suggested Implementation(s)
This could just be added to the existing configuration map that you pass for the builder in the project configuration. I'm not sure how things like for GKE or AKS works with respect to pod roles, so I guess it would be good to have a solution that works equally well for those platforms (although I'm not using them).
How important is this feature for you/your team?
🌵 Not having this feature makes using Garden painful
It's a non-functional requirement as far as the day to day developer experience goes, but not having this means that any Garden pod can assume the instance IAM role, and use any permissions a given EKS node can use. This is not great for security.
The text was updated successfully, but these errors were encountered: