Access for data-catalogue github actions #4413
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MatMoore
force-pushed
the
find-moj-data-access-to-cadet-outputs
branch
from
a7d272a to
b768e5f
Compare
Terraform Component 🧱:
|
MatMoore
force-pushed
the
find-moj-data-access-to-cadet-outputs
branch
4 times, most recently
from
2067a5e to
2bd782d
Compare
Terraform Component 🧱:
|
We want to schedule Datahub DBT ingestions using github actions. (ministryofjustice/data-catalogue#123) To do this, Github actions needs to be able to assume a role via OIDC, and use it to access the s3 bucket containing the outputs from DBT. See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html We already had IRSAs (IAM roles for service accounts) which can be assumed by Datahub itself, but these assume you are running an application in a kubernetes pod on AWS, whereas in this case we are going to run the ingestion from github actions.
MatMoore
force-pushed
the
find-moj-data-access-to-cadet-outputs
branch
from
2bd782d to
2e7dc47
Compare
MatMoore
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Objective
We want to schedule Datahub DBT ingestions using github actions. (ministryofjustice/data-catalogue#123)
To do this, Github actions needs to be able to assume a role via OIDC and use it to read the bucket containing CaDeT outputs. See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html
We already had defined IRSAs (IAM roles for service accounts) for this, that can be assumed by Datahub itself, but these assume you are running an application in a kubernetes pod on AWS, whereas in this case we are going to run the ingestion from github actions.
Checklist
Note
Each items should be checked. Skipping below checks could delay your PR review!
override-static-analysislabel, I've explained why)