Scheduled ingestion of CaDeT assets

[seanprivett]

Where and how should this ingestion be scheduled

DataHub actions API?

Should the ingestion recipes live in GitHub?

Do it live

[MatMoore]

We think yes, github actions is the way to go here. It will be a proof of concept for other ingestion types, and it will enable us to add custom transformers if we need to (e.g. to assign domains in ministryofjustice/find-moj-data#343)

[MatMoore]

TODO:

• allow Github actions to authenticate with datahub (use CATALOGUE_TOKEN)
• allow Github actions to access DBT outputs in s3 (use aws-actions/configure-aws-credentials with a role)
• set up a scheduled workflow

[MatMoore]

To allow github actions to access this bucket, we need

• A policy for reading from the bucket
• A role assumable by github actions

An OIDC assumable role looks something like this:

data "aws_iam_policy_document" "ci_ingestion_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/token.actions.githubusercontent.com"]
    }
    condition {
      test     = "StringEquals"
      values   = ["sts.amazonaws.com"]
      variable = "token.actions.githubusercontent.com:aud"
    }
    condition {
      test     = "StringLike"
      values   = ["repo:ministryofjustice/find-moj-data:*"]
      variable = "token.actions.githubusercontent.com:sub"
    }
  }
}

resource "aws_iam_role" "ci_ingestion_role" {
  name               = "ci-ingestion"
  assume_role_policy = data.aws_iam_policy_document.ci_ingestion_role.json
  tags               = local.tags
}

resource "aws_iam_role_policy_attachment" "ci_ingestion_role" {
  policy_arn = ...
  role       = aws_iam_role.ci_ingestion_role.name
}

See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub

If we want to restrict access it to builds of the main branch, then we can use repo:ministryofjustice/find-moj-data:ref:refs/heads/main for the subject instead of using a wildcard

[MatMoore]

Policy datahubReadCaDeTBucket is already defined in the analytical-platform repo