diff --git a/aks/dfe_analytics/README.md b/aks/dfe_analytics/README.md new file mode 100644 index 0000000..e7f83a4 --- /dev/null +++ b/aks/dfe_analytics/README.md @@ -0,0 +1,118 @@ +# DfE Analytics +Create resources in Google cloud Bigquery and provides the required variables to applications so they can send events. + +## Examples +### Reuse existing dataset and events table + +```hcl +module "dfe_analytics" { + source = "./vendor/modules/dfe-terraform-modules//aks/dfe_analytics" + + azure_resource_prefix = var.azure_resource_prefix + cluster = var.cluster + namespace = var.namespace + service_short = var.service_short + environment = var.environment + gcp_dataset = "events_${var.config}" + gcp_project_number = 385922361840 +} +``` + +### Create new dataset and events table +Use for a new environment. To get the values for `gcp_taxonomy_id` and `gcp_policy_tag_id` see [Taxonomy and policy tag](#taxonomy-and-policy-tag). +```hcl +module "dfe_analytics" { + source = "./vendor/modules/dfe-terraform-modules//aks/dfe_analytics" + + azure_resource_prefix = var.azure_resource_prefix + cluster = var.cluster + namespace = var.namespace + service_short = var.service_short + environment = var.environment + gcp_keyring = "afqts-key-ring" + gcp_key = "afqts-key" + gcp_project_id = "apply-for-qts-in-england" + gcp_project_number = 385922361840 + gcp_taxonomy_id = 5456044749211275650 + gcp_policy_tag_id = 2399328962407973209 +} +``` + +### Configure application +#### Enable in Ruby +```ruby +DfE::Analytics.configure do |config| +... + config.azure_federated_auth = ENV.include? "GOOGLE_CLOUD_CREDENTIALS" +end +``` + +#### Enable in .NET +TBD + +#### Variables +Each variable is available as a separate output. For convenience, the `variables_map` output provides them all: +- BIGQUERY_PROJECT_ID +- BIGQUERY_TABLE_NAME +- BIGQUERY_DATASET +- GOOGLE_CLOUD_CREDENTIALS + +```hcl +module "application_configuration" { + source = "./vendor/modules/dfe-terraform-modules//aks/application_configuration" + ... + secret_variables = merge( + module.dfe_analytics.variables_map, + { + ... + } + ) +} +``` + +#### Enable on each app that requires it +```hcl +module "worker_application" { + source = "./vendor/modules/dfe-terraform-modules//aks/application" + ... + enable_gcp_wif = true +} +``` + +## Authentication - Command line +The user should have Owner role on the Google project. + +- Run `gcloud auth application-default login` +- Run terraform + +## Authentication - Github actions +We set up workfload identity federation on the Google side and configure the workflow. The user should have Owner role on the Google project. This is done once per repository. + +- Run the `autorise_workflow.sh` located in *aks/dfe_analytics*: + ``` + ./authorise_workflow.sh PROJECT_ID REPO + ``` + Example: + ``` + ./authorise_workflow.sh apply-for-qts-in-england apply-for-qualified-teacher-status + ``` +- The script shows the *permissions* and *google-github-actions/auth step* to add to the workflow job +- Adding the permission removes the [default token permissions](https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#permissions-for-the-github_token), which may be an issue for some actions that rely on them. For example, the [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) action requires `pull-requests: write`. It must then be added explicitly. +- Run the workflow + +## Taxonomy and policy tag +The user should have Owner role on the Google project. + +- Authenticate: `gcloud auth application-default login` +- Get projects list: `gcloud projects list` +- Select project e.g.: `gcloud config set project apply-for-qts-in-england` +- Get taxonomies list: + ``` + gcloud data-catalog taxonomies list --location=europe-west2 --format="value(name)"` + ``` + The path contains the taxonomy id as a number e.g. 5456044749211275650 +- Get policy tags e.g.: + ``` + gcloud data-catalog taxonomies policy-tags list --taxonomy="projects/apply-for-qts-in-england/locations/europe-west2/taxonomies/5456044749211275650" --location="europe-west2" --filter="displayName:hidden" --format="value(name)"` + ``` + The path contains the policy tag id as a number e.g. 2399328962407973209 diff --git a/aks/dfe_analytics/tfdocs.md b/aks/dfe_analytics/tfdocs.md new file mode 100644 index 0000000..81ec5df --- /dev/null +++ b/aks/dfe_analytics/tfdocs.md @@ -0,0 +1,62 @@ +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >=1.4 | +| [azurerm](#requirement\_azurerm) | >=3 | +| [google](#requirement\_google) | 6.6.0 | + +## Providers + +| Name | Version | +|------|---------| +| [azurerm](#provider\_azurerm) | 4.5.0 | +| [google](#provider\_google) | 6.6.0 | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [cluster\_data](#module\_cluster\_data) | ../cluster_data | n/a | + +## Resources + +| Name | Type | +|------|------| +| [google_bigquery_dataset.main](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/bigquery_dataset) | resource | +| [google_bigquery_dataset_iam_member.appender](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/bigquery_dataset_iam_member) | resource | +| [google_bigquery_table.events](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/bigquery_table) | resource | +| [google_service_account.appender](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/service_account) | resource | +| [google_service_account_iam_binding.appender](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/service_account_iam_binding) | resource | +| [azurerm_client_config.current](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config) | data source | +| [azurerm_user_assigned_identity.gcp_wif](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/user_assigned_identity) | data source | +| [google_kms_crypto_key.main](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/data-sources/kms_crypto_key) | data source | +| [google_kms_key_ring.main](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/data-sources/kms_key_ring) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [azure\_resource\_prefix](#input\_azure\_resource\_prefix) | Prefix of Azure resources for the service. Required | `string` | n/a | yes | +| [cluster](#input\_cluster) | AKS cluster name e.g. test, production... Required | `string` | n/a | yes | +| [environment](#input\_environment) | Service environment name e.g. production, test, pr-1234... Required | `string` | n/a | yes | +| [gcp\_dataset](#input\_gcp\_dataset) | Name of an existing dataset. Optional: if not provided, create a new dataset | `string` | `null` | no | +| [gcp\_key](#input\_gcp\_key) | Name of an existing customer-managed encryption key (CMEK). Required when creating the dataset | `string` | `null` | no | +| [gcp\_keyring](#input\_gcp\_keyring) | Name of an existing keyring. Required when creating the dataset | `string` | `null` | no | +| [gcp\_policy\_tag\_id](#input\_gcp\_policy\_tag\_id) | Policy tag ID. Required when creating the dataset | `number` | `null` | no | +| [gcp\_project\_id](#input\_gcp\_project\_id) | ID of the Google cloud project e.g. 'rugged-abacus-218110', 'apply-for-qts-in-england'... Required when creating the dataset | `string` | `null` | no | +| [gcp\_project\_number](#input\_gcp\_project\_number) | Google cloud project number. Required | `number` | n/a | yes | +| [gcp\_table\_deletion\_protection](#input\_gcp\_table\_deletion\_protection) | Prevents deletion of the event table. Default: true | `bool` | `true` | no | +| [gcp\_taxonomy\_id](#input\_gcp\_taxonomy\_id) | Policy tags taxonomy ID. Required when creating the dataset | `number` | `null` | no | +| [namespace](#input\_namespace) | AKS Namespace where the service is deployed to. Required | `string` | n/a | yes | +| [service\_short](#input\_service\_short) | Short name for the service e.g. att, aytq... Required | `string` | n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| [bigquery\_dataset](#output\_bigquery\_dataset) | Bigquery dataset name | +| [bigquery\_project\_id](#output\_bigquery\_project\_id) | ID of the Google cloud project e.g. 'rugged-abacus-218110', 'apply-for-qts-in-england'... | +| [bigquery\_table\_name](#output\_bigquery\_table\_name) | Biquery events table name | +| [google\_cloud\_credentials](#output\_google\_cloud\_credentials) | Credentials for Google workload identity federation | +| [variables\_map](#output\_variables\_map) | Map of environment variables required for dfe-analytics. Merge with application configuration secrets. |