Skip to content

Commit

Permalink
dfe_anlytics README and tfdocs
Browse files Browse the repository at this point in the history
  • Loading branch information
@saliceti
saliceti committed Oct 20, 2024
1 parent 3ef56f8 commit 8e91b4c
Show file tree
Hide file tree
Showing 2 changed files with 180 additions and 0 deletions.
118 changes: 118 additions & 0 deletions aks/dfe_analytics/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# DfE Analytics
Create resources in Google cloud Bigquery and provides the required variables to applications so they can send events.

## Examples
### Reuse existing dataset and events table

```hcl
module "dfe_analytics" {
source = "./vendor/modules/dfe-terraform-modules//aks/dfe_analytics"
azure_resource_prefix = var.azure_resource_prefix
cluster = var.cluster
namespace = var.namespace
service_short = var.service_short
environment = var.environment
gcp_dataset = "events_${var.config}"
gcp_project_number = 385922361840
}
```

### Create new dataset and events table
Use for a new environment. To get the values for `gcp_taxonomy_id` and `gcp_policy_tag_id` see [Taxonomy and policy tag](#taxonomy-and-policy-tag).
```hcl
module "dfe_analytics" {
source = "./vendor/modules/dfe-terraform-modules//aks/dfe_analytics"
azure_resource_prefix = var.azure_resource_prefix
cluster = var.cluster
namespace = var.namespace
service_short = var.service_short
environment = var.environment
gcp_keyring = "afqts-key-ring"
gcp_key = "afqts-key"
gcp_project_id = "apply-for-qts-in-england"
gcp_project_number = 385922361840
gcp_taxonomy_id = 5456044749211275650
gcp_policy_tag_id = 2399328962407973209
}
```

### Configure application
#### Enable in Ruby
```ruby
DfE::Analytics.configure do |config|
...
config.azure_federated_auth = ENV.include? "GOOGLE_CLOUD_CREDENTIALS"
end
```

#### Enable in .NET
TBD

#### Variables
Each variable is available as a separate output. For convenience, the `variables_map` output provides them all:
- BIGQUERY_PROJECT_ID
- BIGQUERY_TABLE_NAME
- BIGQUERY_DATASET
- GOOGLE_CLOUD_CREDENTIALS

```hcl
module "application_configuration" {
source = "./vendor/modules/dfe-terraform-modules//aks/application_configuration"
...
secret_variables = merge(
module.dfe_analytics.variables_map,
{
...
}
)
}
```

#### Enable on each app that requires it
```hcl
module "worker_application" {
source = "./vendor/modules/dfe-terraform-modules//aks/application"
...
enable_gcp_wif = true
}
```

## Authentication - Command line
The user should have Owner role on the Google project.

- Run `gcloud auth application-default login`
- Run terraform

## Authentication - Github actions
We set up workfload identity federation on the Google side and configure the workflow. The user should have Owner role on the Google project. This is done once per repository.

- Run the `autorise_workflow.sh` located in *aks/dfe_analytics*:
```
./authorise_workflow.sh PROJECT_ID REPO
```
Example:
```
./authorise_workflow.sh apply-for-qts-in-england apply-for-qualified-teacher-status
```
- The script shows the *permissions* and *google-github-actions/auth step* to add to the workflow job
- Adding the permission removes the [default token permissions](https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#permissions-for-the-github_token), which may be an issue for some actions that rely on them. For example, the [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) action requires `pull-requests: write`. It must then be added explicitly.
- Run the workflow

## Taxonomy and policy tag
The user should have Owner role on the Google project.

- Authenticate: `gcloud auth application-default login`
- Get projects list: `gcloud projects list`
- Select project e.g.: `gcloud config set project apply-for-qts-in-england`
- Get taxonomies list:
```
gcloud data-catalog taxonomies list --location=europe-west2 --format="value(name)"`
```
The path contains the taxonomy id as a number e.g. 5456044749211275650
- Get policy tags e.g.:
```
gcloud data-catalog taxonomies policy-tags list --taxonomy="projects/apply-for-qts-in-england/locations/europe-west2/taxonomies/5456044749211275650" --location="europe-west2" --filter="displayName:hidden" --format="value(name)"`
```
The path contains the policy tag id as a number e.g. 2399328962407973209
62 changes: 62 additions & 0 deletions aks/dfe_analytics/tfdocs.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >=1.4 |
| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | >=3 |
| <a name="requirement_google"></a> [google](#requirement\_google) | 6.6.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | 4.5.0 |
| <a name="provider_google"></a> [google](#provider\_google) | 6.6.0 |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_cluster_data"></a> [cluster\_data](#module\_cluster\_data) | ../cluster_data | n/a |

## Resources

| Name | Type |
|------|------|
| [google_bigquery_dataset.main](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/bigquery_dataset) | resource |
| [google_bigquery_dataset_iam_member.appender](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/bigquery_dataset_iam_member) | resource |
| [google_bigquery_table.events](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/bigquery_table) | resource |
| [google_service_account.appender](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/service_account) | resource |
| [google_service_account_iam_binding.appender](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/resources/service_account_iam_binding) | resource |
| [azurerm_client_config.current](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config) | data source |
| [azurerm_user_assigned_identity.gcp_wif](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/user_assigned_identity) | data source |
| [google_kms_crypto_key.main](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/data-sources/kms_crypto_key) | data source |
| [google_kms_key_ring.main](https://registry.terraform.io/providers/hashicorp/google/6.6.0/docs/data-sources/kms_key_ring) | data source |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_azure_resource_prefix"></a> [azure\_resource\_prefix](#input\_azure\_resource\_prefix) | Prefix of Azure resources for the service. Required | `string` | n/a | yes |
| <a name="input_cluster"></a> [cluster](#input\_cluster) | AKS cluster name e.g. test, production... Required | `string` | n/a | yes |
| <a name="input_environment"></a> [environment](#input\_environment) | Service environment name e.g. production, test, pr-1234... Required | `string` | n/a | yes |
| <a name="input_gcp_dataset"></a> [gcp\_dataset](#input\_gcp\_dataset) | Name of an existing dataset. Optional: if not provided, create a new dataset | `string` | `null` | no |
| <a name="input_gcp_key"></a> [gcp\_key](#input\_gcp\_key) | Name of an existing customer-managed encryption key (CMEK). Required when creating the dataset | `string` | `null` | no |
| <a name="input_gcp_keyring"></a> [gcp\_keyring](#input\_gcp\_keyring) | Name of an existing keyring. Required when creating the dataset | `string` | `null` | no |
| <a name="input_gcp_policy_tag_id"></a> [gcp\_policy\_tag\_id](#input\_gcp\_policy\_tag\_id) | Policy tag ID. Required when creating the dataset | `number` | `null` | no |
| <a name="input_gcp_project_id"></a> [gcp\_project\_id](#input\_gcp\_project\_id) | ID of the Google cloud project e.g. 'rugged-abacus-218110', 'apply-for-qts-in-england'... Required when creating the dataset | `string` | `null` | no |
| <a name="input_gcp_project_number"></a> [gcp\_project\_number](#input\_gcp\_project\_number) | Google cloud project number. Required | `number` | n/a | yes |
| <a name="input_gcp_table_deletion_protection"></a> [gcp\_table\_deletion\_protection](#input\_gcp\_table\_deletion\_protection) | Prevents deletion of the event table. Default: true | `bool` | `true` | no |
| <a name="input_gcp_taxonomy_id"></a> [gcp\_taxonomy\_id](#input\_gcp\_taxonomy\_id) | Policy tags taxonomy ID. Required when creating the dataset | `number` | `null` | no |
| <a name="input_namespace"></a> [namespace](#input\_namespace) | AKS Namespace where the service is deployed to. Required | `string` | n/a | yes |
| <a name="input_service_short"></a> [service\_short](#input\_service\_short) | Short name for the service e.g. att, aytq... Required | `string` | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_bigquery_dataset"></a> [bigquery\_dataset](#output\_bigquery\_dataset) | Bigquery dataset name |
| <a name="output_bigquery_project_id"></a> [bigquery\_project\_id](#output\_bigquery\_project\_id) | ID of the Google cloud project e.g. 'rugged-abacus-218110', 'apply-for-qts-in-england'... |
| <a name="output_bigquery_table_name"></a> [bigquery\_table\_name](#output\_bigquery\_table\_name) | Biquery events table name |
| <a name="output_google_cloud_credentials"></a> [google\_cloud\_credentials](#output\_google\_cloud\_credentials) | Credentials for Google workload identity federation |
| <a name="output_variables_map"></a> [variables\_map](#output\_variables\_map) | Map of environment variables required for dfe-analytics. Merge with application configuration secrets. |

0 comments on commit 8e91b4c

Please sign in to comment.