TF - further fixes needed

[devopsmatt]

Related to #17033

ERROR: (Forbidden) The user, group or application 'appid=***;oid=ac21f8dc-f2ee-4a56-bd88-51a5ae230dd8;iss=https://sts.windows.net/***/' does not have secrets get permission on key vault 'pdhdemo1-keyvaults5m;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287 Code: Forbidden Message: The user, group or application 'appid=***;oid=ac21f8dc-f2ee-4a56-bd88-51a5ae230dd8;iss=https://sts.windows.net/***/' does not have secrets get permission on key vault 'pdhdemo1-keyvaults5m;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287 Inner error: { "code": "AccessDenied" }
https://github.com/CDCgov/prime-reportstream/actions/runs/12795461674/job/35672869047#step:5:559

[emvaldes]

This is not a typo. The pdhdemo*-keyvault* keys do have some random characters appended to the end of the name.

Why?
I have no idea as it was probably some typo when they were created or they are intentionally set like that so that the same environment can have multiple keyvaults.

[emvaldes]

During the execution of Terraform destroy in the GitHub Pipeline "Destroy Demo Environment", the system lingered there for close to 30 mins.

module.database.azurerm_postgresql_database.prime_data_hub_db:
Still destroying... [id=/subscriptions/7d1e3999-6577-4cd5-b296-...
dhdemo1-pgsql/databases/prime_data_hub, 2m30s elapsed]
. . .
module.database.azurerm_postgresql_database.prime_data_hub_db:
Still destroying... [id=/subscriptions/7d1e3999-6577-4cd5-b296-...
dhdemo1-pgsql/databases/prime_data_hub, 29m10s elapsed]

Note: There are more concerning issues to address in the overall objectives of this pipeline.
It's attempting to destroy resources that have ties to other infrastructure services.

[emvaldes]

│ Warning: Resource targeting is in effect
│ 
│ You are creating a plan with the -target option, which means that the
│ result of this plan may not represent all of the changes requested by the
│ current configuration.
│ 
│ The -target option is not for routine use, and is provided only for
│ exceptional situations such as recovering from errors or mistakes, or when
│ Terraform specifically suggests to use it as part of an error message.
│
│ Warning: Applied changes may be incomplete
│ 
│ The plan was created with the -target option in effect, so some changes
│ requested in the configuration may have been ignored and the output values
│ may not be fully updated. Run the following command to verify that no other
│ changes are pending:
│     terraform plan
│ 	
│ Note that the -target option is not suitable for routine use, and is
│ provided only for exceptional situations such as recovering from errors or
│ mistakes, or when Terraform specifically suggests to use it as part of an
│ error message.
│
│ Warning: Deprecated Resource
│ 
│   with module.function_app.azurerm_function_app.admin,
│   on ../../modules/function_app/admin_app.tf line 65, in resource "azurerm_function_app" "admin":
│   65: resource "azurerm_function_app" "admin" {
│ 
│ The `azurerm_function_app` resource has been superseded by the
│ `azurerm_linux_function_app` and `azurerm_windows_function_app` resources.
│ Whilst this resource will continue to be available in the 2.x and 3.x
│ releases it is feature-frozen for compatibility purposes, will no longer
│ receive any updates and will be removed in a future major release of the
│ Azure Provider.
│ 
│ (and 2 more similar warnings elsewhere)
│
│ Error: deleting Front Door (Subscription: "***"
│ Resource Group Name: "prime-data-hub-demo1"
│ Front Door Name: "prime-data-hub-demo1"): 
performing Delete: frontdoors.FrontDoorsClient#
Delete: Failure sending request: 
StatusCode=0 
-- Original Error: Code="Conflict" 
Message="Cannot delete frontend endpoint \"demo1.prime.cdc.gov\" 
because it is still directly or indirectly (using \"afdverify\" prefix) CNAMEd to front door 
\"prime-data-hub-demo1.azurefd.net\".
Please remove the DNS CNAME records and try again."
│
│ Error: Plugin did not respond
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
│
│ Error: execution halted
│
. . .

[emvaldes]

##[debug]Waited 180044ms
##[debug]Configured wait: 180000ms
Warning: Attempt 1 failed. Reason: Timeout of 1800000ms hit
##[debug]Running command for i in {1..3}; do \
##[debug]terraform -chdir=operations/app/terraform/vars/demo destroy \
##[debug]-var-file=demo1/env.tfvars.json \
##[debug]-target=module.app_service_plan \
##[debug]-target=module.application_insights \
##[debug]-target=module.container_registry \
##[debug]-target=module.database \
##[debug]-target=module.function_app \
##[debug]-target=module.log_analytics_workspace \
##[debug]-target=module.sftp_container \
##[debug]-target=module.storage \
##[debug]-refresh=false \
##[debug]-auto-approve; 2>&1; \
##[debug]sleep 60; \
##[debug]done

[emvaldes]

az monitor diagnostic-settings delete --name 'pdhdemo1-postgres_server-diag' \
                                      --resource 'pdhdemo1-pgsql' \
                                      --resource-group prime-data-hub-demo1 \
                                      --resource-type 'microsoft.dbforpostgresql/servers' \
                                      --verbose ;

az monitor diagnostic-settings delete --name 'pdhdemo1-service_plan-diag' \
                                      --resource 'pdhdemo1-serviceplan' \
                                      --resource-group prime-data-hub-demo1 \
                                      --resource-type 'microsoft.web/serverfarms' \
                                      --verbose ;

terraform -chdir=$path destroy -var-file=$env/env.tfvars.json \
                               -target=module.log_analytics_workspace.data.azurerm_monitor_diagnostic_categories.diagnostics[\"postgres_server\"]

terraform -chdir=$path destroy -var-file=$env/env.tfvars.json \
                               -target=module.log_analytics_workspace.data.azurerm_monitor_diagnostic_categories.diagnostics[\"service_plan\"]

resources="$(
    az resource list --resource-group prime-data-hub-demo1 \
                     --query "[?contains(type, 'AlertRules') || contains(type, 'server') || contains(type, 'nsights') || contains(name, 'sftp') || contains(name, 'functionapp')].id" | jq -r '.[] | @base64'
)"

for id in $resources; do \
    echo \"$id\" | jq '@base64d' \
    | xargs az resource delete --resource-group prime-data-hub-demo1 --verbose --ids | sleep 1; \
done