have a way to test Terraform changes more safely

[afeld]

Currently, our different environments are set up as App Service slots. This means infrastructure (a.k.a. Terraform) changes outside of slots apply to all of them, meaning there isn't an easy way to test those changes beyond a plan.

I propose we use a dedicated Resource Group (maybe also a dedicated subscription?) to create a full staging environment, perhaps with some features like the Monitors disabled. We might want to go all the way and make one for dev, and get rid of our use of slots. Open to feedback here.

cc @cal-itp/cdt-devsecops

Acceptance Criteria

• Changes can be made to Terraform tested without concern that production will be impacted

Additional context

This would have helped prevent the 7/21 downtime.

[afeld]

More support for this, from the official docs:

notably, you shouldn't use a single Terraform workspace to manage everything that makes up your production or staging environment

https://developer.hashicorp.com/terraform/cloud-docs/recommended-practices/part1#one-workspace-per-environment-per-terraform-configuration

[afeld]

Have a question out to @cal-itp/cdt-devsecops about whether we need to use two Subscriptions, as that complicates the Pipeline. (From what I can tell, the service connection name is read at "compile time", which restricts what logic can be used to determine it.) Waiting on their answer before continuing, one way or another.

[afeld]

Still ongoing as we propagate the changes up.

[afeld]

The new dev and test app service instances are live and well. The custom domains (dev-benefits.calitp.org and test-benefits.calitp.org) are still pointing to the old slots via Front Door, now waiting on @cal-itp/cdt-devsecops to update those. Once that's done, we can deploy to production, which will delete the old slots etc.

cc #1170

[afeld]

The new dev and test app service instances are live and well. Next, we'll deploy to production, which will delete the old slots etc.

cc #1170

[afeld]

While we're still working on the cleanup of production, this task is technically done.