Upgrade version of tfengine

[alanhughes]

Running tfengine in CI
Rather than running tfengine and then committing the code to our repository to kick off CI jobs, we are attempting to run tfengine in CI and then running terraform immediately after. However, we currently have no way of exactly replicating the environment described in the build instructions, because version 0.4.0 of tfengine does not work properly in either the gcr.io/cloud-foundation-cicd/cft/developer-tools or hashicorp/terraform docker images:

<host>
$ docker run -it --entrypoint="" hashicorp/terraform:0.12.29 sh
<container>
/ # VERSION=v0.4.0
/ #  wget -O /usr/local/bin/tfengine https://github.com/GoogleCloudPlatform/healthcare-data-protection-suite/releases/download/${VERSION}/tfengine_${VERSION}_linux-amd64
Connecting to github.com (140.82.121.3:443)
Connecting to github-releases.githubusercontent.com (185.199.108.154:443)
saving to '/usr/local/bin/tfengine'
tfengine             100% |*****************************************************| 26.5M  0:00:00 ETA
'/usr/local/bin/tfengine' saved
/ # which tfengine
/usr/local/bin/tfengine
/ # tfengine --version
sh: tfengine: not found

We therefore have to either

• run the CI in two stages - doable, but not ideal since we have to run the tfengine stage in a different docker image then cache the results
• use a different version of tfengine - 0.7.0 seems to be the earliest that runs correctly in the docker images above

Describe the solution you'd like
Support version 0.7.0 of tfengine would be preferable of the two solutions described above. In our environment the only difference in the generated terraform is a pubsub topic resource in the firestore configuration getting renamed and thus recreated (see below). However the recreation is quick and causes minimal downtime.

Describe alternatives you've considered
See above

Additional context
Terraform plan changes when generating terraform using tfengine 0.7.0
tfengine-diff.txt

Related feature requests
The reason we run tfengine in CI is because storing the config for multiple environments doesn't seem possible at the present due to the naming conventions - the audit, devops, cicd and kubernetes folders all get duplicated when tfengine runs (other folders have <prefix>-<env> prepended to them). Would you be open to fixing this or there a reason it's set up this way?

Priority
Low/medium

Labels
Deployment

[moschetti]

The reason why tfengine with version < 0.7.0 cannot run in those containers is because they were not built statically and those containers are likely Alpine based or doesn't have a local go environment installed. Building binaries statically was supported since 0.7.0.

After discussing with the Data Protection Suite team, there would be some significant changes that would be needed to MyStudies repo to support this upgrade, and we do not have resources to do that at this time.

Potential concerns with upgrading to tfengine 0.7.0 or higher:

1. Would also need to update the templates to support any changes.
2. Will need to change the terraform version in the CICD container as well
3. Plan for upgrade users. Upgrading existing instances may cause resources to be destroyed and re-deployed. So we'd need to figure out a plan to either let existing customers stay on 0.4.0 with their existing templates, or run enough testing to be sure that it won't cause other issues.

Another option would be to find or build a container which has go, terraform, and basic bash tools installed to be able to run tfengine 0.4.0. This would be much less work, but we'd still need to scope this into existing work plans.