Proposal: Manage Packer build state with Terragrunt?

[joshpurvis]

One pain point I've always had when using Packer/Terraform is managing and keeping track of all the custom AMIs, especially across multiple staging/production environments and regions.

I think the current wisdom is to just use tags, but I feel there's got to be a better way. Initially I was going to start my own project to solve this problem, until I realized the easiest interface is probably a wrapper around Terraform with some sort of state. Given that this project has likely already ironed most of the inevitable hurdles faced when doing this, I figured I'd first propose my idea here.

The basic idea would be to somehow link packer *.json build files to input variables in a Terraform configuration which contains the newly generated AMI. Possibly a mapping of terraform input variables to packer json files?

terragrunt {
  packer {
    docker_ami = "path/to/docker.json"
    consul_ami = "path/to/consul.json"
    # ... etc
  }
}

With these defined, Terragrunt could first ensure that the required AMIs have been built. More specifically, when terragrunt apply is executed, it would:

1. Check a Dynamodb table to see if an AMI has already been built for the given environment/region, and that the packer files haven't changed since that build. Rebuild the AMI(s) with packer if necessary.

2. Lock/Update the Dynamodb table with the latest AMI mappings, so other team members can use the same AMIs without duplicating work / wasting cloud resources.

3. Follow through the terraform apply per usual, but also implicitly pass -var docker_ami=ami-111111 and -var consul_ami=ami-222222.

4. (Optionally) Garbage collect any AMIs that are no longer being used by active instances?

Is this something that belongs in Terragrunt? Or should I consider implementing this separately?

Side notes and related issues
It seems Terraform Enterprise used to have a related feature. They're now suggesting to attack this problem with a CI/CD tool -- maybe they removed this feature for a reason I'm not considering?https://www.terraform.io/docs/enterprise-legacy/packer/builds/index.html

Here are a few related issues I found:
hashicorp/terraform#1586
hashicorp/terraform#13664

[brikis98]

We've run across this same issue, and there are indeed a few challenges.

Check a Dynamodb table to see if an AMI has already been built for the given environment/region, and that the packer files haven't changed since that build. Rebuild the AMI(s) with packer if necessary.

This sounds simple, but there is a lot of complexity hiding here, including:

1. How do you know if the packer files haven't changed since the build? Merely looking at the modified date on the .json file isn't enough, as that file could pull in other files in the same repo, files from other repos, install dependencies from all over the web that are constantly changing (e.g., apt-get install), update the OS to latest (e.g., apt-get upgrade), and so on.

2. The Packer templates often live in separate repos, as you typically want the logic of how to build/package your applications in the same place as the applications themselves. That means Terragrunt would need all sorts of extra logic to check those repos out and you have to deal with what ref (branch, commit, tag) to use.

3. Some Packer builds depend on external steps: e.g., compile some code, put some files in a certain folder, generate self-signed TLS certs, and then run packer build.

4. Packer builds often need variables passed to them, as well as credentials. We'd have to build some mechanism to handle this. Also, a Packer template may define multiple builders, and we'd need a way to specify which one to use.

My guess is that there is enough variation in how each company does Packer builds that it's tricky to build a generic tool to support it. Hence the recommendation for a CI tool, where you can define all of these custom build steps.

[brikis98]

Whoops, didn't mean to close the issue, just clicked the wrong button. Feel free to continue the conversation here :)

[brikis98]

See also: https://github.com/juliosueiras/terraform-provider-packer

[joshpurvis]

Thanks for detailed response! While not what I was hoping to hear, this definitely gives me a bunch to think about. I suspected there was something tricky about it given it had been thought about and brought to the attention of Hashicorp at least once in the issues I listed.

I was aware of that provider, but it didn't look appealing to me because it required rewriting Packer files into Terraform, and I think it still requires you to run packer manually -- so doesn't help with the AMI problem.

Going to close this for now, since I don't have any good responses to your complexity concerns.