chore(layers): add release pipeline in GitHub Actions

[am29d]

Issue number: #1183

Summary

We have been using internal pipeline to release public layers for with powertools. Because of the initial design we could not ship public layers to all commercial regions. This PR introduces a CDK project and GitHub action workflow and we will integrate it into the release process in this repository. The goal is to deploy public layers after a release and update the documentation with the latest ARNs automatically.

Changes

The CDK project has two stacks LayerStack and CanaryStack. We use CDK customer resource capability to run a canary function after the layer deployment so we can verify that the layer works. We also need to track layer ARNs, powertools and the available region, so we can put this information into our documentation. To achieve that, there is dedicated version tracking application. After a successful deployment the canary functions sends an event with all the details. This application is maintained separately. The only information we need is the event bus ARN to send the data.

To deploy into an AWS account we assume a role by using GitHub OIDC configuration. We store the target account and the role name as secrets (though the target account will be public after the public layer is released). The OIDC configuration for target accounts will be provided, no additional work is required in this project.

Open tasks

• add secrets
• add python caching for workflow jobs

After merge

• remove single deployment manually, to have consistent ARNs across all regions
• uncomment to enable all region

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of my this change
• Changes are tested
• Changes are documented
• PR title follows conventional commit semantics

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

---

View rendered layer/README.md

[heitorlessa]

quick initial review on the Workflow with some tips

[heitorlessa]

quick typos and branding

[heitorlessa]

some comments and suggestions on the canary

[heitorlessa]

This is so so so exciting. I can't believe we're finally at the stage we will have Layers with Extras, ARM, all commercial regions, except the special ones (yet), and more importantly tightly integrated with the release process now.

Can't thank you enough!

[am29d]

I have configured the GitHub OIDC connection to AWS accounts and added the secrets used in for layer deployments.

[heitorlessa]

looks awesome! loved how reusable workflow turned out, so clean! Made some questions as I think you accidentally left a workflow dependency which might fail in the first run.

The only thing missing is the ability to add custom builds for when we introduce additional layers like Extra deps, and when we optimize boto size after our E2E are done -- we can do that later as part of introducing them.

[heitorlessa]

two final comments: workflow naming, and address a potential security vulnerability when downloading cdk.out

[heitorlessa]

I can't believe it's finally here for the next release ;) In a separate PR, we should include a new section in the MAINTAINERS playbook to explain a bit more on Lambda Layers - how to check it worked, areas in the docs to update (until we have automation), etc.