Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(build): publish lib as a Lambda Layer #884

Merged
merged 35 commits into from
Aug 9, 2022
Merged
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
dcf724d
Add a cdk app capable of building and publishing Lambda Layer
May 19, 2022
18ed8c9
add github action for e2e
May 20, 2022
6c8939e
add local layer construct npm package
May 20, 2022
76fa1d8
disable fail fast
May 20, 2022
e063843
fix RUNTIME naming in matrix
May 20, 2022
5a884ee
fix tests
May 20, 2022
bc0a318
fix outputs export name
May 20, 2022
b057127
Add capability to pass package version to layer builder/publisher
May 20, 2022
583a12f
Add initial doc
May 20, 2022
e12dc75
Add github workflows
Jul 21, 2022
022c517
add gitignore
Jul 21, 2022
e24df14
fix install deps
Jul 22, 2022
0f50311
pass layer name
Jul 22, 2022
01b91a8
make layer public and store details in ssm
Jul 22, 2022
e286e8d
fix e2e tests
Jul 22, 2022
cdf4e1f
fix context
Jul 22, 2022
b03413e
remvove groups for tests
Jul 22, 2022
e7a254d
publish layer on release
Jul 22, 2022
44f0728
fix account in doc
Jul 22, 2022
8401da8
fix test deps
Jul 22, 2022
fb99afc
deploy to all region
Jul 22, 2022
297907e
fix account number for layer
Jul 22, 2022
2ebe192
fix unit tests
Jul 22, 2022
5d8ad97
add install for layer deps in pr workflow
Jul 22, 2022
e70e5c1
run unit test of layer publisher for supported node versions only
Jul 22, 2022
f509c6a
fix node version in doc
Jul 22, 2022
a2d1be5
add node 16 to layer e2e
Jul 22, 2022
1e58383
fix comments
Jul 22, 2022
f8fee27
fix doc
Jul 22, 2022
810e0dc
rename layer to TypeScript
Aug 4, 2022
d07ccf9
take care of comments
Aug 4, 2022
c8465e2
remove layer from doc for now
Aug 4, 2022
c2b0f1b
remove pasted commenbt
Aug 4, 2022
4a1c436
chore(build): broke up pr workflow & measure package size (#1031)
dreamorosi Aug 9, 2022
fa2f59b
Merge branch 'feat/layerPublisher' of github.com:awslabs/aws-lambda-p…
dreamorosi Aug 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/measure-packages-size.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Measure packages size

on:
workflow_dispatch:
inputs:
prNumber:
description: "PR Number"
required: true

jobs:
measure-utils-sizes:
runs-on: ubuntu-latest
env:
NODE_ENV: dev
PR_NUMBER: ${{ inputs.prNumber }}
steps:
# Since we are manually triggering the workflow the previous checkout has the main branch. In order to checkout the branch/code of the PR
# we need first to use the PR number to retrieve the PR SHA number. This means we need three steps to: checkout the repo,
# run a custom script to get the SHA, and then finally checkout the PR branch
- name: Checkout Repo
uses: actions/checkout@v3
- name: Extract PR details
id: extract_PR_details
uses: actions/github-script@v6
with:
script: |
const script = require('.github/scripts/get_pr_info.js');
await script({github, context, core});
- name: Checkout PR code
uses: actions/checkout@v3
with:
ref: ${{ steps.extract_PR_details.outputs.headSHA }}
- name: Packages size report
uses: flochaz/[email protected]
with:
build-command: mkdir dist && npm run package -w packages/logger -w packages/tracer -w packages/metrics -w packages/commons && npm run package-bundle -w packages/logger -w packages/tracer -w packages/metrics -w packages/commons && bash -c "mv ./packages/*/dist/* dist/" && ls dist
dist-directory: /dist
pr-number: ${{ inputs.prNumber }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3 changes: 3 additions & 0 deletions .github/workflows/on-merge-to-main.yml
Original file line number Diff line number Diff line change
@@ -67,6 +67,9 @@ jobs:
# the dependencies in a separate step
working-directory: ./examples/cdk
run: npm ci
- name: Install Layer publisher app
working-directory: ./layer-publisher
run: npm ci
- name: "Setup SAM"
# We use an ad-hoc action so we can specify the SAM CLI version
uses: aws-actions/setup-sam@v2
134 changes: 83 additions & 51 deletions .github/workflows/pr_lint_and_test.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
name: pr-lint-and-test
name: On PR code update

on:
pull_request:
types: [opened, synchronize]
jobs:
on_push:
run-unit-tests-on-utils:
runs-on: ubuntu-latest
env:
NODE_ENV: dev
@@ -12,63 +13,94 @@ jobs:
version: [12, 14, 16]
fail-fast: false
steps:
- uses: actions/checkout@v3
- name: "Use NodeJS"
- name: Checkout code
uses: actions/checkout@v3
- name: Setup NodeJS
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.version }}
- name: Install [email protected]
cache: "npm"
- name: Setup npm
run: npm i -g npm@next-8
- name: "Setup npm"
run: |
npm set "//registry.npmjs.org/:_authToken=${{ secrets.NPM_TOKEN }}"
- name: Install monorepo packages
# This installs all the dependencies of ./packages/*
- name: Cache node modules
id: cache-node-modules
uses: actions/cache@v3
with:
path: "./node_modules"
# Use the combo between node version, name, and SHA-256 hash of the lock file as cache key so that
# if one of them changes the cache is invalidated/discarded
key: ${{ matrix.version }}-cache-utils-node-modules-${{ hashFiles('./package-lock.json') }}
- name: Install dependencies
# We can skip the install if there was a cache hit
if: steps.cache-node-modules.outputs.cache-hit != 'true'
# See https://github.com/npm/cli/issues/4475 to see why --foreground-scripts
run: npm ci --foreground-scripts
- name: Install CDK example packages
# Since we are not managing the CDK examples with npm workspaces we install
# the dependencies in a separate step
working-directory: ./examples/cdk
run: npm ci
- name: "Setup SAM"
# We use an ad-hoc action so we can specify the SAM CLI version
uses: aws-actions/setup-sam@v2
- name: Build packages
# If there's a cache hit we still need to manually build the packages
# this would otherwise have been done automatically as a part of the
# postinstall npm hook
if: steps.cache-node-modules.outputs.cache-hit == 'true'
run: |
npm run build -w packages/commons
npm run build -w packages/logger & npm run build -w packages/tracer & npm run build -w packages/metrics
- name: Lint
run: npm run lint -w packages/commons -w packages/logger -w packages/tracer -w packages/metrics
- name: Run unit tests
run: npm t -w packages/commons -w packages/logger -w packages/tracer -w packages/metrics
check-examples:
runs-on: ubuntu-latest
env:
NODE_ENV: dev
strategy:
matrix:
example: ["sam", "cdk"]
fail-fast: false
defaults:
run:
working-directory: examples/${{ matrix.example }}
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup NodeJS
uses: actions/setup-node@v3
with:
node-version: 16
cache: "npm"
- name: Cache node modules
id: cache-node-modules
uses: actions/cache@v3
with:
version: 1.49.0
- name: Install SAM example packages
# Since we are not managing the SAM examples with npm workspaces we install
# the dependencies in a separate step
working-directory: ./examples/sam
path: "./examples/${{ matrix.example }}/node_modules"
# Use the combo between example, name, and SHA-256 hash of all example lock files as cache key.
# It's not possible to use the ${{ matrix.example }} key in the hashFiles fn so
# if any of the lock files (wich should be fairly similar anyway) changes the cache is
# invalidated/discarded for all.
key: ${{ matrix.example }}-cache-examples-node-modules-${{ hashFiles('./examples/*/package-lock.json') }}
- name: Install dependencies
run: npm ci
- name: Run lint
run: npm run lerna-lint
- name: Run tests
run: npm run lerna-test
- name: Collate Coverage Reports
if: ${{ github.actor != 'dependabot[bot]' }}
run: |
for d in ./packages/*/ ; do
mkdir -p coverage
if [[ ! -f coverage/lcov.info ]]
then
continue
fi
filename="$d""coverage/lcov.info"
targetSource="SF:""$d""src"
sed "s|SF:src|$targetSource|g" $filename >> coverage/lcov.info
done
- name: Report Coverage
#Dependabot user will only have read-only perms, so don't try to report coverage
if: ${{ github.actor != 'dependabot[bot]' }}
uses: romeovs/[email protected]
run: npm t
check-layer-publisher:
runs-on: ubuntu-latest
env:
NODE_ENV: dev
defaults:
run:
working-directory: layer-publisher
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup NodeJS
uses: actions/setup-node@v3
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
lcov-file: ./coverage/lcov.info
- name: Packages size report
uses: flochaz/[email protected]
node-version: 16
cache: "npm"
- name: Cache node modules
id: cache-node-modules
uses: actions/cache@v3
with:
build-command: mkdir dist && npm run lerna-package && npm run lerna-package-bundle && bash -c "mv ./packages/*/dist/* dist/" && ls dist
dist-directory: /dist
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
path: "./layer-publisher/node_modules"
# Use the combo between example, name, and SHA-256 hash of the layer-publisher lock files as cache key.
key: cache-layer-publisher-node-modules-${{ hashFiles('./layer-publisher/*/package-lock.json') }}
- name: Install Layer publisher app
run: npm ci
81 changes: 81 additions & 0 deletions .github/workflows/publish_layer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Deploy layer to all regions

permissions:
id-token: write
contents: read

on:
# Manual trigger
workflow_dispatch:
inputs:
latest_published_version:
description: "Latest npm published version to rebuild corresponding layer for, e.g. v1.0.2"
default: "v1.0.2"
required: true
# Automatic trigger after release
workflow_run:
workflows: ["release"]
types:
- completed

jobs:
# Build layer by running cdk synth in layer-publisher directory and uploading cdk.out for deployment
build-layer:
runs-on: ubuntu-latest
if: ${{ (github.event.workflow_run.conclusion == 'success') || (github.event_name == 'workflow_dispatch') }}
ijemmy marked this conversation as resolved.
Show resolved Hide resolved
defaults:
run:
working-directory: ./layer-publisher
steps:
- name: checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: "16.12"
- name: Set release notes tag
run: |
RELEASE_INPUT=${{ inputs.latest_published_version }}
LATEST_TAG=$(git describe --tag --abbrev=0)
RELEASE_TAG_VERSION=${RELEASE_INPUT:-$LATEST_TAG}
echo "RELEASE_TAG_VERSION=${RELEASE_TAG_VERSION:1}" >> $GITHUB_ENV
ijemmy marked this conversation as resolved.
Show resolved Hide resolved
- name: install cdk and deps
run: |
npm install -g [email protected]
cdk --version
- name: install deps
run: |
npm ci
- name: CDK build
run: cdk synth --context PowerToolsPackageVersion=$RELEASE_TAG_VERSION -o cdk.out
- name: zip output
run: zip -r cdk.out.zip cdk.out
- name: Archive CDK artifacts
uses: actions/upload-artifact@v3
with:
name: cdk-layer-artefact
path: layer-publisher/cdk.out.zip

# Deploy layer to all regions in beta account
deploy-beta:
needs:
- build-layer
uses: ./.github/workflows/reusable_deploy_layer_stack.yml
with:
stage: "BETA"
artefact-name: "cdk-layer-artefact"
secrets:
target-account-role: ${{ secrets.AWS_LAYERS_BETA_ROLE_ARN }}

# Deploy layer to all regions in prod account
deploy-prod:
needs:
- deploy-beta
uses: ./.github/workflows/reusable_deploy_layer_stack.yml
with:
stage: "PROD"
artefact-name: "cdk-layer-artefact"
secrets:
target-account-role: ${{ secrets.AWS_LAYERS_PROD_ROLE_ARN }}
81 changes: 81 additions & 0 deletions .github/workflows/reusable_deploy_layer_stack.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Deploy cdk stack

permissions:
id-token: write
contents: read

on:
workflow_call:
inputs:
stage:
required: true
type: string
artefact-name:
required: true
type: string
secrets:
target-account-role:
required: true

jobs:
deploy-cdk-stack:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./layer-publisher
strategy:
fail-fast: false
matrix:
region:
[
"af-south-1",
"eu-central-1",
"us-east-1",
"us-east-2",
"us-west-1",
"us-west-2",
"ap-east-1",
"ap-south-1",
"ap-northeast-1",
"ap-northeast-2",
"ap-southeast-1",
"ap-southeast-2",
"ca-central-1",
"eu-west-1",
"eu-west-2",
"eu-west-3",
"eu-south-1",
"eu-north-1",
"sa-east-1",
"ap-southeast-3",
"ap-northeast-3",
"me-south-1",
]
steps:
- name: checkout
uses: actions/checkout@v3
- name: aws credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: ${{ matrix.region }}
role-to-assume: ${{ secrets.target-account-role }}
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: "16.12"
- name: install cdk and deps
run: |
npm install -g [email protected]
cdk --version
- name: install deps
run: |
npm ci
- name: Download artifact
uses: actions/download-artifact@v3
with:
name: ${{ inputs.artefact-name }}
path: layer-publisher
- name: unzip artefact
run: unzip cdk.out.zip
- name: CDK Deploy Layer
run: cdk deploy --app cdk.out --context region=${{ matrix.region }} 'LayerPublisherStack' --require-approval never --verbose
ijemmy marked this conversation as resolved.
Show resolved Hide resolved
Loading