Skip to content
This repository has been archived by the owner on Dec 3, 2021. It is now read-only.

Commit

Permalink
WIP converting image build logic to pure GH actions
Browse files Browse the repository at this point in the history 
Signed-off-by: Matt Oswalt <[email protected]>
  • Loading branch information
@Mierdin
Mierdin committed Mar 23, 2021
1 parent dcb6a6d commit ddf774b
Show file tree
Hide file tree
Showing 4 changed files with 292 additions and 51 deletions.
13 changes: 11 additions & 2 deletions .github/workflows/generate-preview.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,23 @@ on:
branches: [master]
types:
- completed
inputs:

# TODO - note in https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/ this can be invoked
# via a webhook as well. This may be necessary to communicate the pr number as an input - you MUST have this.
#
# TODO - for security reasons (that we will absolutely want) this will run on the default branch. However we still want to check out the PR branch
# so we can build the images. So we need the PR number
pr_number:
description: 'Number of the PR to check out'
required: true

jobs:
prebuild:
runs-on: ubuntu-latest
steps:

# TODO - for security reasons (that we will absolutely want) this will run on the default branch. However we still want to check out the PR branch
# so we can build the images.


- uses: actions/checkout@v2

Expand Down
302 changes: 264 additions & 38 deletions .github/workflows/main.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@ name: CI
on:
pull_request:
branches: [ master ]
env:
emptyArray: "[]"
emptyLiteral: "empty"
retagSource: "v1.3.0"

jobs:
build:
prebuild:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
Expand All @@ -28,40 +32,262 @@ jobs:
run: ./check-changelog.sh
if: ${{ success() }}

# ---------------------------


# - name: Create Preview
# id: create_preview
# run: |
# echo ::set-output name=preview_id::$(./create-preview.sh | jq -r '.ID')
# if: ${{ success() }}

# - name: Get PR number
# id: get_pr_number
# run: |
# echo ::set-output name=pr_number::$(jq --raw-output .pull_request.number "$GITHUB_EVENT_PATH")
# if: ${{ success() }}

# - name: Build Preview Images
# id: build_preview_images
# run: |
# curl -k -X POST https://abathur.nrelabs.io/api/v1/webhooks/preview_images_build -H "St2-Api-Key: $PREVIEWER_APIK" -H "Content-Type: application/json" \
# --data "{\"preview_id\": \"$PREVIEW_ID\", \"pr_number\": \"$PREVIEW_PR_NUMBER\", \"status_commit_id\": \"$PREVIEW_STATUS_COMMIT\", \"github_token\": \"$GH_TOKEN\"}"
# env:
# PREVIEW_ID: ${{ steps.create_preview.outputs.preview_id }}
# PREVIEW_STATUS_COMMIT: ${{ github.event.pull_request.head.sha }}
# PREVIEW_PR_NUMBER: ${{ steps.get_pr_number.outputs.pr_number }}
# GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

# # Locked down account which can only invoke preview webhook
# PREVIEWER_APIK: MjE0ZTlkYWZjMDg1OTNkOWJkMjQxZDA0Mzk0NzIzNDI1MTc2Nzk0NDVkMjk0MGE5NTNhODkxOTNiMzVmNWM5Mg
# if: ${{ success() }}

# - name: Wait for status
# run: ./wait-for-status.sh ${{ github.event.pull_request.head.sha }}
# if: ${{ success() }}

# - name: Request preview
# run: ./start-preview.sh ${{ steps.create_preview.outputs.preview_id }}
# if: ${{ success() }}

# https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/

# WRITE SCRIPTS AS ACTIONS
# https://github.com/actions/checkout/issues/331
# https://github.com/actions/github-script

# EXPRESSIONS CHEAT SHEET
# https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions

# I wanted a pure "as-code" experience. No local infrastructure needed.
# The preview system was modeled after netlify and powered by github actions and some custom software

- name: Create Preview
id: create_preview
run: |
echo ::set-output name=preview_id::$(./create-preview.sh | jq -r '.ID')
- name: Create preview check
id: create_preview_check
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
var check_result = github.checks.create({
owner: context.repo.owner,
repo: context.repo.repo,
name: "nrelabs-preview",
head_sha: "${{ github.event.pull_request.head.sha }}",
status: "in_progress",
output: {
"title": "Preview is being provisioned...",
"summary": "Your content preview is currently being provisioned, please wait. Once provisioned, detailed information about how to view it will appear here.",
},
});
return check_result.id
- name: Create preview logs check
id: create_preview_logs_check
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
var check_result = github.checks.create({
owner: context.repo.owner,
repo: context.repo.repo,
name: "nrelabs-preview-logs",
head_sha: "${{ github.event.pull_request.head.sha }}",
status: "in_progress",
output: {
"title": "Preview is being provisioned...",
"summary": "Your content preview is currently being provisioned, please wait. Once provisioned, detailed logs about the infrastructure provisioned to serve this preview will be provided here.",
},
});
return check_result.id
- name: Get PR number
id: get_pr_number
run: |
echo ::set-output name=pr_number::$(jq --raw-output .pull_request.number "$GITHUB_EVENT_PATH")
if: ${{ success() }}

- name: Check out PR
run: |
hub pr checkout ${{ steps.get_pr_number.outputs.pr_number }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Get Buildables
id: get_buildables
run: |
echo ::set-output name=buildables::$(cd images && find . -maxdepth 2 -type f -name 'Makefile' -printf '%h;' | tr -d './' | rev | cut -c 2- | rev | jq -Rc 'split(";")')
- name: Get Changed
id: get_changed
run: |
echo ::set-output name=changed::$(git diff --name-only master..HEAD images/ | sed -rn 's/images\/([^/]*)\/.*/\\1/p' | tr '\n' ';' | rev | cut -c 2- | rev)
- name: Get Image Directives
uses: actions/github-script@v3
id: get_image_directives
with:
script: |
var to_retag = [
"vqfx-snap1",
"vqfx-snap2",
"vqfx-snap3",
]
var to_build = []
var buildables = ${{ steps.get_buildables.outputs.buildables || env.emptyArray }};
var changed = ${{ steps.get_changed.outputs.changed || env.emptyArray }};
buildables.forEach(function(buildable) {
if (!changed.includes(buildable)) {
console.log(buildable + " not found in changed, adding to to_retag");
to_retag.push(buildable);
} else {
console.log(buildable + " was found in changed, adding to to_build");
to_build.push(buildable);
}
});
// The strategy matrix can't handle empty arrays so we'll add this here and check for it during the retag/build jobs
if (to_build.length == 0) {
to_build.push("empty")
}
if (to_retag.length == 0) {
to_retag.push("empty")
}
return {
"to_build": to_build,
"to_retag": to_retag
}
outputs:
preview_id: ${{ steps.create_preview.outputs.preview_id }}
pr_number: ${{ steps.get_pr_number.outputs.pr_number }}
image_directives: ${{ steps.get_image_directives.outputs.result }}
preview_check_id: ${{ steps.create_preview_check.outputs.result }}
preview_logs_check_id: ${{ steps.create_preview_logs_check.outputs.result }}

logins:
needs: prebuild
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@master
with:
project_id: ${{ secrets.GCP_PROJECT_ID }}
service_account_key: ${{ secrets.GCP_SA_KEY }}
export_default_credentials: true

build_image:
needs: [prebuild, logins]
runs-on: ubuntu-latest
strategy:
matrix:
image_to_build: ${{ fromJson(needs.prebuild.outputs.image_directives).to_build }}
steps:
- uses: actions/checkout@v2
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

# - name: Set up QEMU
# uses: docker/setup-qemu-action@v1
# - name: Set up Docker Buildx
# uses: docker/setup-buildx-action@v1

- name: Build image
run: |
cd images/${{ matrix.image_to_build }} && make dockerfast TARGET_VERSION=${{ env.retagSource }}
if: ${{ matrix.image_to_build != env.emptyLiteral }} # Ugly hack to prevent this from running when the matrix array is "empty"

retag_image:
needs: [prebuild, logins]
runs-on: ubuntu-latest
strategy:
matrix:
image_to_retag: ${{ fromJson(needs.prebuild.outputs.image_directives).to_retag }}
steps:
- uses: actions/checkout@v2
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Re-tag image
run: |
docker pull antidotelabs/${{ matrix.image_to_retag }}:${{ env.retagSource }} && docker tag antidotelabs/${{ matrix.image_to_retag }}:${{ env.retagSource }} antidotelabs/${{ matrix.image_to_retag }}:preview-${{ needs.prebuild.outputs.preview_id }} && docker push antidotelabs/${{ matrix.image_to_retag }}:preview-${{ needs.prebuild.outputs.preview_id }}
if: ${{ matrix.image_to_build != env.emptyLiteral }} # Ugly hack to prevent this from running when the matrix array is "empty"

deploy_preview:
needs: [prebuild, build_image, retag_image]
runs-on: ubuntu-latest
steps:

- uses: actions/checkout@v2

- name: Request preview
run: ./start-preview.sh ${{ needs.prebuild.outputs.preview_id }}
if: ${{ success() }}

- name: Wait for status
run: ./wait-for-status.sh ${{ needs.prebuild.outputs.preview_id }}
if: ${{ success() }}

- name: Retrieve Logs
id: retrieve_preview_logs
run: |
echo ::set-output name=preview_logs::$(curl https://preview.nrelabs.io/status\?id\=${{ needs.prebuild.outputs.preview_id }} | jq -r '.PreviewLogs')
- name: Update logs check
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
summary = `Aggregated logs for the provisioning of preview infrastructure are provided below.
Note that these are only logs for the provisioning of the infrastructure necessary for your preview. For troubleshooting problems with a specific piece of
content or endpoint, your best bet is the Jaeger deployment available [here](https://inspect-preview.nrelabs.io/search?lookback=12h&operation=api_livelesson_request&service=preview-${{ needs.prebuild.outputs.preview_id }})
`;
# https://octokit.github.io/rest.js/v18#checks-update
github.checks.update({
owner: context.repo.owner,
repo: context.repo.repo,
check_run_id: "${{ needs.prebuild.outputs.preview_logs_check_id }}",
output: {
"title": "Preview Provisioning Logs",
"summary": summary,
"text": ${{ steps.retrieve_preview_logs.outputs.preview_logs }},
},
conclusion: "neutral",
});
- name: Update status check
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
summary = `Your content is ready to be previewed, and is available at the link below:
[Open Preview](https://preview-${{ needs.prebuild.outputs.preview_id }}.nrelabs.io)
Note that this is a fully-deployed version of the main NRE Labs site, but includes the changes you've made in
this branch. Once you navigate to the site above, use the lesson catalog or other site navigation to find the content
you've added/changed.
For more information, see the [NRE Labs documentation](https://docs.nrelabs.io/creating-contributing/preview-your-changes).
`;
detail_text = `Some additional tools for you to use can be found below:
- [Jaeger Traces (for troubleshooting lesson startup, etc)](https://inspect-preview.nrelabs.io/search?lookback=12h&operation=api_livelesson_request&service=preview-${{ needs.prebuild.outputs.preview_id }})
`;
# https://octokit.github.io/rest.js/v18#checks-update
github.checks.update({
owner: context.repo.owner,
repo: context.repo.repo,
check_run_id: "${{ needs.prebuild.outputs.preview_check_id }}",
output: {
"title": "Preview is Ready!",
"summary": summary,
"text": detail_text,
},
conclusion: "success",
});
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## In development

- Adding image build to preview pipeline [#352](https://github.com/nre-learning/nrelabs-curriculum/pull/352)
- More image build changes (moving to GH actions) [#354](https://github.com/nre-learning/nrelabs-curriculum/pull/354)

## v1.3.0 - December 13, 2020

Expand Down
27 changes: 16 additions & 11 deletions wait-for-status.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,30 +4,35 @@

if [ -z "$1" ]
then
echo "Must provide status commit ID as parameter to this script"
echo "Must provide preview ID as parameter to this script"
exit 1
fi

for i in {1..120}
do

# TODO - May want to consider checking for the status of the commit in general, which is always there - it's a key "state"
# at the top level of the returned object here, outside of the "statuses" array.
build_status=$(curl \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/nre-learning/nrelabs-curriculum/commits/$1/status | jq -r '.statuses[] | select(.context=="Building Endpoint Images").state')
build_status=$(curl https://preview.nrelabs.io/status\?id\=$1 | jq -r .Status)
if [[ -z "$build_status" ]]
then
echo "Error retrieving build status"
exit 1
fi

if [[ "$build_status" == "failure" ]]
if [[ "$build_status" == "FAILED" ]]
then
echo "Preview deployment failed"
exit 1
elif [[ "$build_status" == "CREATED" ]]
then
echo "Status failed"
echo "Preview deployment not started - call this script after a created preview has also been started."
exit 1
elif [[ "$build_status" == "success" ]]
elif [[ "$build_status" == "READY" ]]
then
echo "Status succeeded"
echo "Preview deployment succeeded"
exit 0
fi

echo "Sleeping for 10 seconds..."
echo "Status for preview $1 is $build_status; sleeping for 10 seconds..."
sleep 10

done
Expand Down

0 comments on commit ddf774b

Please sign in to comment.