chore(build): switch to github workflows for robot stack CI #6632

sfoster1 · 2020-09-24T17:49:45Z

This PR switches robot stack check, test, build, and deploy workflows to Github Actions, implementing the CI tasks that were not previously covered in other PRs. This includes

Code for getting human-friendly ref slugs (e.g. chore_robot-stack-gh-workflows instead of refs/heads/chore_robot-stack-gh-workflows) is moved to an action
- it uses js instead of sed, because, well, come on now
New workflow for testing, building, and deploying the runapp, cross-platform where necessary
- This workflow is triggered by changes to what I understand of the runapp's dependencies; please verify
- Signing and notarization is implemented
- The new build workflows don't put the api wheel in the app build anymore because it is not used at all
Workflow for api and shared-data linting and testing now deploys as well to the appropriate pypi repository
- This is implemented with a new action mostly because the makefile invocation was really long, and accepting upload failures on test pypi only was a pain
📣 Appveyor CI configuration completely removed 📣
📣 robot stack travis configuration for python lint/test and app build/deploy removed 📣

Please look it over and poke holes in it.

This DO NOT MERGE tag is here to remind us that we SHOULD NOT MERGE THIS until we switch the app deploy S3 prefix back to the standard one.

Add github workflow definitions for all tasks related to robot stack checks, tests, builds, and deploys that were not previously configured. Specifically, - The api-lint-test and shared-data-lint-test workflows now deploy to pypi when necessary - New workflow js-checks run javascript check actions on any javascript change, since the checkers do not reasonably work on project subsets - New workflow app-test-build runs js tests for the app and its dependencies, cross-platform when necessary; it also builds the app, with full signing support, and deploys it as travis does - New actions for deploying to pypi and setting up environment variables

and appveyor That means removing appveyor entirely :O

codecov · 2020-09-24T17:49:50Z

Codecov Report

❗ No coverage uploaded for pull request base (edge@6d3bcf8). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             edge    #6632   +/-   ##
=======================================
  Coverage        ?   73.78%           
=======================================
  Files           ?       10           
  Lines           ?      759           
  Branches        ?        0           
=======================================
  Hits            ?      560           
  Misses          ?      199           
  Partials        ?        0

Impacted Files	Coverage Δ
update-server/otupdate/buildroot/update.py	`83.33% <0.00%> (ø)`
update-server/otupdate/buildroot/config.py	`92.50% <0.00%> (ø)`
update-server/otupdate/buildroot/__main__.py	`0.00% <0.00%> (ø)`
...te-server/otupdate/buildroot/ssh_key_management.py	`78.40% <0.00%> (ø)`
update-server/otupdate/buildroot/constants.py	`100.00% <0.00%> (ø)`
...pdate-server/otupdate/buildroot/name_management.py	`29.50% <0.00%> (ø)`
update-server/otupdate/buildroot/update_session.py	`95.06% <0.00%> (ø)`
update-server/otupdate/buildroot/control.py	`91.30% <0.00%> (ø)`
update-server/otupdate/buildroot/file_actions.py	`91.66% <0.00%> (ø)`
update-server/otupdate/buildroot/__init__.py	`85.29% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d3bcf8...2a915f5. Read the comment docs.

This lets us avoid checking in a smooshed-together (Technical term) output and instead take advantage of the github core packages installed by default on the runner.

shlokamin · 2020-09-28T19:43:19Z

.github/workflows/app-test-build.yaml

+      - name: 'set complex environment variables and get yarn cache'
+        id: 'set-vars-get-cache'
+        uses: actions/github-script@v2
+        with:
+          script: |
+            const { buildComplexEnvVars, findYarnCacheDir } = require(`${process.env.GITHUB_WORKSPACE}/.github/workflows/utils.js`)
+            buildComplexEnvVars(core, context)
+            findYarnCacheDir(core, context)
+      - name: 'cache yarn cache'
+        uses: actions/cache@v2
+        with:
+          path: ${{ steps.set-vars-get-cache.outputs.yarnCacheDir }}
+          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-yarn-


Out of curiosity, instead of grabbing the global yarn cache from your util, why not just have a step that sets the cache directory like here?

Also, I don't see where behavior changes when there's a cache hit, you're still going to do a yarn from inside of setup-js even if there's a cache hit right?

Yeah but that should prevent yarn from redownloading everything shouldn't it?

In terms of why put it in a file, it accomplishes essentially the same thing and the file really had to be there for the custom env vars so I built it in anyway

Yeah but that should prevent yarn from redownloading everything shouldn't it?

Ah yeah that makes sense. Is it working as expected now? Speaking of which, how are you validating that this works? I assume just looking through the action logs and making sure yarn doesn't redownload everything?

Nvm, can see it's working now. Nice! https://github.com/Opentrons/opentrons/runs/1193602613?check_suite_focus=true

shlokamin · 2020-10-01T22:57:04Z

It looks like these lines are being copied and pasted a few times (I'm going to be using them too in my e2e PR). Maybe it's worth having this be its own task or something?

- name: 'set complex environment variables and get yarn cache'
        id: 'set-vars-get-cache'
        uses: actions/github-script@v3
        with:
          script: |
            const { buildComplexEnvVars, findYarnCacheDir } = require(`${process.env.GITHUB_WORKSPACE}/.github/workflows/utils.js`)
            buildComplexEnvVars(core, context)
            findYarnCacheDir(core, context)
      - name: 'cache yarn cache'
        uses: actions/cache@v2
        with:
          path: ${{ steps.set-vars-get-cache.outputs.yarnCacheDir }}
          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

mcous

This is looking really cool, and thanks for waiting for me to get this review in. I've got some questions in comments, as well as one extra one:

Should .github/workflows/api-lint-test.yaml be renamed to reflect that it is also deploying?

mcous · 2020-10-02T19:30:01Z

.github/actions/webstack/deploy-to-sandbox/action.yaml

-        SHORTSLUG=`echo ${{ github.ref }} | sed -e "s/^\/\?refs\/\(?:tags\|heads\)\///"`
-        echo "Found ref slug ${SHORTSLUG} from ref ${{ github.ref }}"
-        aws s3 sync ${{ inputs.distPath }} s3://sandbox.${{ inputs.domain }}/${SHORTSLUG} --acl=public-read
+        aws s3 sync ${{ inputs.distPath }} s3://sandbox.${{ inputs.domain }}/${{ inputs.destPrefix }} --acl=public-read


Prior to switching PD, we will need to confirm that this is successfully uploading dotfiles in distPath to S3, which is behavior PD relies on

.github/workflows/api-lint-test.yaml

.github/workflows/app-test-build.yaml

.github/workflows/shared-data-lint-test.yaml

.github/workflows/utils.js

.github/workflows/app-test-build.yaml

Also cache npm prebuilds

They should now require tests to pass. The app ones also don't rebuild but instead pull the prebuilt apps.

mcous

sfoster1 added 3 commits September 24, 2020 13:40

remove app and python builds as well as all python checks from travis

917b176

and appveyor That means removing appveyor entirely :O

add yarn cache for checks

3f057b2

sfoster1 added the chore label Sep 24, 2020

sfoster1 requested review from a team and mcous and removed request for a team September 24, 2020 17:49

need to run make format now that we're chekcing

ad991d4

sfoster1 requested a review from a team as a code owner September 24, 2020 18:52

don't use a dev tag on wheels for real pypi

23bf57f

sfoster1 added DO NOT MERGE Indicates a PR should not be merged, even if there's a shiny green merge button available ready for review labels Sep 24, 2020

sfoster1 added 5 commits September 25, 2020 09:00

move env var generation to checked-out script instead of action

f3d5806

This lets us avoid checking in a smooshed-together (Technical term) output and instead take advantage of the github core packages installed by default on the runner.

actually check in the js file

b732edb

fix typo and incur rebuild on utils change

d570cd9

fix format

b1a2dc0

had one more usage of the old action

d8de51a

shlokamin reviewed Sep 28, 2020

View reviewed changes

sfoster1 added 3 commits September 30, 2020 17:12

switch to real app deploy bucket

a0cf969

Every damn time

f1fac3e

actions v3 may be needed for run number

11986c7

sfoster1 removed the DO NOT MERGE Indicates a PR should not be merged, even if there's a shiny green merge button available label Oct 1, 2020

mcous reviewed Oct 2, 2020

View reviewed changes

sfoster1 added 3 commits October 2, 2020 16:06

always upload coverage

0569a62

trim

d7de48a

specify yarn cache instead of discovering it

c414d1c

Also cache npm prebuilds

sfoster1 added 5 commits October 2, 2020 16:38

separate deploy workflows

69a5f10

They should now require tests to pass. The app ones also don't rebuild but instead pull the prebuilt apps.

fix some syntax and rename app workflow

3820afb

remmove duplicate name

c8972d5

sure hope that "may omit" really means may

aefa5bf

app prefix back to builds

2a915f5

mcous approved these changes Oct 2, 2020

View reviewed changes

sfoster1 merged commit b80706a into edge Oct 2, 2020

sfoster1 deleted the chore_robot-stack-gh-workflows branch October 2, 2020 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(build): switch to github workflows for robot stack CI #6632

chore(build): switch to github workflows for robot stack CI #6632

sfoster1 commented Sep 24, 2020 •

edited

Loading

codecov bot commented Sep 24, 2020 •

edited

Loading

shlokamin Sep 28, 2020 •

edited

Loading

sfoster1 Sep 30, 2020

shlokamin Oct 1, 2020

shlokamin Oct 1, 2020

shlokamin commented Oct 1, 2020

mcous left a comment

mcous Oct 2, 2020

mcous left a comment

chore(build): switch to github workflows for robot stack CI #6632

chore(build): switch to github workflows for robot stack CI #6632

Conversation

sfoster1 commented Sep 24, 2020 • edited Loading

codecov bot commented Sep 24, 2020 • edited Loading

Codecov Report

shlokamin Sep 28, 2020 • edited Loading

Choose a reason for hiding this comment

sfoster1 Sep 30, 2020

Choose a reason for hiding this comment

shlokamin Oct 1, 2020

Choose a reason for hiding this comment

shlokamin Oct 1, 2020

Choose a reason for hiding this comment

shlokamin commented Oct 1, 2020

mcous left a comment

Choose a reason for hiding this comment

mcous Oct 2, 2020

Choose a reason for hiding this comment

mcous left a comment

Choose a reason for hiding this comment

sfoster1 commented Sep 24, 2020 •

edited

Loading

codecov bot commented Sep 24, 2020 •

edited

Loading

shlokamin Sep 28, 2020 •

edited

Loading