ci: schedule a Docker deployment test #4133

michaelfig · 2021-11-29T22:04:20Z

Description

Have CI bootstrap a network with the packages/deployment machinery. This should help detect both obvious multinode nondeterminisms as well as test out our deployment tooling.

Security Considerations

Documentation Considerations

Testing Considerations

This PR produces the current known-failure at https://github.com/Agoric/agoric-sdk/runs/4361816825#step:7:2272

JimLarson

The shell stuff looks okay. The yaml doesn't set off alarms. You might want another pair of eyes on the JS.

JimLarson · 2021-11-30T03:00:26Z

packages/deployment/scripts/install-deps.sh

+esac
+
+TERRAFORM_RELEASE=terraform_${TERRAFORM_VERSION}_${TERRAFORM_OS}_${TERRAFORM_ARCH}
+curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/${TERRAFORM_RELEASE}.zip > terraform.zip && \


I'm not sure of the context in which this script is run, but double-check that it's okay to create potentially large files in whatever the current directory is. unzip can't extract from stdin? Lame!

Changed to use a temporary file with mktemp.

kriskowal

Glossed for bash style. I highly recommend shellcheck.

kriskowal · 2021-11-30T03:06:52Z

packages/deployment/scripts/docker-deployment.sh

@@ -0,0 +1,59 @@
+#! /bin/bash
+# Get a deployment.json for a 2-node docker setup.
+set -e


Recommending set -ueo pipefail as the standard utterance.

kriskowal · 2021-11-30T03:19:58Z

packages/deployment/scripts/docker-deployment.sh

+  "PROVIDER_NEXT_INDEX": {
+    "docker": 1
+  },
+  "NETWORK_NAME": "$NETWORK_NAME"


I distrust interpolation here on principle. If there's any chance you can rely on jq, I like the pattern of jq --arg NETWORK_NAME "$NETWORK_NAME" -n <<EOF. If you can rely on node instead...

Duly Noded.

kriskowal · 2021-11-30T03:22:35Z

packages/deployment/scripts/install-deps.sh

+esac
+
+TERRAFORM_RELEASE=terraform_${TERRAFORM_VERSION}_${TERRAFORM_OS}_${TERRAFORM_ARCH}
+curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/${TERRAFORM_RELEASE}.zip > terraform.zip && \


shellcheck would tell you to quote this so that it means the same thing if a field separator snuck into one of the variables.

I installed the Shellcheck VSCode plugin. Very nice!

kriskowal · 2021-11-30T03:23:10Z

packages/deployment/scripts/install-deps.sh

+  # Install Ansible.
+  echo 'deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main' >> /etc/apt/sources.list && \
+      apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367 && \
+      apt-get update --allow-releaseinfo-change -y && \


The \ is superfluous since the line ends with an operator.

kriskowal

The JavaScript looks good to me.

kriskowal · 2021-11-30T03:28:38Z

patches/ses+0.14.3.patch

+           // explanation.
+           // eslint-disable-next-line @endo/no-polymorphic-call
+-          console.log(`Removing ${subPath}`);
+          console.warn(`Removing ${subPath}`);


I see this. I should prioritize fixing this generally.

Yeah I'm not sure we should patch this out

The patch is an expedient to prevent actual failures when stdout is captured. When we upgrade SES and it fails to apply, we'll get a warning.

Would it fail to apply tho? The fix will not touch these console statements.

Why would stdout cause failures?

Would it fail to apply tho? The fix will not touch these console statements.

patch-packages tracks the version number of the package you're patching. It will warn when the version doesn't match the one named by the patches/*+VERSION.patch file.

Why would stdout cause failures?

./my-ses-program-running-on-node16 > ./critical-file.txt

Oops, now your critical-file.txt starts with:

Removing intrinsics.Object.hasOwn

oh you're right, these should go to stderr instead of stdout! For some reason I misread the patch, sorry.
@kriskowal Any reason these are console.log instead of console.warn in the first place?

kriskowal · 2021-11-30T03:30:04Z

packages/deployment/src/init.js

@@ -323,8 +326,7 @@ const doInit = ({
  });
  config.NETWORK_NAME = overrideNetworkName;

-  // eslint-disable-next-line no-constant-condition
-  while (true) {
+  while (!noninteractive) {


I presume this halts!

mhofman · 2021-11-30T02:44:09Z

.github/workflows/test-all-packages.yml

@@ -34,7 +34,7 @@ jobs:
    # 'yarn install' must be done at the top level, to build all the
    # cross-package symlinks
    - name: yarn install
-      run: yarn install
+      run: yarn install --frozen-lockfile


packages/deployment/scripts/integration-test.sh

mhofman · 2021-11-30T03:57:24Z

patches/ses+0.14.3.patch

+           // explanation.
+           // eslint-disable-next-line @endo/no-polymorphic-call
+-          console.log(`Removing ${subPath}`);
+          console.warn(`Removing ${subPath}`);


Yeah I'm not sure we should patch this out

mhofman · 2021-11-30T04:38:27Z

packages/deployment/scripts/integration-test.sh

+"$thisdir/setup.sh" init --noninteractive
+
+# Go ahead and bootstrap.
+exec "$thisdir/setup.sh" bootstrap ${1+"$@"}


Is there a reason to exec here?

Should the test be nice and cleanly stop the bootstrapped chain?

Those are reasonable improvements, but not necessary for the Github Actions test, so I didn't do them.

michaelfig added the deployment Chain deployment mechanism (e.g. testnet) label Nov 29, 2021

michaelfig self-assigned this Nov 29, 2021

michaelfig force-pushed the mfig-docker-deployment-test branch 12 times, most recently from a5071a3 to 2475cca Compare November 30, 2021 02:33

michaelfig changed the title ~~ci: run a simple Docker deployment test~~ ci: schedule a Docker deployment test Nov 30, 2021

michaelfig marked this pull request as ready for review November 30, 2021 02:36

JimLarson approved these changes Nov 30, 2021

View reviewed changes

kriskowal reviewed Nov 30, 2021

View reviewed changes

kriskowal approved these changes Nov 30, 2021

View reviewed changes

michaelfig added 11 commits November 29, 2021 21:33

feat(deployment): enable ag-setup-cosmos init --noninteractive

e866975

fix(deployment): properly detect faucet address in faucet-helper.sh

3b9e8b1

refactor: separate out scripts/install-deps.sh

0de794d

ci(deployment-test): run Docker local chain network deployment

7c23b16

ci: use yarn install --frozen-lockfile to vet yarn.lock

5c88f95

build(patches): remove noise on stdout from ses

ed7177b

ci(golang): download package dependencies

e3ce528

fix(deployment): accomodate $GOBIN

d75868b

ci(deployment-test): put source checkout at /usr/src/agoric-sdk

9c8c09a

ci(deployment): speed up by mounting /usr/src/agoric-sdk

d587990

fix(deployment): change token send timeout from 10m to ~3m

1791164

mhofman reviewed Nov 30, 2021

View reviewed changes

chore(deployment): spruce up scripts with some Node and ShellCheck

aa551d6

michaelfig force-pushed the mfig-docker-deployment-test branch from 2475cca to aa551d6 Compare November 30, 2021 04:12

michaelfig added the automerge:no-update (expert!) Automatically merge without updates label Nov 30, 2021

mergify bot merged commit ad91d2b into master Nov 30, 2021

mergify bot deleted the mfig-docker-deployment-test branch November 30, 2021 04:25

mhofman reviewed Nov 30, 2021

View reviewed changes

mhofman mentioned this pull request Jan 8, 2022

Add loadgen to deployment integration test #4256

Merged

mhofman mentioned this pull request Feb 15, 2022

Restart based test for divergence #4557

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: schedule a Docker deployment test #4133

ci: schedule a Docker deployment test #4133

michaelfig commented Nov 29, 2021 •

edited

Loading

JimLarson left a comment

JimLarson Nov 30, 2021

michaelfig Nov 30, 2021

kriskowal left a comment

kriskowal Nov 30, 2021

kriskowal Nov 30, 2021

michaelfig Nov 30, 2021

kriskowal Nov 30, 2021

michaelfig Nov 30, 2021

kriskowal Nov 30, 2021

kriskowal left a comment

kriskowal Nov 30, 2021

mhofman Nov 30, 2021

michaelfig Nov 30, 2021

mhofman Nov 30, 2021 •

edited

Loading

michaelfig Nov 30, 2021

mhofman Nov 30, 2021

kriskowal Nov 30, 2021

mhofman Nov 30, 2021

mhofman Nov 30, 2021

mhofman Nov 30, 2021

michaelfig Nov 30, 2021

ci: schedule a Docker deployment test #4133

ci: schedule a Docker deployment test #4133

Conversation

michaelfig commented Nov 29, 2021 • edited Loading

Description

Security Considerations

Documentation Considerations

Testing Considerations

JimLarson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kriskowal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kriskowal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhofman Nov 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelfig commented Nov 29, 2021 •

edited

Loading

mhofman Nov 30, 2021 •

edited

Loading