Refresh project: Update dependencies, update node support, add CI, fix duplex streams

[jwhitlock]

Updating this code base for node v16, and to fix issue #34. This involved changing a lot of things while I experimented to get the tests working:

• Use hijacking, a feature introduced in Docker engine v1.22, released with Docker 1.10 (February 2016). This allows the socket connection to better support stdin, stdout, and stderr streams, including closing a non-tty stdin stream (also see dockerode: Streams goodness). This is what fixed issue "docker exec wc -c" test fails with Node.js v16.13.2 #34, which blocked stdout in node v14 and earlier, and hung in node v16 and later.
• When shutting down the server, use socket.terminate() on the WebSocket, which will call socket.destroy() on the underlying Node socket.
• Fix the test setup / teardown, so that tests exit without mocha --exit (thanks to Stack Overflow commenter Martin P for suggesting why-is-node-running)
• Update new Buffer(<array or size>) (deprecation warning) to Buffer.from(array) and Buffer.alloc(size).
• Drop node v8, require v12. Tests continue to pass in node v10, so keeping it in the test matrix for now.
• Set debug prefixes from docker-exec-server to docker-exec-websocket-server
• Update to the latest versions of all dependencies. This includes:
  • Update lodash (from PR Bump lodash from 4.17.19 to 4.17.21 #31)
  • Update ws (from PR Bump ws from 7.1.2 to 7.4.6 #32), and then to 8.4.2
  • Update glob-parent (from PR Bump glob-parent from 5.0.0 to 5.1.2 #33)
  • Update es-lint to 8.8, upgrade .eslintrc, and fix issues
  • Update mocha to 9.2.0, migrate settings from mocha.opts to package.json
• Use community-tc for continuous integration testing. It tests node v10, v12, v14, and v16, as well as the "current" version. It requires a generic worker that is setup to run Docker. It uses docker-compose, which it downloads and installs.
• Remove tests/runtests.sh, since the test setup creates its own ubuntu container for testing.
• Remove TravisCI and Vagrant

[jwhitlock]

No CI ran when opening this PR.

I'm removing two repo hooks:

• The TravisCI hook
• A hook to waffle.io, which was shut down in March 2019.

[jwhitlock]

Added the Community-TC Integration app to this repo (and -client) in the Taskcluster organization settings

[community-tc-integration]

Uh oh! Looks like an error! Details

bad indentation of a mapping entry (45:94)

42 | ...
43 | ...
44 | ...
45 | ... val: 'env.image.split(":")[1]'}
-----------------------------------------^
46 | ...

[community-tc-integration]

Uh oh! Looks like an error! Details

missed comma between flow collection entries (41:44)

38 | ... RunTime: 3600
39 | ... mand:
40 | ... [git, --version]
41 | ... [git, clone, --no-progress, ${repository}, repo]
-----------------------------------------^
42 | ... [cd, repo]
43 | ... [git, config, advice.detachedHead, false]

[jwhitlock]

I've fixed the failing test. The test wasn't actually working before either - the exit code of 0 was being read, but the output of wc -c was not. A stronger version of the test now passes.

I'll summarize the changes in the PR summary.

[jwhitlock]

Because we're updating node versions, I think the next release is a major version bump to 3.0.0. I can include that version bump in this PR.

[djmitche]

Wow, @jwhitlock, I'm impressed! I looked through all of the commits here, in light of the PR comments, and it all checks out. Nice work!

[jwhitlock]

Thanks! This is a week or so of effort, and changes more than I'd like in a PR. I'd be willing to break it up into smaller bits for review. I'd also like @petemoore to look at .taskcluster.yml and see if there are any bits he'd want changed first.

[lotas]

nice 👍

[matt-boris]

Awesome! Thanks John. Not sure on the .taskcluster.yml file config, but everything else all looks great!

[petemoore]

This looks truly amazing, thank you so much for this mammoth effort!

Thanks for migrating from Vagrant to docker-compose, that seems like a much more modern and maintainable solution. I see this uses v1 rather than v2 (which I think are incompatible?). Is there a reason for this choice, and if so, is it worth adding a note about why?

[petemoore]

If you want to lose all the && references, you can format it with -| and just have line returns, see e.g. this yaml extract.

[petemoore]

Is it worth having retries here to protect against network blips?

One option to do that would be to use the mounts feature which already uses an exponential backoff algorithm (that doesn't impact maxRunTime when there are delays), e.g. have the following inside the task payload (i.e. sibling of maxruntime, env, command):

          mounts:
            - file: bin/docker-compose
              content:
                url: https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
                sha256: f3f10cf3dbb8107e9ba2ea5f23c1d2159ff7321d16f0a23051d68d8e2547b323

Note, the sha256 is optional, but I see that it is provided in the release, so might be a nice extra to have to protect against man-in-the-middle modifications. Another advantage (of using the mounts feature) is that the download also gets cached on the worker, so if another task comes in for the same worker that requires the same file, it would get reused.

[community-tc-integration]

Uh oh! Looks like an error! Details

InterpreterError at template.tasks.payload.command[0][2]: unknown context value USER

[community-tc-integration]

Uh oh! Looks like an error! Details

InterpreterError at template.tasks.payload.command[0][2]: unknown context value USER

[community-tc-integration]

Uh oh! Looks like an error! Details

InterpreterError at template.tasks.payload.command[0][2]: unknown context value USER

[jwhitlock]

Thanks @petemoore! I applied the suggestions in the last commit:

• Use mounts to download docker-compose. It took a few iterations to determine where it was downloaded and to make it executable, but the iterations are squashed
• Switch to docker-compose v2.2.3
• Change to a list of command rather than a chain of && commands.