Add Windows platform for dcp-cli CI

[mikebaumann]

@kozbo commented on Wed Feb 27 2019

User story

As a user of the HCS DSS CLI, I want the CLI to work correctly/reliably when run on Windows platforms

This is being done now in support of the concurrent work to 1) enable paths in bundle file names, and 2) improve dcp-cli download performance/efficiency, including through the use of hard links, etc.

Definition of done

DCP CLI unit tests are automatically and successfully run on Windows, in addition to the existing Linux test platform, as part of continuous integration

---

@mikebaumann commented on Thu Feb 28 2019

The current plan consists of the following:

• Move the current dcp-cli CI setup/execution from Travis to GitLab. The existing tests and CI test execution will remain the same, including testing of the four currently tested Python versions (2.7, 3.4, 3.5, 3.6), etc.
• Enhance the pipeline above with additional job(s) to run the dcp-cli unit tests on Windows. This will at least be done for the latest supported version of Python (3.6), not sure if we want/need to test all versions of Python on both Linux and Windows. TBD, open to comment.
• Setup Terraform script for the ec2 instance that holds the gitlab-windows-runner, this should live in the allspark repo.

[xbrianh]

Some notes:

• Eventually we may be able to add a Windows node into the EKS cluster running GitLab CI/CD jobs: EKS Windows Nodes aws/containers-roadmap#69
• In the meantime, we should look into parameterizing an EC2 instance running windows, along with runner installation scripting, in the allspark repo: https://github.com/HumanCellAtlas/allspark

[amarjandu]

Notes 3/25/19 : 8:34 AM

Complete

added sections to make file for windows build

In Progress

secrets fetching mechanism

TODO

run tests via make

Awaiting

terraform resources; add runner to cli

[amarjandu]

Notes 3/26/19 : 10:07 AM

docker within windows defaults to looking at the hyper-v subnet for AWS Metadata Server, have to adjust this for each docker instance created.

Complete

ec2 + gitlab runner + docker working

In Progress

Updating gitlab to get allspark -> ec2 windows runner communication,
see https://gitlab.com/gitlab-org/gitlab-runner/issues/3755

Edi** unable to get runner/docker volume setup correctly, moving to shell testing

TODO

• get blank test from github to allspark-windows
• create docker compose images for gitlab runner w/ subnet adjustment
• Terraform ec2 runner

Awaiting

update to allspark

[amarjandu]

Notes 3/27/19 : 7:46 AM

Complete

Windows Testing is present for jobs with tag "windows" in dcp-cli, runner is not globally configured.

Unable to configure docker executor per issue with gitlab runner volume configuration, attempted manual overrides within config.toml file for gitlab for volumes (cache / build) with no success, appears that `/build/HumanCellAtlas' is being passed into docker by Allspark, this may change with future fixes to gitlab, the Windows Runner is a new feature (~1 month)

Shell Executor was used instead for just the windows portion; at a later time this should be changed to using docker containers.

Ensure To use route -p {aws metadata ip} default gateway for docker containers if/when thats implemented to get around the signature issue for AWS auth.

In Progress

TODO

Terraforming + building out docker image for current setup of the ec2 container.

Awaiting

[kislyuk]

I'll follow up on one point here (there are several other issues with this line of work, but I'll discuss them separately):

Move the current dcp-cli CI setup/execution from Travis to GitLab.

The DCP CLI CI tests that are required to pass to merge code into this repo are and should remain on Travis CI (publicly accessible on github). The only time tests should run on our private GitLab is when they require access to our infrastructure or run for a very long time (longer than Travis CI supports).

[mikebaumann]

Thanks for the clear definition/distinction of the roles of Travis CI and GitLab CI currently and going forward. This is more specific guidance than I had received from other team members when selecting the approach to Windows CI early on. It would have been helpful to have this information sooner. I have added the information above to the GitLab RunBook, please let me know if there is a better location for this information.

Travis CI relatively recently added early/limited support for Windows builds, we will look into using that. It currently does have a number of limitations, and Python is not yet an officially supported language, yet it looks like there may be enough functionality and documentation available to meet the current dcp-cli needs:

https://docs.travis-ci.com/user/reference/windows/
https://docs.travis-ci.com/user/multi-os/

[amarjandu]

Notes 4/3/19 : 9:58 AM

Complete

Refactored Travis YML to create more organized structure
Windows workaround with using bash/sh as default language, then installing python within the build is not working correctly; known issue with travis and secure strings that are used to protect passwords // settings causes the output for powershell or choco to be hidden, which causes builds to no operate correctly
Unable to use powershell to get around choco package manager for windows, issue with secure also manifests here.

In Progress

NA

TODO

Need to talk to @mikebaumann about where to drive this, potentially sticking with gitlab for 'windows features'

[amarjandu]

Some tests are being skipped in Windows and need to be refactored accordingly, see #350