Epic - Improve development & testing infrastructure

[marccarre]

Summary

Re-factor the current smoke tests in order to:

• better re-use provisioning & configuration "modules" / "APIs"
• run integration tests on top of these layers
• run cross-version tests on top of these layers
• run stress tests on top of these layers
• be able to easily spawn development & manual testing environments.

This should hopefully:

• lead to a nicer development & testing experience
• boost Weave Net developer productivity
• enabler richer testing scenarios.

+------------------+ +-----------------+ +-----------------------+ +---------------------+
| make stress-test | | make cross-test | | make integration-test | | Weave Net developer |
+-------+----------+ +--------+--------+ +--------+--------------+ +--+--------------+---+
        |                     |                   |                   |              |
        | +-------------------+-----------------+ |                   |              |
        | | Cartesian-product of deps. versions | |                   |              |
        | +-------------------+-----------------+ |                   |              |
        |                      \                  |                   |              |
+-------+------+                \        +--------+----------+        |              |
| Stress Tests |                 `-------+ Integration Tests |        |              |
+-------+------+                         +--------+----------+        |              |
        |                                         |                   |              |
+-------+-----------------------------------------+-------------------+---+          |
|          Configuration (Go, Docker, K8S, etc. via Ansible)              |          |
+-------+-----------------------------------------+-------------------+---+          |
        |                                         |                   |              |
+-------+-----------------------------------------+-------------------+--------------+---+
|                Provisioning (VMs via Vagrant, AWS/GCP/DO via Terraform)                |
+----------------------------------------------------------------------------------------+

Progress

• Scripts now source dependencies using relative paths #2696 Scripts now source dependencies using relative paths
• Evaluate Docker Infrakit: does not seem appropriate, more to deploy a pod/a set of containers.
• Add new targets to Makefile
• Script to list versions of Go, Docker, Kubernetes #2664 Script to list versions of Go, Docker, Kubernetes
• Script to orchestrate integration testing
• Script to generate cross product of versions
• Ansible playbooks to configure development and testing machines #2682 Ansible script to configure test machine with Weave Net (standalone)
• Ansible playbooks to configure development and testing machines #2682 Ansible script to configure test machine with Weave Net & Kubernetes
• Terraform script to add provisioning on GCP, DO and AWS #2676 Vagrant script to add provisioning on local VM
• Terraform script to add provisioning on GCP, DO and AWS #2676 Terraform script to add provisioning on GCP, DO and AWS
• Integrate with CircleCI

Details

Current state

Weave Net is currently integration-tested in two main ways:

• locally by developers, using VMs (see test/Vagrantfile) and make run-smoketests;
• on every git push, using CircleCI and Google Cloud Platform (GCP -- see circle.yml and test/gce.sh)

Software used during the build & testing currently is:

• Go 1.5.2 (see build/Dockerfile and vagrant-common.rb)
• Docker:
  • 1.6.2 (see Makefile)
  • 1.10.2 (see cd weave; vagrant up; vagrant ssh; docker --version)
  • 1.12.3 (see cd weave/test; vagrant up; vagrant ssh host1; docker --version)
• Ubuntu:
  • 15.10 Wily Werewolf (see cd weave; vagrant up; vagrant ssh; cat /etc/*-release and cd weave/test; vagrant up; vagrant ssh host1; cat /etc/*-release)
  • 16.04 LTS Xenial Xerus (see test/gce.sh)

Limitations & Potential Issues

1. We only test against one version of Docker, which will not flag issues with other versions used in Production by our users.
2. We do not systematically test against Docker release candidates -- and when we do, do it manually -- so cannot consistently react to or feed back issues before Docker releases.
3. We only test against one Operating System (Ubuntu), which will not flag issues with other ones used in Production by our users.
4. We only test against one version of Ubuntu, which will not flag issues with other ones used in Production by our users.
5. We only test against one cloud provider (GCP), which will not flag issues with other ones used in Production by our users.
6. Some of these versions are hardcoded, while some others depend on what is available in the OS' package manager at the time of the build which may change over time, i.e. their version is a "moving target".
7. Vagrant and GCP are not using the same versions of Docker and Ubuntu, which could cause issues.
8. When using Vagrant, different versions of Docker are used, which could cause issues.
9. It is very manual to set up testing environments for arbitrary configurations, which can be useful to reproduce various issues quickly.

Proposed Target State

1. Decouple dependencies definition, provisioning, configuration and testing:
   i. It should be easy to list all available versions of software we rely upon (OS, Docker, Go, etc.).
   ii. It should be easy to define a combination of these versions to use for development/testing.
   iii. It should be easy to provision a local or remote environment. In the case of remote environments, it should be easy to leverage any cloud provider.
   iv. It should be easy to configure the environment created in 1.iii. using the versions defined in 1.ii.
   v. It should be easy to then perform any kind of testing on this environment: manual, automated, integration tests, cross-version tests, stress tests, etc.

2. "Dependency-inject" versions to the build: OS, Docker and Go versions should all be configurable in one single place, and then passed in a "top-down" fashion to the provisioning and configuration layers and NOT hardcoded anywhere else below (other layers, in actual tests, etc.).

3. Testing:
   i. Integration testing should simply be one instance of 1.ii/iii/iv., with, e.g. the latest versions, or a hardcoded combination of versions of all dependencies. This would continue to be run on every git push.
   ii. Cross-version testing: it should be easy to repeat 3.i. for multiple combinations of versions, or even all combinations (cartesian product of versions of all dependencies), hence making it easy to:

   • spot incompatibilities between versions
   • document compatible versions (e.g. via a "compatibility matrix")

   This would typically be run weekly and/or manually before releases.

   iii. Cross-version testing should ideally be done concurrently, for faster feedback.
   iv. Stress testing & more advanced testing scenarios could also re-use 1.ii/iii/iv.

Proposed Implementation

1. Dependencies listing: a set of shell scripts would list all available versions of all dependencies (OSes/Kernels, Go, Docker, K8S, etc.).
2. Dependencies definition: make would, depending on the target, either:
   i. call 1. and select versions using pre-defined rules (e.g. latest production version, latest RC version, etc.)
   ii. use hardcoded versions.
3. Provisioning:
   i. Locally: a set of Vagrantfiles (see Vagrant) could be used to provision local VMs.
   ii. Remotely: a set of .tf (see Terraform) files could be used to provision machines from AWS, GCP and Digital Ocean.
   Docker Infrakit may also be a good tool to implement the above -- see also this article
4. Configuration: a set of Ansible scripts, as potentially more portable and idempotent than bash scripts, could be used to automatically configure the newly provisioned environments.

[rade]

Does this replace #229?

[rade]

Does this encompass #1747?

[rade]

Does this replace #2018?

[rade]

Does this encompass #2620?

[marccarre]

• automated system tests #229: yes, it does replace it IMO. It definitely overlaps with automated system tests #229 on 4., 5. and 6., and my understanding is that the existing "smoke tests" cover 1., 2. and 3., at least for simple scenarios. Maybe automated system tests #229 should be closed as this issue covers slightly more?
• Support varying Docker daemon configurations across smoke tests #1747 is encompassed: testing of Weave Net with various versions of Docker.
• Setup AWS for system tests run by CircleCI #2018 is encompassed: testing of Weave Net on AWS.
• Automated integration tests for weave-npc #2620 is encompassed: testing of Weave Net together with Kubernetes.

Let's keep #1747, #2018 and #2620 as an individual issues though. #2647 is more of an "epic" or meta-issue.

[marccarre]

Next: cross-version testing.

• Script to parallelize/orchestrate cross version testing
• Script to generate compatibility matrix

Note that cross-version testing of Kubernetes depends on kubernetes/release/issues/234.

[brb]

Fixed by #2694