Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Parallelise sanity checks in order to speed them up. #2699

Merged
merged 7 commits into from
Jan 9, 2017
Merged

Parallelise sanity checks in order to speed them up. #2699

merged 7 commits into from
Jan 9, 2017

Conversation

marccarre
Copy link
Contributor

@marccarre marccarre commented Dec 14, 2016 •
edited
Loading

Time spent running the sanity checks becomes a blocker when trying to ramp up the number of test servers, given ping check is O(n^2) and docker check is O(n). This change makes it closer to "perceived" O(1).
This is a pre-change refactoring for the #2647 and #2648 work.

@marccarre marccarre added this to the 1.9.0 milestone Dec 14, 2016
@marccarre marccarre self-assigned this Dec 14, 2016
@marccarre marccarre force-pushed the 2647-parallelise-sanity-checks branch from e88744b to 75ee528 Compare December 14, 2016 18:55
@marccarre marccarre requested a review from bboreham December 15, 2016 13:30
bboreham
bboreham previously requested changes Dec 19, 2016
View reviewed changes
test/sanity_check.sh
for host in $HOSTS; do
for other in $HOSTS; do
[ $host = $other ] || run_on $host $PING $other
if [ "$host" != "$other" ]; then
echo $(run_on $host $PING $other) &

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

test/sanity_check.sh
echo "# weave version"
docker inspect -f {{.Created}} weaveworks/weave:${WEAVE_VERSION:-latest}
weave_on $host version
check_docker $host &

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

test/sanity_check.sh
done
done
for pid in $pids; do wait $pid; done
unset pids

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

bboreham
bboreham reviewed Dec 19, 2016
View reviewed changes
test/sanity_check.sh
@@ -6,10 +6,17 @@ set -e

whitely echo Ping each host from the other

function check_ping() {

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

@marccarre marccarre force-pushed the 2647-parallelise-sanity-checks branch 4 times, most recently from aa30872 to 5c5ab18 Compare December 21, 2016 10:35
@marccarre
Copy link
Contributor Author

Sample output on CircleCI:

$ ls -1 /tmp/weave_net_sanity_check_host*
/tmp/weave_net_sanity_check_host1-7507-0.us-central1-a.positive-cocoa-90213.log
/tmp/weave_net_sanity_check_host2-7507-0.us-central1-a.positive-cocoa-90213.log
/tmp/weave_net_sanity_check_host3-7507-0.us-central1-a.positive-cocoa-90213.log
/tmp/weave_net_sanity_check_host4-7507-0.us-central1-a.positive-cocoa-90213.log
/tmp/weave_net_sanity_check_host5-7507-0.us-central1-a.positive-cocoa-90213.log

$ cat weave_net_sanity_check_host1-7507-0.us-central1-a.positive-cocoa-90213.log 
# =====================================
# Host Ping Check: host1-7507-0.us-central1-a.positive-cocoa-90213
# =====================================
host1-7507-0.us-central1-a.positive-cocoa-90213> PING host3-7507-0.us-central1-a.positive-cocoa-90213 (10.128.0.6) 56(84) bytes of data.
host1-7507-0.us-central1-a.positive-cocoa-90213> 
host1-7507-0.us-central1-a.positive-cocoa-90213> --- host3-7507-0.us-central1-a.positive-cocoa-90213 ping statistics ---
host1-7507-0.us-central1-a.positive-cocoa-90213> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
host1-7507-0.us-central1-a.positive-cocoa-90213> rtt min/avg/max/mdev = 0.228/0.228/0.228/0.000 ms
host1-7507-0.us-central1-a.positive-cocoa-90213> PING host4-7507-0.us-central1-a.positive-cocoa-90213 (10.128.0.8) 56(84) bytes of data.
host1-7507-0.us-central1-a.positive-cocoa-90213> 
host1-7507-0.us-central1-a.positive-cocoa-90213> --- host4-7507-0.us-central1-a.positive-cocoa-90213 ping statistics ---
host1-7507-0.us-central1-a.positive-cocoa-90213> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
host1-7507-0.us-central1-a.positive-cocoa-90213> rtt min/avg/max/mdev = 1.117/1.117/1.117/0.000 ms
host1-7507-0.us-central1-a.positive-cocoa-90213> PING host2-7507-0.us-central1-a.positive-cocoa-90213 (10.128.0.7) 56(84) bytes of data.
host1-7507-0.us-central1-a.positive-cocoa-90213> 
host1-7507-0.us-central1-a.positive-cocoa-90213> --- host2-7507-0.us-central1-a.positive-cocoa-90213 ping statistics ---
host1-7507-0.us-central1-a.positive-cocoa-90213> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
host1-7507-0.us-central1-a.positive-cocoa-90213> rtt min/avg/max/mdev = 0.188/0.188/0.188/0.000 ms
host1-7507-0.us-central1-a.positive-cocoa-90213> PING host5-7507-0.us-central1-a.positive-cocoa-90213 (10.128.0.5) 56(84) bytes of data.
host1-7507-0.us-central1-a.positive-cocoa-90213> 
host1-7507-0.us-central1-a.positive-cocoa-90213> --- host5-7507-0.us-central1-a.positive-cocoa-90213 ping statistics ---
host1-7507-0.us-central1-a.positive-cocoa-90213> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
host1-7507-0.us-central1-a.positive-cocoa-90213> rtt min/avg/max/mdev = 0.936/0.936/0.936/0.000 ms
# =====================================
# Host Version Info: host1-7507-0.us-central1-a.positive-cocoa-90213
# =====================================
# docker version
Client:
 Version:      1.9.1-circleci-cp-workaround
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   517b158
 Built:        Thu Dec 17 02:33:18 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        2016-06-01T22:00:43.211343203+00:00
 OS/Arch:      linux/amd64

# docker info
Containers: 0
Images: 39
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
CPUs: 1
Total Memory: 3.613 GiB
Name: host1-7507-0
ID: MLMI:OHJY:Z5DO:ERHC:LDJ6:C225:HAY4:3FTC:MRYG:MSIF:6CBD:H5GG

# docker inspect -f {{.Created}} weaveworks/weave:<version>
2016-12-21T11:16:30.949179355Z

# weave version
weave script unreleased
weave router unreleased
weave proxy  unreleased
weave plugin unreleased

@marccarre marccarre force-pushed the 2647-parallelise-sanity-checks branch from 4e42de9 to 6a073cc Compare December 21, 2016 11:41
@marccarre
Copy link
Contributor Author

Example of artifact files': https://circleci.com/gh/weaveworks/weave/7509#artifacts/containers/0

@marccarre marccarre removed their assignment Jan 9, 2017
@marccarre marccarre force-pushed the 2647-parallelise-sanity-checks branch from 6a073cc to f73fb53 Compare January 9, 2017 09:18
@marccarre marccarre dismissed bboreham’s stale review January 9, 2017 10:02

All changes made as requested. Unless I missed something, for some reason GitHub thinks there is still something outstanding.

@marccarre
Sanity checks' output is now redirected to $CIRCLE_ARTIFACTS directly…
edc0cdb 
… when running smoke tests in CircleCI.

Previously sanity checks' output was redirected to /tmp and then copied to $CIRCLE_ARTIFACTS, which is more "moving pieces".
@marccarre marccarre self-assigned this Jan 9, 2017
@marccarre
Copy link
Contributor Author

Sanity checks' output still gets properly redirected, but now under $CIRCLE_ARTIFACTS.
See also: https://circleci.com/gh/weaveworks/weave/7610#artifacts/containers/0

@bboreham bboreham merged commit e370443 into master Jan 9, 2017
@marccarre marccarre deleted the 2647-parallelise-sanity-checks branch January 10, 2017 11:19
@marccarre marccarre removed their assignment Jan 10, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bboreham bboreham bboreham left review comments

Assignees
No one assigned
Labels
chore
Projects
None yet
Milestone
1.9.0
Development

Successfully merging this pull request may close these issues.
2 participants
@marccarre @bboreham