change conditional parallel tests to check for JULIA_TESTFULL

[amitmurthy]

As discussed in #10505 (comment)

[amitmurthy]

cc @tkelman

[tkelman]

Cool, trying this out on my machine. @timholy do these break anything for you, if you've been running the full version of your subarray test set?

[tkelman]

Hm, this doesn't work for me, I get a lot of "Host key verification failed" and nothing visibly happening for "Testing SSH addprocs with 30 workers"

[amitmurthy]

Yes, "ssh addprocs requires sshd to be running locally with passwordless login enabled." as mentioned in the comments.

We could print that requirement to screen before running ssh manager tests.

[amitmurthy]

Pushed an update.

The required setup for SSH Manager tests is printed now. Also introduced another environment variable JULIA_ADDPROCS_NUM, which, if defined, sets the number of worker processes to be created. If undefined, defaults to 30, a minimum of 4GB RAM is recommended for a 30 worker test.

I usually test with around 90 worker setup and teardown.

[amitmurthy]

OSX timeout is after all core tests have passed.

[tkelman]

I can get JULIA_TESTFULL=1 make test-parallel to work after fiddling a bit with ssh keys, but so far JULIA_TESTFULL=1 make testall is hitting 60-second timeouts, annoyingly.

[amitmurthy]

What are your machine specs?

[tkelman]

Pretty old and slow, I think it has 3gb ram? I'll try with fewer than 30.

[amitmurthy]

Yes, that could be the problem - especially if the machine is swapping.

We can lower the default counts further. As things stand SSHManager does not get tested as part of CI. It will be good if it gets any testing done by some developers turning on JULIA_TESTFULL !

[tkelman]

It will be good if it gets any testing done by some developers turning on JULIA_TESTFULL !

I agree. Some of this could also go into perf, or a new category for stress tests.

[amitmurthy]

It is not really performance or stress. To speed up things, addprocs executes some of the setup tasks in parallel. Testing with a larger number of workers catches any errors due to the parallel initialization.

[tkelman]

Well, it certainly stresses small machines to run it, and we don't do it on every CI build for presumably the same reason. We could call it test-big or test-thorough, the name's not too important, but this kind of thing should maybe be moved out into a separate category of test runs at some point, rather than going off an env var. For now I think it's fine though.

[amitmurthy]

Reduced default additional workers to 9.