Reduce the number of procs of MPI test for robust CI #136
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
keisukefukuda
changed the title
keisukefukuda
changed the title
shu65
approved these changes
Nov 20, 2017
.travis.yml
Outdated
| - (for NP in 1 2 3; do PYTHONWARNINGS='ignore::FutureWarning,module::DeprecationWarning' mpiexec -n ${NP} nosetests -v -a '!nccl,!gpu,!slow' || exit $?; done) | ||
| # - cd tests | ||
| # - PYTHONWARNINGS='ignore::FutureWarning,module::DeprecationWarning' nosetests -a '!gpu,!slow' --with-doctest chainer_tests | ||
| - if mpiexec --version | grep -q OpenRTE; then NOBIND="--bind-to none"; else NOBIND= ; fi |
There was a problem hiding this comment.
I put it for future safety to prevent potential errors, but it works without it now. I guess we can remove it as far as it works. Removed it.
.travis.yml
Outdated
| # - cd tests | ||
| # - PYTHONWARNINGS='ignore::FutureWarning,module::DeprecationWarning' nosetests -a '!gpu,!slow' --with-doctest chainer_tests | ||
| - if mpiexec --version | grep -q OpenRTE; then NOBIND="--bind-to none"; else NOBIND= ; fi | ||
| - (for NP in 1 2; do for T in $(find tests -name "test_*.py") ; do mpiexec -n ${NP} nosetests -s -v -a '!nccl,!gpu,!slow' $T || exit $?; done; done) |
There was a problem hiding this comment.
Why are the tests run separately?
I think it is not needed to change.
There was a problem hiding this comment.
This was also "safety guard" for future potential errors... but I removed it for the same reason.
.travis.yml
Outdated
| # - PYTHONWARNINGS='ignore::FutureWarning,module::DeprecationWarning' nosetests -a '!gpu,!slow' --with-doctest chainer_tests | ||
| - if mpiexec --version | grep -q OpenRTE; then NOBIND="--bind-to none"; else NOBIND= ; fi | ||
| - (for NP in 1 2; do for T in $(find tests -name "test_*.py") ; do mpiexec -n ${NP} nosetests -s -v -a '!nccl,!gpu,!slow' $T || exit $?; done; done) | ||
| # - (for NP in 1 2 4; do for T in $(find tests -name "test_*.py") ; do PYTHONWARNINGS='ignore::FutureWarning,module::DeprecationWarning' mpiexec ${NOBIND} -n ${NP} nosetests -s -v -a '!nccl,!gpu,!slow' $T || exit $?; done; done) |
There was a problem hiding this comment.
Could you remove this comments because I do not think that it is necessary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, CI on Travis CI often fails because of deadlocks in 3-process test_mnist.
Although I don't find the exact reason of the deadlock, tests with up to 2 processes
does not suffer from the deadlock issue.
This PR changes the number of processes in the MPI tests (1,2,3) to two processes in
.travis.yml.Also, this PR changes to invoke
nosetestscommand for eachtest_*.pyfiles so it can avoid deadlocks.