From 6d6d9cb966b952a8bf424c41703074644a8346cb Mon Sep 17 00:00:00 2001 From: "Matthew T. Warkentin" Date: Mon, 16 Nov 2020 14:36:15 -0500 Subject: [PATCH 01/11] Consolidated options documentation (#218) Added documentation for all of the cmq options in one place --- vignettes/userguide.Rmd | 45 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-) diff --git a/vignettes/userguide.Rmd b/vignettes/userguide.Rmd index 2cd1cd87..fb3d7425 100644 --- a/vignettes/userguide.Rmd +++ b/vignettes/userguide.Rmd @@ -95,7 +95,7 @@ welcome](https://github.com/mschubert/clustermq/issues/new). remotes::install_github('mschubert/clustermq', ref="develop") ``` -## Configuration +## Configuration {#configuration} Choose your preferred parallelism using: @@ -148,7 +148,7 @@ customized](#configuration), e.g. to activate [compute environments or containers](#Environments). -### SSH connector +### SSH connector {#ssh-connector} There are reasons why you might prefer to not to work on the computing cluster directly but rather on your local machine instead. @@ -316,6 +316,45 @@ load_mtcars_example() make(my_plan, parallelism = "clustermq", jobs = 2, verbose = 4) ``` +## Options + +The various configurable options are mentioned throughout the documentation, +where applicable, however, we list all of the options here for reference. + +Options can be set by including a call to `options( = )` in your +`.Rprofile`, or by calling `options( = )` in a script or +interactively during a session. + +* `clustermq.scheduler` - One of the supported + [`clustermq` schedulers](#configuration); options are `"LOCAL"`, + `"multiprocess"`, `"multicore"`, `"lsf"`, `"sge"`, `"slurm"`, `"pbs"`, + `"Torque"`, or `"ssh"` (default is the HPC scheduler found in `$PATH`, + otherwise `"LOCAL"`) +* `clustermq.host` - The name of the node or device for constructing the + `ZeroMQ` host address (default is `Sys.info()["nodename"]`) +* `clustermq.ssh.host` - The user name and host for + [connecting to the HPC via SSH](#ssh-connector) (e.g. `user@host`); we + recommend setting up SSH keys for password-less login +* `clustermq.ssh.log` - Path for a file (on the SSH host) that will be created + and populated with logging information regarding the SSH connection + (e.g. `"~/cmq_ssh.log"`); helpful for debugging purposes +* `clustermq.ssh.timeout` - The amount of time to wait (in seconds) for a SSH + start-up connection before timing out (default is 5 seconds) +* `clustermq.worker.timeout` - The amount of time to wait (in seconds) for + master-worker communication before timing out (default is 600 seconds) +* `clustermq.error.timeout` - The amount of time to wait (in seconds), in case + of a worker error, for the remaining workers to finish their computations + and shut down cleanly (default is `min(timeout, 30)` seconds) +* `clustermq.template` - Path to a [template file](#scheduler-templates) for + submitting HPC jobs; only necessary if using your own template, otherwise + the default template will be used (default depends on + `clustermq.scheduler`) +* `clustermq.data.warning` - The threshold for the size of the common data (in + Mb) before `clustermq` throws a warning (default is 1000) +* `clustermq.defaults` - A named-list of default values for the HPC template; + this takes precedence over defaults specified in the template file + (default is an empty list (i.e. `list()`)) + ## Troubleshooting ### Debugging workers @@ -530,7 +569,7 @@ chmod u+x ~/bin/qsub SINGULARITYENV_APPEND_PATH=~/bin ``` -## Scheduler templates +## Scheduler templates {#scheduler-templates} ### LSF {#LSF} From 6c1436eb4248f1df433721d2f66549863e990fb0 Mon Sep 17 00:00:00 2001 From: Michael Schubert Date: Sat, 12 Dec 2020 13:32:27 +0100 Subject: [PATCH 02/11] mention functions can be exported (fixes #220) --- vignettes/userguide.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vignettes/userguide.Rmd b/vignettes/userguide.Rmd index fb3d7425..fc109c29 100644 --- a/vignettes/userguide.Rmd +++ b/vignettes/userguide.Rmd @@ -237,7 +237,7 @@ Q(fx, x=1:3, const=list(y=10), n_jobs=1) ``` If a function relies on objects in its environment that are not passed as -arguments, they can be exported using the `export` argument: +arguments (including other functions), they can be exported using the `export` argument: ```{r} fx = function(x) x * 2 + y From 0db22c7b170babf7ce09b48bc7e60d4386cc8d1d Mon Sep 17 00:00:00 2001 From: mschubert Date: Fri, 25 Dec 2020 22:01:52 +0100 Subject: [PATCH 03/11] add gh actions check --- .Rbuildignore | 1 + .github/workflows/check-standard.yaml | 84 +++++++++++++++++++++++++++ .gitignore | 1 + 3 files changed, 86 insertions(+) create mode 100644 .github/workflows/check-standard.yaml diff --git a/.Rbuildignore b/.Rbuildignore index d2e3d2cc..7672ddb1 100644 --- a/.Rbuildignore +++ b/.Rbuildignore @@ -10,3 +10,4 @@ README\.md CITATION _pkgdown\.yml clustermq_[0-9.]+\.tar\.gz +^\.github$ diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml new file mode 100644 index 00000000..9cba71fb --- /dev/null +++ b/.github/workflows/check-standard.yaml @@ -0,0 +1,84 @@ +# For help debugging build failures open an issue on the RStudio community with the 'github-actions' tag. +# https://community.rstudio.com/new-topic?category=Package%20development&tags=github-actions +on: + push: + branches: + - main + - master + pull_request: + branches: + - main + - master + +name: R-CMD-check + +jobs: + R-CMD-check: + runs-on: ${{ matrix.config.os }} + + name: ${{ matrix.config.os }} (${{ matrix.config.r }}) + + strategy: + fail-fast: false + matrix: + config: + - {os: windows-latest, r: 'release'} + - {os: macOS-latest, r: 'release'} + - {os: ubuntu-20.04, r: 'release', rspm: "https://packagemanager.rstudio.com/cran/__linux__/focal/latest"} + - {os: ubuntu-20.04, r: 'devel', rspm: "https://packagemanager.rstudio.com/cran/__linux__/focal/latest"} + + env: + R_REMOTES_NO_ERRORS_FROM_WARNINGS: true + RSPM: ${{ matrix.config.rspm }} + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + + steps: + - uses: actions/checkout@v2 + + - uses: r-lib/actions/setup-r@v1 + with: + r-version: ${{ matrix.config.r }} + + - uses: r-lib/actions/setup-pandoc@v1 + + - name: Query dependencies + run: | + install.packages('remotes') + saveRDS(remotes::dev_package_deps(dependencies = TRUE), ".github/depends.Rds", version = 2) + writeLines(sprintf("R-%i.%i", getRversion()$major, getRversion()$minor), ".github/R-version") + shell: Rscript {0} + + - name: Cache R packages + if: runner.os != 'Windows' + uses: actions/cache@v2 + with: + path: ${{ env.R_LIBS_USER }} + key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }} + restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1- + + - name: Install system dependencies + if: runner.os == 'Linux' + run: | + while read -r cmd + do + eval sudo $cmd + done < <(Rscript -e 'writeLines(remotes::system_requirements("ubuntu", "20.04"))') + + - name: Install dependencies + run: | + remotes::install_deps(dependencies = TRUE) + remotes::install_cran("rcmdcheck") + shell: Rscript {0} + + - name: Check + env: + _R_CHECK_CRAN_INCOMING_REMOTE_: false + run: rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning", check_dir = "check") + shell: Rscript {0} + + - name: Upload check results +# if: failure() + uses: actions/upload-artifact@main + with: + name: ${{ runner.os }}-r${{ matrix.config.r }}-results + path: check diff --git a/.gitignore b/.gitignore index 340239ea..4c7bb928 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,4 @@ src/*.so *.log clustermq.Rcheck src/Makevars +.github/*.html From f6c4d7a475fbe3adcf636e74c01e225736935d60 Mon Sep 17 00:00:00 2001 From: mschubert Date: Fri, 25 Dec 2020 23:48:17 +0100 Subject: [PATCH 04/11] add zeromq via brew for macOS --- .github/workflows/check-standard.yaml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index 9cba71fb..fc2c8382 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -56,7 +56,7 @@ jobs: key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }} restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1- - - name: Install system dependencies + - name: Install system dependencies (Linux) if: runner.os == 'Linux' run: | while read -r cmd @@ -64,6 +64,12 @@ jobs: eval sudo $cmd done < <(Rscript -e 'writeLines(remotes::system_requirements("ubuntu", "20.04"))') + - name: Install system dependencies (macOS) + if: runner.os == 'macOS' + run: | + brew update + brew install zeromq + - name: Install dependencies run: | remotes::install_deps(dependencies = TRUE) @@ -77,7 +83,7 @@ jobs: shell: Rscript {0} - name: Upload check results -# if: failure() + if: failure() uses: actions/upload-artifact@main with: name: ${{ runner.os }}-r${{ matrix.config.r }}-results From e88892feb8802a6e3146178a064cf76d3248f279 Mon Sep 17 00:00:00 2001 From: mschubert Date: Sun, 27 Dec 2020 20:32:56 +0100 Subject: [PATCH 05/11] replace travis by gha --- .github/workflows/check-standard.yaml | 34 ++++++++++++++++ .travis.yml | 58 --------------------------- README.md | 2 +- 3 files changed, 35 insertions(+), 59 deletions(-) delete mode 100644 .travis.yml diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index fc2c8382..01c3837b 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -5,10 +5,12 @@ on: branches: - main - master + - develop pull_request: branches: - main - master + - develop name: R-CMD-check @@ -70,18 +72,50 @@ jobs: brew update brew install zeromq + - name: Set up local key-based SSH + if: runner.os != 'Windows' # GHA does not allow empty passphrase on Windows + run: | + ssh-keygen -t rsa -f ~/.ssh/id_rsa -N "" -q + cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys + ssh-keyscan -t rsa 127.0.0.1 >> ~/.ssh/known_hosts + echo "Host 127.0.0.1" >> ~/.ssh/config + echo " IdentityFile ~/.ssh/id_rsa" >> ~/.ssh/config + echo "$(hostname) 127.0.0.1" >> ~/.hosts + chmod og-rw ~ + + - name: Add fake scheduler to PATH + run: | + echo "$(pwd)/tests/bin" >> $GITHUB_PATH + echo "$(dirname $(which R))" >> $GITHUB_PATH + - name: Install dependencies run: | remotes::install_deps(dependencies = TRUE) remotes::install_cran("rcmdcheck") shell: Rscript {0} + - name: Query capabilities + run: | + which R + which sbatch || echo "sbatch not found" + ssh 127.0.0.1 'which R; which sbatch; echo $PATH' || true + ssh 127.0.0.1 'R --no-save --no-restore -e ".libPaths()"' || true +# R --no-save --no-restore -e "message(clustermq:::qsys_default)" > /dev/null || true + - name: Check env: _R_CHECK_CRAN_INCOMING_REMOTE_: false run: rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning", check_dir = "check") shell: Rscript {0} + - name: Print logs if failure + if: failure() + run: | + cat ~/*.log + cat ~/worker.log || true + cat ~/ssh_proxy.log || true + cat clustermq.Rcheck/tests/* || true + - name: Upload check results if: failure() uses: actions/upload-artifact@main diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index be67292f..00000000 --- a/.travis.yml +++ /dev/null @@ -1,58 +0,0 @@ -language: r - -jobs: - - os: linux - r: oldrel - - os: linux - r: release - - os: linux - r: devel - - os: osx - r: oldrel - - os: osx - r: release - -warnings_are_errors: true - -cache: packages - -before_install: - - if [ "$TRAVIS_OS_NAME" = "linux" ]; then - sed -i "1iexport PATH=$(dirname $(which R)):$(pwd)/tests/bin:\$PATH" ~/.bashrc; - sed -i "2iexport R_LIBS_USER=$R_LIBS_USER" ~/.bashrc; - sed -i "3iexport R_LIBS_SITE=$R_LIBS_SITE" ~/.bashrc; - source ~/.bashrc; - sudo apt-get install libzmq3-dev; - elif [ "$TRAVIS_OS_NAME" = "osx" ]; then - brew update; - brew install zeromq; - fi - - ssh-keygen -t rsa -f ~/.ssh/id_rsa -N "" -q - - cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys - - ssh-keyscan -t rsa 127.0.0.1 >> ~/.ssh/known_hosts - - echo "Host 127.0.0.1 - IdentityFile ~/.ssh/id_rsa" >> ~/.ssh/config - - echo "$(hostname) 127.0.0.1" >> ~/.hosts - -before_script: - - pwd - - cat ~/.bashrc - - which R - - which sbatch || true - - ssh 127.0.0.1 'which R; which sbatch; echo $PATH' || true - - ssh 127.0.0.1 'R --no-save --no-restore -e ".libPaths()"' || true - -script: - - R CMD build . - - R CMD INSTALL *.tar.gz - - R --no-save --no-restore -e "message(clustermq:::qsys_default)" > /dev/null - - ssh 127.0.0.1 'R --no-save --no-restore -e "message(clustermq:::qsys_default)" > /dev/null' || true - - make test - - timeout 300 make check -# - timeout 120 R CMD check --as-cran *.tar.gz - -after_failure: - - cat ~/*.log - - cat ~/worker.log || true - - cat ~/ssh_proxy.log || true - - cat clustermq.Rcheck/tests/* || true diff --git a/README.md b/README.md index 0806dd8b..19bcef59 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ClusterMQ: send R function calls as cluster jobs ================================================ [![CRAN version](http://www.r-pkg.org/badges/version/clustermq)](https://cran.r-project.org/package=clustermq) -[![Build Status](https://travis-ci.org/mschubert/clustermq.svg?branch=master)](https://travis-ci.org/mschubert/clustermq) +[![Build Status](https://github.com/mschubert/clustermq/workflows/R-CMD-check/badge.svg?branch=master)](https://github.com/mschubert/clustermq/actions) [![CRAN downloads](http://cranlogs.r-pkg.org/badges/clustermq)](http://cran.rstudio.com/web/packages/clustermq/index.html) [![DOI](https://zenodo.org/badge/DOI/10.1093/bioinformatics/btz284.svg)](https://doi.org/10.1093/bioinformatics/btz284) From 0aea9472b890a840e3f584eaa9827900ebe9043e Mon Sep 17 00:00:00 2001 From: mschubert Date: Sun, 27 Dec 2020 21:30:08 +0100 Subject: [PATCH 06/11] add 'make test' --- .github/workflows/check-standard.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index 01c3837b..fd6584b9 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -102,6 +102,12 @@ jobs: ssh 127.0.0.1 'R --no-save --no-restore -e ".libPaths()"' || true # R --no-save --no-restore -e "message(clustermq:::qsys_default)" > /dev/null || true + - name: make test + if: runner.os != 'Windows' + run: | + R CMD INSTALL . + make test + - name: Check env: _R_CHECK_CRAN_INCOMING_REMOTE_: false From bb30d614155b3d0673455922a37c1875a56f3d7a Mon Sep 17 00:00:00 2001 From: mschubert Date: Sun, 27 Dec 2020 23:26:34 +0100 Subject: [PATCH 07/11] add test timeout --- .github/workflows/check-standard.yaml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index fd6584b9..e4d41a35 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -70,7 +70,8 @@ jobs: if: runner.os == 'macOS' run: | brew update - brew install zeromq + brew install zeromq coreutils +# echo "/usr/local/opt/coreutils/libexec/gnubin" >> $GITHUB_PATH - name: Set up local key-based SSH if: runner.os != 'Windows' # GHA does not allow empty passphrase on Windows @@ -106,7 +107,7 @@ jobs: if: runner.os != 'Windows' run: | R CMD INSTALL . - make test + timeout 300 make test - name: Check env: @@ -115,7 +116,7 @@ jobs: shell: Rscript {0} - name: Print logs if failure - if: failure() + if: failure() && runner.os != 'Windows' run: | cat ~/*.log cat ~/worker.log || true From 463c1425a655b5fcd9d90a6084e3e2eac51d1664 Mon Sep 17 00:00:00 2001 From: mschubert Date: Mon, 28 Dec 2020 00:09:47 +0100 Subject: [PATCH 08/11] schedule build, all branches except gh-pages --- .github/workflows/check-standard.yaml | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index e4d41a35..342a211d 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -2,15 +2,11 @@ # https://community.rstudio.com/new-topic?category=Package%20development&tags=github-actions on: push: - branches: - - main - - master - - develop + branches-ignore: gh-pages pull_request: - branches: - - main - - master - - develop + branches-ignore: gh-pages + schedule: + - cron: "0 0 * * 2" name: R-CMD-check @@ -71,7 +67,6 @@ jobs: run: | brew update brew install zeromq coreutils -# echo "/usr/local/opt/coreutils/libexec/gnubin" >> $GITHUB_PATH - name: Set up local key-based SSH if: runner.os != 'Windows' # GHA does not allow empty passphrase on Windows @@ -95,18 +90,23 @@ jobs: remotes::install_cran("rcmdcheck") shell: Rscript {0} + - name: Install R package + if: runner.os != 'Windows' + run: | + R CMD INSTALL . + - name: Query capabilities + if: runner.os != 'Windows' # last line does not recognize -e run: | which R which sbatch || echo "sbatch not found" ssh 127.0.0.1 'which R; which sbatch; echo $PATH' || true ssh 127.0.0.1 'R --no-save --no-restore -e ".libPaths()"' || true -# R --no-save --no-restore -e "message(clustermq:::qsys_default)" > /dev/null || true + R --no-save --no-restore -e "message(clustermq:::qsys_default)" || true - name: make test if: runner.os != 'Windows' run: | - R CMD INSTALL . timeout 300 make test - name: Check From 01c58356cb65ec30d11dfc3943d381a1dfd36551 Mon Sep 17 00:00:00 2001 From: mschubert Date: Mon, 28 Dec 2020 13:17:51 +0100 Subject: [PATCH 09/11] check cmq via ssh --- .github/workflows/check-standard.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index 342a211d..e55fb407 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -96,13 +96,15 @@ jobs: R CMD INSTALL . - name: Query capabilities - if: runner.os != 'Windows' # last line does not recognize -e + if: runner.os != 'Windows' # does not recognize -e run: | + set -x which R which sbatch || echo "sbatch not found" ssh 127.0.0.1 'which R; which sbatch; echo $PATH' || true - ssh 127.0.0.1 'R --no-save --no-restore -e ".libPaths()"' || true - R --no-save --no-restore -e "message(clustermq:::qsys_default)" || true + ssh 127.0.0.1 'R --slave --no-restore -e ".libPaths()"' || true + R --slave --no-restore -e "message(clustermq:::qsys_default)" || true + ssh 127.0.0.1 'R --slave --no-restore -e "message(clustermq:::qsys_default)"' || true - name: make test if: runner.os != 'Windows' From d78ea380da2390fd0fa77557ef8459a45aee61d2 Mon Sep 17 00:00:00 2001 From: mschubert Date: Mon, 28 Dec 2020 14:02:18 +0100 Subject: [PATCH 10/11] add cmq to installed R pkgs --- .github/workflows/check-standard.yaml | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index e55fb407..2b012fa7 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -79,21 +79,19 @@ jobs: echo "$(hostname) 127.0.0.1" >> ~/.hosts chmod og-rw ~ - - name: Add fake scheduler to PATH - run: | - echo "$(pwd)/tests/bin" >> $GITHUB_PATH - echo "$(dirname $(which R))" >> $GITHUB_PATH - - name: Install dependencies run: | remotes::install_deps(dependencies = TRUE) remotes::install_cran("rcmdcheck") shell: Rscript {0} - - name: Install R package + - name: Install R package and add paths if: runner.os != 'Windows' run: | R CMD INSTALL . + echo '.libPaths("~/work/_temp/Library")' >> ~/.Rprofile # cmq package in R + echo "$(pwd)/tests/bin" >> $GITHUB_PATH # local cmq + sed -i -e "1iexport PATH=$(pwd)/tests/bin:\$PATH" ~/.bashrc # ssh cmq - name: Query capabilities if: runner.os != 'Windows' # does not recognize -e @@ -120,7 +118,8 @@ jobs: - name: Print logs if failure if: failure() && runner.os != 'Windows' run: | - cat ~/*.log + set -x + cat ~/*.log || true cat ~/worker.log || true cat ~/ssh_proxy.log || true cat clustermq.Rcheck/tests/* || true From 0b7546836272fd908e5c5db567ebf4195beaeffa Mon Sep 17 00:00:00 2001 From: mschubert Date: Tue, 29 Dec 2020 11:59:45 +0100 Subject: [PATCH 11/11] disable sed on macOS --- .github/workflows/check-standard.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/check-standard.yaml b/.github/workflows/check-standard.yaml index 2b012fa7..63851bbb 100644 --- a/.github/workflows/check-standard.yaml +++ b/.github/workflows/check-standard.yaml @@ -91,7 +91,7 @@ jobs: R CMD INSTALL . echo '.libPaths("~/work/_temp/Library")' >> ~/.Rprofile # cmq package in R echo "$(pwd)/tests/bin" >> $GITHUB_PATH # local cmq - sed -i -e "1iexport PATH=$(pwd)/tests/bin:\$PATH" ~/.bashrc # ssh cmq + sed -i "1iexport PATH=$(pwd)/tests/bin:\$PATH" ~/.bashrc || true # ssh cmq - name: Query capabilities if: runner.os != 'Windows' # does not recognize -e