Skip to content

PRJenkins

Jump to bottom
Josh Hursey edited this page Feb 5, 2020 · 45 revisions

Interacting with Jenkins CI from a PR

Triggering a retesting by all or a subset of CI

Add one of these commands to a new comment on the PR. Testing should be retriggered shortly (some sites have a few minute delay in polling).

  • All CI tests: bot:retest
  • Open MPI Pull Request Build Checker: bot:ompi:retest
  • IBM: bot:ibm:retest (see section below for fine tuning)
    • GNU only: bot:ibm:gnu:retest
    • PGI only: bot:ibm:pgi:retest
    • XL only: bot:ibm:xl:retest
  • Mellanox: bot:mellanox:retest

Request that a PR be CI tested

To request that a PR not be CI tested add both of the following to the PR description (not a comment on the PR). This is useful when making documentation only changes (e.g., README)

  • [skip ci]
  • bot:notest

Handling the White list

If you see one of these comments on a PR (or similar from a CI Build Bot):

  • Can one of the admins verify this patch?
  • Can one of the admins or Open MPI members verify this patch? This means that the user submitting the PR is not a member of the GitHub Open MPI organization posted the PR.

Any member of the GitHub Open MPI organization can cause the test to run by adding a comment to the PR. However there are some options here.

Option 1: Run a one time test of the PR

Safest option. Only tests the current state at the time of the comment.

Add either of the following to a comment on the PR:

  • bot:retest
  • test this please

Option 2: Run tests for the life of the pull request

Test the current state and any time this PR is updated.

Add the following to a comment on the PR:

  • ok to test

Option 3: Add the author to a whitelist of users

For the current PR and future PRs. This user can post/update PRs and will be treated as if they are a whitelisted member of the GitHub Open MPI group.

Add the following to a comment on the PR:

  • add to whitelist

IBM CI Fine Tuning

IBM CI Compile and Run Triggers

  • IBM CI (GNU): bot:ibm:gnu:retest / bot:ibm:scale:retest
    • Retest GNU compiler
    • Compile and run test (functional level).
      • Default: 10 nodes (adjustable via bot:ibm:nodes:#:test)
    • Scale run option available
      • Default: Disabled, but can be enabled with adjustment options.
  • IBM CI (XL): bot:ibm:xl:retest
    • Retest IBM XL compiler
    • Compile and run test (functional level).
      • Default: 2 nodes (adjustable via bot:ibm:nodes:#:test)
    • Scale run - Disabled.
  • IBM CI (PGI): bot:ibm:pgi:retest
    • Retest PGI compiler
    • Compile and run test (functional level).
      • Default: 2 nodes (adjustable via bot:ibm:nodes:#:test)
    • Scale run - Disabled.

IBM CI Scale Testing Adjustment Triggers

Adjustment triggers for scale testing. Notice that they end with :test and not :retest. If you want to specify one of the below it must be in addition to the bot:ibm:gnu:retest trigger. For example:

bot:ibm:gnu:retest
bot:ibm:nodes:11:test
bot:ibm:scale:42:test
  • Functional Testing: bot:ibm:nodes:#:test
    • Number of "nodes" to use for functional testing.
    • Default: 10 nodes
    • Minmum: 1 node
    • Maximum: 37 nodes
  • Scale Testing (Default): bot:ibm:scale:test
    • Enable scale functional testing with default number of "nodes"
    • Default: 64 nodes
  • Scale Testing (Adjustable): bot:ibm:scale:#:test
    • Enable scale functional testing with the specified number of "nodes"
    • Default: 0 (disabled)
    • Minmum: 0 nodes (disabled)
      • Special Value of 0 to disable scale testing
    • Maximum: 160 nodes

Setting up your own Jenkins PR Builder

NOTE: AS OF SEPTEMBER 2016, THIS PAGE IS A BIT STALE; WE NO LONGER HAVE AN ompi-release REPO. THIS PAGE NEEDS TO BE UPDATED.

Testing OMPI with Jenkins

Every Pull Request (PR) submitted to the ompi or ompi-release repositories is monitored by the Mellanox Jenkins server which will start a smoke-test.

  • The smoke-test script repository is located here:

https://github.com/mellanox-hpc/jenkins_scripts

  • The smoke-test script for OMPI is located here:

https://github.com/mellanox-hpc/jenkins_scripts/blob/master/jenkins/ompi/ompi_jenkins.sh

The smoke-test consists of tests organized by topics:

  • build - runs autogen/configure/make sequence (default: on)
  • examples - builds and run examples from ompi source tree (default: on)
  • threads - builds ompi with threads support and run thread tests (default: on)
  • src_rpm - create ompi.src.rpm (default: on) - (run make distclean, make distcheck, create src.rpm)
  • check - runs "make check" after build (default: on)
  • help_txt - runs contrib/check-help-strings.pl and test help files (default: off)
  • oshmem - runs oshmem tests (default: on)
  • vader - runs vader tests (default: on)
  • known_issues - run tests for known issues (default: off)
  • cov - runs coverity tests and generate html report (default: on)
  • all - all of the above (default: off)

In order to select specific test topic for submitted pull request - one can:

  • set the PR title (check "Edit" button next to the title) with a jenkins commands. (see below)
  • add as PR comment in the special format (see below)
jenkins: check,help_txt,threads,src_rpm

and add comment to the PR, in form:

bot:retest

Also, it is possible to unselect specific test topics and re-trigger jenkins on PR. To unselect specific tests - please add "no" to the test token name, i.e.

jenkins: nosrc_rpm,nocheck
bot: retest

Please note - only last comment in PR is consulted by Jenkins script for user-selected action.

Viewing Jenkins tests result online

  • The Jenkins test results for Pull requests for v1.8 branch can be found here:

http://bgate.mellanox.com/jenkins/job/gh-ompi-v1.8-pr/

  • The Jenkins test results for Pull requests for master branch can be found here:

http://bgate.mellanox.com/jenkins/job/gh-ompi-master-pr/

  • The jenkins test results for Merge commits to the master branch can be found here:

http://bgate.mellanox.com/jenkins/job/gh-ompi-master-merge/

  • When viewing results online, you can access following byproducts for every job:

  • Coverity report (code static analysis, reports number of detected error events per component in PR comment)

Hint: to suppress false positives coverity errors - use code annotations as described here: https://doclazy.wordpress.com/2011/07/14/coverity-suppressing-false-positives-with-cod/

How to re-trigger Jenkins testing

Every newly submitted PR will be tested by Jenkins automatically, based on PR title/comment Jenkins directives (if not present - default tests topics will be used).

Jenkins will retest PR if it was updated with new code, or if a new comment added.

Also, one can re-trigger Jenkins manually with the following PR comment:

bot:retest

Some organizations also have a site-specific request to re-trigger just their Jenkins setup. These are in addition to the bot:retest global trigger above. Below is a list of those additional re-trigger options:

  • IBM: bot:ibm:retest
    • GNU only: bot:ibm:gnu:retest
    • PGI only: bot:ibm:pgi:retest
    • XL only: bot:ibm:xl:retest
  • LANL: bot:lanl:retest
  • Mellanox: bot:mellanox:retest
  • OS-X only: bot:osx:retest

To request that a PR not be CI tested add both of the following to the PR description (not a comment on the PR). This is useful when making documentation only changes (e.g., README)

  • [skip ci]
  • bot:notest

Travis will also honor the [skip ci] or [ci skip] if added to the final commit in a PR.

TODO

batman

Happy testing!

Clone this wiki locally