Evaluate job restrictions plugin to improve build/test/(trestle?) machine isolation

[sxa]

Part of SSDF phase 3 PO 5.2

We installed the job restrictions plugin last year. This issue will cover testing it out and seeing if we can prevent jobs run by the test-triage team from being scheduled on production build machines. This should reduce the risks of any security issues on those machines. While we have improved isolation on Linux through the use of containerised systems, there is still a risk elsewhere, or if test jobs are scheuled on systems used for hosting build jobs.

[sxa]

Initial tests have not been fruitful - may be an issue with the expressions but not clear what the issues are.

[sxa]

Tried this again. I have tested this on one of the machines and it successfully restricts execution of anything other than build jobs on the machine. I will look at adding this criteria onto the AIX, ppc64le and windows x64 dockerhost_* and build_* machines for further testing (Doing it on Windows/x64 will also enhance our testing on ephemeral machines since there will be fewer real ones available!) This should give us good coverage in terms of seeing if there are any problems, although there may be cases where some jobs can run on "any" machine with e.g. build&&linux which we won't pick up with this testing (I've allowed centos7_docker_image_updater on the ppc64le ones for now but there may well be others) which should be shown up with these tests.

This has been applied to the following

• build&&aix:
  • https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-1
  • https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-2
  • https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-3
• dockerBuild&ppc64le (Expression is centos7_docker_image_updater.*|openjdk_build_docker_multiarch.*|build-scripts/jobs/.*|adoptium-packages-linux-pipeline_new.* on these):
  • https://ci.adoptium.net/computer/dockerhost-skytap-ubuntu2004-ppc64le-1
  • https://ci.adoptium.net/computer/docker-osuosl-ubuntu2004-ppc64le-1
  • https://ci.adoptium.net/computer/dockerhost%2Dosuosl%2Dubuntu2404%2Dppc64le%2D1/configure
• build&&windows (build-scripts/jobs/.*|build-scripts/release/create_installer_windows.*|build-scripts-pr-tester/build-test/.*|SXA-processCheck.* on these - unclear why it doesn't work without .* on the installer job expression - presumably a subtlety of the plugin). NOTE: I've only implemented this on the -1 and -2 systems. -3 is still free for testing (We should perhaps rename -3 as a test machine):
  • https://ci.adoptium.net/computer/build-azure-win2022-x64-1
  • https://ci.adoptium.net/computer/build-azure-win2022-x64-2
  • https://ci.adoptium.net/computer/build-azure-win2022-x64-3

I've kicked off https://ci.adoptium.net/job/build-scripts/job/openjdk11-pipeline/2723 to see how it goes and where the jobs end up. Individual build pipelines are as follows:

• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-aix-ppc64-temurin/380
• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-linux-ppc64le-temurin/365
• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-temurin/405
• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x86-32-temurin/376

If a build fails to get scheduled you'll see something like this:

15:44:59  LABEL: docker-osuosl-ubuntu2004-ppc64le-1
[Pipeline] stage
[Pipeline] { (Queue)
[Pipeline] nodesByLabel
15:44:59  Found a total of 1 nodes with the 'docker-osuosl-ubuntu2004-ppc64le-1' label
[Pipeline] echo
15:44:59  dynamicAgents: [fyre]
[Pipeline] node
15:45:14  Still waiting to schedule task
[...]
15:45:14  ‘build-marist-rhel8-s390x-1’ doesn’t have label ‘docker-osuosl-ubuntu2004-ppc64le-1’
[...]

And then no further progress

[sxa]

Other jobs that may be affected:

• https://ci.adoptium.net/job/openjdk_build_docker_multiarch/
• https://ci.adoptium.net/job/rhel7_docker_image_updater/

[sxa]

October release is complete, so this can now been rolled out to other machines. Expression should be:

• centos7_docker_image_updater.*|openjdk_build_docker_multiarch.*|build-scripts/jobs/.*|build-scripts-pr-test/jobs/.*|adoptium-packages-linux-pipeline_new.*|SXA-processCheck.* for the Linux systems (rhel7_ instead of centos7_ at the start for s390x)
• build-scripts/jobs/.*|build-scripts-pr-test/build-test/jobs/.*|SXA-processCheck.* for other UNIX-based systems (Solaris/AIX)
• build-scripts/jobs/.*|build-scripts/utils/cross-compiled-version-out.*|build-scripts/release/create_installer_windows.*|build-scripts-pr-tester/build-test/jobs/.*|SXA-processCheck.* for Windows.
  Noting that the small number of Solaris machines available mean that this may cause a bit of a backlog on the test system. I'll make the change and monitor the length of time the pipelines take during this week's EA cycle. Also noting that at present we only have one AIX machine witht he OpenXL17 compiler so that needs to be retained as a shared build/test system for recent JDKs, but the older ones can be set up as described above. The following machines have been changed:
• https://ci.adoptium.net/computer/build-azure-solaris10-x64-1/
• https://ci.adoptium.net/computer/build-azure-win2022-x64-1/
• https://ci.adoptium.net/computer/build-azure-win2022-x64-2/
• Note that https://ci.adoptium.net/computer/build-azure-win2022-x64-3/ has had the build label removed as it is better allocated to test with two exclusive build machines now in play.
• https://ci.adoptium.net/computer/build-ibmcloud-win2022-x64-1/ (Currently out of action)
• https://ci.adoptium.net/computer/build-marist-rhel8-s390x-1/ (Has rhel7_docker_image_updater instead of the centos7 version)
• https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-1/
• https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-2/
• NOT MODIFIED: https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-3/
• https://ci.adoptium.net/computer/dockerhost-azure-ubuntu2404-x64-1/
• https://ci.adoptium.net/computer/dockerhost-equinix-ubuntu2204-armv8-1/
• https://ci.adoptium.net/computer/dockerhost-equinix-ubuntu2404-armv8-1/
• https://ci.adoptium.net/computer/dockerhost-skytap-ubuntu2004-ppc64le-1/
• https://ci.adoptium.net/computer/dockerhost-skytap-ubuntu2204-x64-1/
• https://ci.adoptium.net/computer/docker-osuosl-ubuntu2004-ppc64le-1

[sxa]

If you hit an issue with the plugin rejecting your job you will typically see something like this in the log (similar to if you specify an incorrect label that cannot be matched):

16:51:24  ‘build-azure-solaris10-x64-1’ doesn’t have label ‘build-azure-win2022-x64-1’
16:51:24  ‘build-azure-win2022-x64-2’ doesn’t have label ‘build-azure-win2022-x64-1’
16:51:24  ‘build-azure-win2022-x64-3’ doesn’t have label ‘build-azure-win2022-x64-1’
16:51:24  ‘build-digitalocean-centos69-x64-2’ doesn’t have label ‘build-azure-win2022-x64-1’
16:51:24  ‘build-equinix-ubuntu1604-armv7l-2’ doesn’t have label ‘build-azure-win2022-x64-1’
16:51:24  ‘build-ibmcloud-win2022-x64-1’ doesn’t have label ‘build-azure-win2022-x64-1’
16:51:24  ‘build-linux-x64-dd9180’ doesn’t have label ‘build-azure-win2022-x64-1’
[...] Repeated for every agent on jenkins.

[sxa]

Closing this as it has now been implemented and appears to be working. Any problems can be raised as separate issues.

[sxa]

ci.role.test has been removed from all active build machines with the exception of the one AIX machine that we use for JDK23+ as we have no others at present. build-osuosl-aix72-ppc64-3 in order to avoid any confusion with scheduling.