EPIC: Investigate Solution For Running Jenkins Agent On Solaris with Java 17

[steelhead31]

See: #3741 

A solution will need to be investigated and implemented for running Jenkins agents on Solaris with Java 17.

Since JDK 17 is not supported on solaris, this will need additional work.

Specifically these 4 agents..

• build-siteox-solaris10u11-sparcv9-1
• test-siteox-solaris10u11-sparcv9-1
• build-azure-solaris10-x64-1
• test-azure-solaris10-x64-1

Whatever solution is found, will likely also need to be applied to the TCK solaris nodes.

[sxa]

Options:

1. Attempt to rebuild the new LTS agent ourselves such that it will run on JDK11.
2. Attempt to build a JDK17 on Solaris 10 (There are patches to make it work with gcc but I struggled last time I tried to use them)
3. Attempt to build a JDK17 on Solaris 11 and switch to that (also with GCC as most Solaris support was removed in JDK15), while building against a Solaris 10 sysroot (subject to that being feasible in JDK8).
4. Same as the point above, but run a Solaris 10 zone under Solaris 11, similar to what we do with docker on Linux (Would also potentially allow us to satisfy Investigate feasibility of ephemeral build systems for non-Linux platforms temurin-build#3264 (comment)). This would require a bit of 'roll your own' since I doubt there's a docker plugin that can make this easy...
5. Running a non-Solaris JDK17 using qemu on Solaris
6. Running some proxy machine to run the jenkins agent and pass commands to the remote machine (likely hard because of the amount of shell invocations in our jenkins pipelines)
7. Run a second jenkins server running an older version which can communicate with the Solaris node.

Anything else?

Notes:

• All of the "build JDK17" choices would likely force us into building each JDK up from 11 to get there.
• We could probably survive with a 'zero' implementation instead of one with a full JIT if required. I suspect that may be all we could get on SPARC ...

[sxa]

IMHO Options 1 or 3 would be the cleanest ones if we can make them work.

[steelhead31]

1. A quick test of the tribblix Solaris 11 / JDK17 proved it was incompatible with Solaris 10
2. Reached out and confirmed that there is 100% no Solaris 10 / JDK17 package.

[sxa]

Diagram from a discussion this morning:

The diagram in the previous comment is intended to summarise the different ways where builds (also applicable to test, but I'll start with build)
Lines from top to bottom:

1. The normal situation where the jenkins agent runs on a system, and it runs the actual work on the same machine.
2. The 'docker build' situation where the agent runs in an environment (host that can run docker) but the actual work is done in a container managed by the jenkins plugins.
3. Agent runs on a machine, then connects into another environment on the machine which it runs work on over a connection (e.g. ssh). this is what we do with VagrantPlaybook check where the VM is created dynamically
4. Use an isolated old jenkins server to connect to the Solaris agents, so they can run with the older version. This version would not be updated, so needs to be firewalled off from anything that might connect to it (exception for the main jenkins server so it can initiate and copy artifacts back)
5. The jobs are run on a schedule to perform a build and then initiate tests (potentially over ssh to a remote machine to ensure build/test machine isolation) The output can be put in a staging area that can be retrieved by a jenkins job which has the artifacts in a way that 'looks' like the existing pipelines.
6. Run a Solaris 11 host (where it is technically possible to have a JDK17 - third parties have patched it to work) and then use that to run the build (using a shared file system with the host to allow the jenkins agent's file workspace to be shared with the Solaris 10 environment)

Given that above, option 6 is probably the cleanest and give isolation of the Solaris 10 environment from the internet. However this would require updating our machines and adjusting the pipelines to be 'zone aware' (unless there is a jenkins plugin for this as there is for docker, but that seems optimistic!) The simplest solution, which we discussed today, is option 3, where we replace the existing build and test pipelines - for Solaris only - with a time pipeline that will ssh to the Solaris machine and run "git clone temurin-build; run make-adopt-build.farm.sh; copy build artifacts back and artifact them into the jenkins job. We will look at prototyping this.

[sxa]

Machine defined as dockerhost-azure-solaris-proxy (Currently another jenkins agent on the host machine) and will be prototyped with the jdk8u-solaris-x64-temurin-simple job

[sxa]

https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-solaris-x64-temurin-simple/
First job which ran green and collected some artifacts. Need to make sure it includes anything else required - testimages etc. and ideally ensure the artifact layout is the same as a normal build. At the moment it doesn't have the timestamps etc. in the filenames. (Although that's probably supplied from the parent job in the normal pipelines

[sxa]

Todo:

• Ensure metadata files are archived
• GPG signing
• Test case execution
• Ensure test .tap results can be parsed

[sxa]

I had a discussion with @smlambert today where I shared the scripts I'm using for this prototype activity. It sounds like what I've got will be feasible. There is a script that runs on the proxy machine which copies the test execution script (dotests.sh) across to the machine and runs it with two parameters to indicate the tests suite it should run (e.g. ./dotests.sh sanity openjdk) and will copy the output* directory back to the proxy workspace where it can be processed as normal.
There are a few options which we can decide on once an initial jenkins job (does not yet exist at the time of writing) has been verified to work:

1. Have one jenkins jobs that just sequentially runs all of the test suites (e.g. invokes dotests.sh multiple times)
2. Have a parameterised jenkins job that is invoked once for each test suite (sanity openjdk, extended openjdk etc.) so you have multiple separate runs the same jenkins job
3. Have individual test jobs for each, which is the most consistent with the existing way the jobs are structured and therefore is likely easier to process the results.

[smlambert]

Any of those 3 approaches will work @sxa (with 2. or 3. being preferred so that the TAP files map to the 9 top-level targets that are required for AQAvit verification).

[sxa]

I'm starting with option 2 as a proof-of-concept then we can adapt later if desired:

• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-solaris-x64-temurin-simpletest/

Note that the TEST_JDK_PARAMETER in that job does nothing - the place to download from is currently hardcoded in the dotests.sh script that it calls (Which is stored in ~solaris on the machine) Currently it's the latest artifact of jdk8u-solaris-x64-temurin-simpletest

(For reference to any infra people who need to read this - the two jenkins agents being used for the proxies are running as the same user running the vagrant box on the azure ubuntu dockerhost machine. They are not configured to restart themselves if the machine is rebooted)