Refactor Jenkins pipelines

[henrirosten]

Refactor Jenkins pipelines as outlined in https://ssrc.atlassian.net/wiki/x/kwItQw#Jenkins-pipelines.

Example instance deployed with the changes from this PR is running in: https://ghaf-jenkins-controller-henrirosten.northeurope.cloudapp.azure.com/.

Also see the corresponding change in the ghaf-infra: tiiuae/ghaf-infra#178.

After this change, there will be three pipelines:

• ghaf-pre-merge-pipeline
  • Runs pre-merge for Ghaf PRs as described in https://ssrc.atlassian.net/wiki/x/dwB5Qg
  • Pushes to binary cache, but does not store any other artifacts
• ghaf-main-pipeline
  • Runs after each merge to Ghaf main branch
  • Builds selected targets, storing the (selected) build results as artifacts
• ghaf-nightly-pipeline
  • Runs nightly if there are any new commits on Ghaf main since the previous nightly run
  • Builds selected targets, storing the (selected) build results as artifacts
  • In addition, runs the 'scs' jobs: Provenance, SBOM, Vulnxscan

I would like to keep the number of targets executed on each pipeline to as small as possible, so we would not waste resources building or storing artifacts from targets no-one needs. Feel free to propose other targets if the set of targets now included in this PR is missing a target we need to build, keeping the pipeline intent in mind.

Currently, the content of the 'nightly' pipeline is the same as 'release' pipeline except 'release' pipeline would be manually triggered. Since the content would be the same, I did not include a separate 'release' pipeline for now.

Artifacts are stored in a hierarchy similar to that in the example instance at: https://ghaf-jenkins-controller-henrirosten.northeurope.cloudapp.azure.com/artifacts/. The main changes from earlier are:

• Top-level indicates the pipeline name
• Second-level indicates the build ID as well as the Ghaf target commit
• Third-level indicates the flake target
• SCS job results are stored under each target in 'scs' directory

This PR introduces utils.groovy to collect common functions used in all pipelines. Currently, it provides helpers to run the nix builds and SCS jobs storing the artifacts in a hierarchy described above. It attempts to simplify the calling convention for provenance and other SCS jobs, reducing the need for repetition in the pipeline scripts. It also attempts to capture more accurate build start and end times for each target provenance.

Finally, this PR changes the pre-merge pipeline so the selected targets are build in parallel. An example of the new pre-merge-pipeline is running in https://ghaf-jenkins-controller-dev.northeurope.cloudapp.azure.com/job/ghaf-pre-merge-pipeline/. Earlier evaluation errors seem to have been caused by the parallel evaluation of the same targets, which should not happen after setting the number of executors to one in: tiiuae/ghaf-infra#178. I would still like to trial running this in the pre-merge pipeline for a while to make sure there are no remaining issues, and then later think of having similar optimizations in other pipelines also.
^^^ The parallel build optimization on pre-merge pipeline had to be removed, see: #29 (comment)

[mnokka]

This is logical step ahead!

Some clarifications for the story; The usage of "normal Jenkins artifacts" have been been replaced with "azure blob" usage (works as normal artifacts from the user point of view). There is coming "release build" pipeline with yet-more targets added. Main CI build pipeline needs also some yet-more targets later.

The "utils" section is ok, specially as ties together vulnxscan and its conversions to cleaner txt format (or whatever we wish to add later)

[mnokka]

ok, go

[ktusawrk]

bpmp targets should be added to at least one pipeline. No need to run SCS jobs at this point.

• x86
  • nvidia-jetson-orin-nx-debug-bpmp-from-x86_64.x86_64-linux
  • nvidia-jetson-orin-agx-debug-bpmp-from-x86_64.x86_64-linux
• aarch64
  • nvidia-jetson-orin-nx-debug-bpmp.aarch64-linux
  • nvidia-jetson-orin-agx-debug-bpmp.aarch64-linux

[henrirosten]

Removed 'ghaf-merge-main-pipeline', which was included to this PR as a result of incorrect rebase on main.

Added 'bpmp' targets from the removed pipeline - which @ktusawrk also mentioned above - to the new nightly pipeline so that the 'bpmp' targets are built nightly, but no scs-jobs are run against them.

[ktusawrk]

Let's deploy these pipelines and see how they perform.

[henrirosten]

I had to remove the parallel build steps, and revert to running nix build steps sequentially.

Apparently, builds triggered in parallel on jenkins-controller VM still cause issues with nix build/evaulation as manifested e.g. in https://ghaf-jenkins-controller-dev.northeurope.cloudapp.azure.com/job/ghaf-pre-merge-pipeline/274/console