System unavailable: x64&&solaris&&build

[sxa]

• Please put the system name in the title of this issue: All nodes with that label

• Link to any log file showing the problem: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-solaris-x64-temurin/520/console

21:02:47  No active node matches this label: build&&solaris&&x64
[Pipeline] echo
21:02:47  Will check again periodically until a node labelled build&&solaris&&x64 comes online, or 5 minutes (ACTIVE_NODE_TIMEOUT) has passed.
[Pipeline] sleep
21:02:47  Sleeping for 1 min 0 sec
[Pipeline] sleep
21:03:47  Sleeping for 1 min 0 sec
[Pipeline] sleep
21:04:47  Sleeping for 1 min 0 sec
[Pipeline] sleep
21:05:47  Sleeping for 1 min 0 sec
[Pipeline] sleep
21:06:47  Sleeping for 1 min 0 sec
[Pipeline] }
[Pipeline] // stage
[Pipeline] echo
21:07:47  Execution error: java.lang.Exception: No node matching this label became active prior to the timeout: build&&solaris&&x64
[Pipeline] echo
21:07:47  java.lang.Exception: No node matching this label became active prior to the timeout: build&&solaris&&x64

• Please describe the issue:
  Solaris VM unavailable - build failed

[sxa]

@steelhead31 can you take a look at this please? I don't think this is the first time recently it's gone down, but I wanted to create this issue so we can close/reopen it as required to track how often it might be occurring.

FYI @adamfarley since this will show up during the triage of this week's runs.

[steelhead31]

These 2 vagrant VMs are down, due to the disk filling up a couple of days ago. The disk space issue has been resolved, but these containers have not been restarted. I'll return them to service. Worth noting that if any out of disk issues occur on this host, the 2 vagrant boxes should be checked, I'll look to add some monitoring to nagios specifically for these 2 vagrant boxes to help alert if they are not up in the future

[steelhead31]

Both nodes returned to service... investigating possibilty of alerting if these boxes are not up in nagios

[steelhead31]

Nagios updated for the host machine to check and alert on the status of both vagrant boxes..

[sxa]

Both nodes returned to service... investigating possibilty of alerting if these boxes are not up in nagios

Good shout, especially if this becomes a frequent occurrance. Thanks.

These 2 vagrant VMs are down, due to the disk filling up a couple of days ago.

They're each on their own separate filesystem on the host aren't they? Is this an issue on the machine where something is exhausting it inside the VM, or do we need to expand the file systems that host the machine? Noting that the host filesystems were recently modified under #3796 (Possibly reduced in capacity from 60GiB to 50GiB)

[steelhead31]

IIRC it was more the root system that filled up that caused the downtime, the alerts should allow us to gauge why they fail next time, and then if needed expand their individual filesystems.

[sxa]

(Previous related issue for reference: #3819)