System unavailable: build-osuosl-aix72-ppc64-2

[sxa]

• Please put the system name in the title of this issue. https://ci.adoptium.net/computer/build-osuosl-aix72-ppc64-2/

• Link to any log file showing the problem:

• Please describe the issue:
  Resource issue encountered when running a build in jenkins for the JDK11 release build (Restricted access, but it looks like the regular JDK11 build has had issues on that system recently)

Marked offline for now

[aixtools]

Rather too hard to read - but I'll guess there is something running in the background that causes the number of processes to be exceeded. I'll also look at the swap space:

• Failed log excerpt

23:48:38      1501-201: (W) Maximum number of common component diagnostics, 10  has been exceeded.
23:48:38  Compiling 8 files for jdk.net
23:48:38  Compiling 1 files for jdk.pack
23:48:38  2 warnings generated.
23:48:38  Compiling 25 files for jdk.sctp
23:48:38  Compiling 593 files for jdk.scripting.nashorn
23:48:39  Compiling 9 files for jdk.unsupported
23:48:39  In file included from ./src/hotspot/share/compiler/compilerDirectives.cpp:26:
23:48:39  In file included from ./src/hotspot/share/ci/ciMethod.hpp:29:
23:48:39  In file included from ./src/hotspot/share/ci/ciInstanceKlass.hpp:28:
23:48:39  In file included from ./src/hotspot/share/ci/ciConstantPoolCache.hpp:28:
23:48:39  In file included from ./src/hotspot/share/memory/resourceArea.hpp:29:
23:48:39  In file included from ./src/hotspot/share/runtime/thread.hpp:31:
23:48:39  In file included from ./src/hotspot/share/gc/shared/threadLocalAllocBuffer.hpp:29:
23:48:39  In file included from ./src/hotspot/share/oops/typeArrayOop.hpp:28:
23:48:39  In file included from ./src/hotspot/share/oops/arrayOop.hpp:28:
23:48:39  In file included from ./src/hotspot/share/memory/universe.hpp:29:
23:48:39  In file included from ./src/hotspot/share/runtime/handles.hpp:29:
23:48:39  In file included from ./src/hotspot/share/oops/oop.hpp:31:
23:48:39  ./src/hotspot/share/oops/metadata.hpp:61:9: warning: 'this' pointer cannot be null in well-defined C++ code; comparison may be assumed to always evaluate to false [-Wtautological-undefined-compare]
23:48:39      if (this == NULL)
23:48:39          ^~~~    ~~~~
23:48:39  ./src/hotspot/share/oops/metadata.hpp:67:9: warning: 'this' pointer cannot be null in well-defined C++ code; comparison may be assumed to always evaluate to false [-Wtautological-undefined-compare]
23:48:39      if (this == NULL)
23:48:39          ^~~~    ~~~~
23:48:39  In file included from ./src/hotspot/share/compiler/compilerOracle.cpp:28:
23:48:39  In file included from ./src/hotspot/share/compiler/methodMatcher.hpp:29:
23:48:39  In file included from ./src/hotspot/share/runtime/handles.hpp:29:
23:48:39  In file included from ./src/hotspot/share/oops/oop.hpp:31:
23:48:39  ./src/hotspot/share/oops/metadata.hpp:61:9: warning: 'this' pointer cannot be null in well-defined C++ code; comparison may be assumed to always evaluate to false [-Wtautological-undefined-compare]
23:48:39      if (this == NULL)
23:48:39          ^~~~    ~~~~
23:48:39  ./src/hotspot/share/oops/metadata.hpp:67:9: warning: 'this' pointer cannot be null in well-defined C++ code; comparison may be assumed to always evaluate to false [-Wtautological-undefined-compare]
23:48:39      if (this == NULL)
23:48:39          ^~~~    ~~~~
23:48:40  Compiling 15 files for jdk.zipfs
23:48:40  2 warnings generated.
23:48:40  2 warnings generated.
23:48:40  Compiling 131 files for BUILD_NASGEN
23:48:41  Creating support/native/java.base/libverify/static/libverify.a from 2 file(s)
23:48:41  Creating support/native/java.base/libjava/static/libjava.a from 58 file(s)
23:48:41  Creating support/native/java.base/libfdlibm/static/libfdlibm.a from 57 file(s)
23:48:41  Creating support/native/java.base/libzip/static/libzip.a from 5 file(s)
23:48:41  Creating support/native/java.base/libjimage/static/libjimage.a from 6 file(s)
23:48:41  Creating support/native/java.base/libjli/static/libjli.a from 9 file(s)
23:48:41  Creating support/native/java.base/libjli_static/static/libjli_static.a from 9 file(s)
23:48:41  Creating support/native/java.base/libnet/static/libnet.a from 21 file(s)
23:48:41  Creating support/native/java.base/libnio/static/libnio.a from 21 file(s)
23:48:41  Creating support/native/java.base/libjsig/static/libjsig.a from 1 file(s)
23:48:42  Creating support/native/java.instrument/libinstrument/static/libinstrument.a from 12 file(s)
23:48:42  Creating support/native/java.management/libmanagement/static/libmanagement.a from 9 file(s)
23:48:44  Creating support/native/java.smartcardio/libj2pcsc/static/libj2pcsc.a from 2 file(s)
23:48:45  /opt/freeware/bin/bash: fork: retry: Resource temporarily unavailable
23:48:45      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_gc,LogTag::type::_verify,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::error(const char *, ...) could not be inlined into VerifyAllBlksClosure::do_blk(HeapWord *).
23:48:45      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_gc,LogTag::type::_verify,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::error(const char *, ...) could not be inlined into VerifyAllBlksClosure::do_blk(HeapWord *).
23:48:45      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_gc,LogTag::type::_freelist,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into AllocationStats::compute_desired(unsigned long, float, float, float).
23:48:45      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_gc,LogTag::type::_plab,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into CompactibleFreeListSpaceLAB::retire(int).
23:48:45      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_gc,LogTag::type::_plab,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into CompactibleFreeListSpaceLAB::compute_desired_plab_size().
23:48:45      1500-029: (W) WARNING: subprogram LogTargetImpl<LogLevel::type::Debug,LogTag::type::_gc,LogTag::type::_freelist,LogTag::type::_census,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::print(const char *, ...) could not be inlined into CompactibleFreeListSpace::printFLCensus(unsigned long) const.
23:48:45      1500-029: (W) WARNING: subprogram LogTargetImpl<LogLevel::type::Debug,LogTag::type::_gc,LogTag::type::_freelist,LogTag::type::_census,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::print(const char *, ...) could not be inlined into CompactibleFreeListSpace::printFLCensus(unsigned long) const.
23:48:45      1500-029: (W) WARNING: subprogram LogTargetImpl<LogLevel::type::Debug,LogTag::type::_gc,LogTag::type::_freelist,LogTag::type::_census,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::print(const char *, ...) could not be inlined into CompactibleFreeListSpace::printFLCensus(unsigned long) const.
23:48:45      1500-029: (W) WARNING: subprogram LogTargetImpl<LogLevel::type::Debug,LogTag::type::_gc,LogTag::type::_freelist,LogTag::type::_census,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::print(const char *, ...) could not be inlined into CompactibleFreeListSpace::printFLCensus(unsigned long) const.
23:48:45      1501-201: (W) Maximum number of common component diagnostics, 10  has been exceeded.
23:48:45  Creating support/native/jdk.crypto.ec/libsunec/static/libsunec.a from 28 file(s)
23:48:45  Creating support/native/jdk.jdwp.agent/libdt_socket/static/libdt_socket.a from 2 file(s)
23:48:45  Creating support/native/jdk.jdwp.agent/libjdwp/static/libjdwp.a from 42 file(s)
23:48:46  1586-495 (U) SYSTEM LIMIT EXCEEDED: Too many processes are active.
23:48:46  gmake[3]: *** [CoreLibraries.gmk:101: /home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk11u/jdk11u-release-aix-ppc64-temurin/workspace/build/src/build/aix-ppc64-normal-server-release/support/native/java.base/libjava/static/BootLoader.o] Error 1
23:48:46  gmake[2]: *** [make/Main.gmk:228: java.base-static-libs] Error 2

• same bit from successful system (seems gmake is running in parallel, so line sequences are not identical)

11:52:20  Compiling 8 files for jdk.net
11:52:22  Compiling 1 files for jdk.pack
11:52:24  Compiling 188 files for jdk.rmic
11:52:26      1500-029: (W) WARNING: subprogram FormatBuffer<256>::FormatBuffer<256>(const char *, ...) could not be inlined into ClassFileParser::fix_anonymous_class_name(Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_preview,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::fill_instance_klass(InstanceKlass *, bool, Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_resolve,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::fill_instance_klass(InstanceKlass *, bool, Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_resolve,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::fill_instance_klass(InstanceKlass *, bool, Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_fingerprint,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::create_instance_klass(bool, Thread *).
11:52:26      1500-029: (W) WARNING: subprogram FormatBuffer<256>::FormatBuffer<256>(const char *, ...) could not be inlined into ClassFileParser::fix_anonymous_class_name(Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_preview,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::fill_instance_klass(InstanceKlass *, bool, Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_resolve,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::fill_instance_klass(InstanceKlass *, bool, Thread *).
11:52:26      1500-029: (W) WARNING: subprogram LogImpl<LogTag::type::_class,LogTag::type::_resolve,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG,LogTag::type::__NO_TAG>::write(const char *, ...) could not be inlined into ClassFileParser::fill_instance_klass(InstanceKlass *, bool, Thread *).
11:52:26      1501-201: (W) Maximum number of common component diagnostics, 10  has been exceeded.
11:52:30  Compiling 593 files for jdk.scripting.nashorn
11:52:32  Note: Some input files use or override a deprecated API.
11:52:32  Note: Recompile with -Xlint:deprecation for details.
11:52:33  Compiling 25 files for jdk.sctp
11:52:35  Compiling 131 files for BUILD_NASGEN
11:52:35  Compiling 9 files for jdk.unsupported
11:52:40  Note: Some input files use or override a deprecated API that is marked for removal.
11:52:40  Note: Recompile with -Xlint:removal for details.
11:52:42  Running nasgen

[aixtools]

• fyi: build1 has a slightly smaller proc/cpu configuration than build2
• filesystem sizes are identical, usage is nearly identical - except /home on build2 is basically empty.

[aixtools]

ok. something I thought was addressed in the playbooks - but obviously not. ALTHOUGH both build1 and build2 are at the wrong value (now):

dsh-adopt lsattr -El sys0 -a maxuproc
adopt01:
maxuproc 512 Maximum number of PROCESSES allowed per user True
==============
adopt02:
maxuproc 128 Maximum number of PROCESSES allowed per user True
==============
adopt03:
maxuproc 512 Maximum number of PROCESSES allowed per user True
==============
adopt04:
maxuproc 512 Maximum number of PROCESSES allowed per user True
==============
adopt05:
maxuproc 128 Maximum number of PROCESSES allowed per user True
==============
adopt06:
maxuproc 128 Maximum number of PROCESSES allowed per user True
==============
adopt07:
maxuproc 128 Maximum number of PROCESSES allowed per user True
==============
adopt08:
maxuproc 128 Maximum number of PROCESSES allowed per user True
==============
adopt10:
maxuproc 128 Maximum number of PROCESSES allowed per user True
==============

[aixtools]

OK. I got a different error - needed to set OBJECT_MODE=64 I expect.

=== Output from failing command(s) repeated here ===
* For target support_native_java.base_libverify_static_BUILD_LIBVERIFY_link:
ar: 0707-126 /home/jenkins/workspace/temurin-build/workspace/build/src/build/aix-ppc64-normal-server-release/support/native/java.base/libverify/static/check_code.o is not valid with the current object file mode.
        Use the -X option to specify the desired object mode.

[aixtools]

Manually updated the maxuproc setting to all nodes, and have recycled all jenkins agents to be sure the new setting is recognized by jenkins (should be independent, but wanted to be sure).