Bazel server crashing abruptly in 7.4+ #24389

luispadron · 2024-11-19T16:23:29Z

Description of the bug:

Since upgrading to 7.4.0 and subsequently 7.4.1 we've been hit by a server crash during random CI (and local) builds:

Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/private/var/tmp/_bazel_build/a50428ec9d717925b6582741c513dab3/server/jvm.out')

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have not found a way to reliably reproduce this issue but were seeing about 1-3 of these per day on our ~300 machine CI fleet.

Which operating system are you running Bazel on?

macOS

What is the output of `bazel info release`?

7.4.1

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse HEAD` ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

There was a GC threading issue in 7.4.0 that was also causing a server crash but that was fixed in 7.4.1 so this might be another case of that issue or something new entirely. I do not see any

Any other information, logs, or outputs that you want to share?

Attached are the Bazel server JVM log from the build it terminated on along with the .ips file generated from the crash.

jvm_logs_and_ips.zip

...

Thread 51 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x0000000000000000   x3: 0x0000000000000000
    x4: 0x0000000000000001   x5: 0x00000001777affc8   x6: 0x000000000000002e   x7: 0x0000000000000000
    x8: 0xb48a3bbbfd395f71   x9: 0xb48a3bba8a426f71  x10: 0x0000000000000002  x11: 0x00000000fffffffd
   x12: 0x0000010000000000  x13: 0x0000000000000000  x14: 0x0000000000000000  x15: 0x0000000000000000
   x16: 0x0000000000000148  x17: 0x000000020168e4e8  x18: 0x0000000000000000  x19: 0x0000000000000006
   x20: 0x00000001777b3000  x21: 0x000000000000e707  x22: 0x00000001777b30e0  x23: 0x000000000000000a
   x24: 0x0000000102ade007  x25: 0x00000001305a5e00  x26: 0x0000000000000000  x27: 0x00000000000007d0
   x28: 0x00000000ffffffff   fp: 0x00000001777b1070   lr: 0x000000018f13dc20
    sp: 0x00000001777b1050   pc: 0x000000018f1055f0 cpsr: 0x40001000
   far: 0x0000000000000000  esr: 0x56000080  Address size fault

The text was updated successfully, but these errors were encountered:

meisterT · 2024-11-20T15:55:31Z

Do you still have the file /Users/build/.jenkins/workspace/cash-ios/ios-builder/s/c/hs_err_pid55320.log?

fmeum · 2024-11-20T16:05:29Z

Another JVM crash on macOS that may or may not be related: #23497

meisterT · 2024-11-20T16:31:10Z

It may just be that the best path forward is to use an upgraded version of the JDK in the hope that this will fix the crashes.

While looking at this, I realized that 78b8871 has not made the cut into Bazel 8 - was that intentional?

cc @Wyverald

luispadron · 2024-11-20T16:43:45Z

Do you still have the file /Users/build/.jenkins/workspace/cash-ios/ios-builder/s/c/hs_err_pid55320.log?

Unfortunately we don't collect these right now but I'm going to start doing that soon and report back once we get a log to share.

fmeum · 2024-11-20T16:47:26Z

While looking at this, I realized that 78b8871 has not made the cut into Bazel 8 - was that intentional?

It wasn't, but if this issue really only shows up in 7.4, then that commit may actually be what caused it. I haven't checked whether the JDK can be updated further.

ivan-golub · 2024-11-20T17:36:12Z

On our side we see consistent stream of SIGBUS/SIGSEGV on 7.2 running on Macs.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x000000041914bb80, pid=1927, tid=49963
#
# JRE version: OpenJDK Runtime Environment Zulu21.28+85-CA (21.0+35) (build 21+35)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.28+85-CA (21+35, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x000000041914bb80
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib:/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib/server:/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/ -Dfile.encoding=ISO-8859-1 -Duser.country= -Duser.language= -Duser.variant= -DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 -Xmx30g /Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/A-server.jar --max_idle_secs=10800 --shutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64 --install_base=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c --install_md5=f4e81c77e3f92a1e901b18f8adecf60c --output_base=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f --workspace_directory=/private/var/lib/jenkins/workspace/phantom --default_system_javabase=/nix/store/l57is3k03hrl5ds9vmnnzmvy1izx9fw7-zulu17.44.53-ca-jdk-17.0.8.1 --failure_detail_out=/Users/devicelab/Snapchat/Dev/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 --host_jvm_args=-Xmx30g --product_name=Bazel --option_sources=home_Urc::host_Ujvm_Uargs:/private/var/lib/jenkins/workspace/phantom/.bazelrc:output_Uuser_Uroot:/Users/devicelab/Snapchat/Dev/.cache/bzl/default.451.bazelrc:shutdown_Uon_Ulow_Usys_Umem:/Users/devicelab/Snapchat/Dev/.cache/bzl/default.451.bazelrc:system_Urc::workspace_Urc:

Host: "Mac13,2" arm64, 20 cores, 128G, Darwin 23.5.0, macOS 14.5 (23F79)
Time: Wed Nov 20 01:24:47 2024 GMT elapsed time: 44.782824 seconds (0d 0h 0m 44s)

---------------  T H R E A D  ---------------

Current thread (0x000000012b8f9600):  JavaThread "skyframe-evaluator-cpu-heavy-18" daemon [_thread_in_Java, id=49963, stack(0x0000000b0d064000,0x0000000b0d267000) (2060K)]

Stack: [0x0000000b0d064000,0x0000000b0d267000],  sp=0x0000000b0d265ab0,  free space=2054k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x000000041914bb80
j  java.lang.invoke.LambdaForm$MH+0x000000700075c800.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+59 java.base@21
J 14847 c2 net.starlark.java.eval.MethodDescriptor.call(Ljava/lang/Object;[Ljava/lang/Object;Lnet/starlark/java/eval/Mutability;)Ljava/lang/Object; (373 bytes) @ 0x0000000111bc90cc [0x0000000111bc8f40+0x000000000000018c]
J 16212 c2 net.starlark.java.eval.BuiltinFunction.fastcall(Lnet/starlark/java/eval/StarlarkThread;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (53 bytes) @ 0x00000001113f46c0 [0x00000001113f4600+0x00000000000000c0]
J 16297 c2 net.starlark.java.eval.Starlark.fastcall(Lnet/starlark/java/eval/StarlarkThread;Ljava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (143 bytes) @ 0x00000001114f921c [0x00000001114f9180+0x000000000000009c]
J 18557 c2 net.starlark.java.eval.Eval.evalCall(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/CallExpression;)Ljava/lang/Object; (579 bytes) @ 0x0000000111ea3838 [0x0000000111ea2a00+0x0000000000000e38]
J 18570 c2 net.starlark.java.eval.Eval$1Lambda.execClauses(I)V (291 bytes) @ 0x0000000111e87428 [0x0000000111e85080+0x00000000000023a8]
J 11383 c2 net.starlark.java.eval.Eval.evalComprehension(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/Comprehension;)Ljava/lang/Object; (72 bytes) @ 0x00000001116bd1d4 [0x00000001116bcec0+0x0000000000000314]
J 18557 c2 net.starlark.java.eval.Eval.evalCall(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/CallExpression;)Ljava/lang/Object; (579 bytes) @ 0x0000000111ea73b8 [0x0000000111ea2a00+0x00000000000049b8]
J 6397 c2 net.starlark.java.eval.Eval.eval(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/Expression;)Ljava/lang/Object; (322 bytes) @ 0x000000011101478c [0x00000001110145c0+0x00000000000001cc]
J 7846 c2 net.starlark.java.eval.Eval.execStatements(Lnet/starlark/java/eval/StarlarkThread$Frame;Ljava/util/List;Z)Lnet/starlark/java/syntax/TokenKind; (171 bytes) @ 0x0000000111201f80 [0x0000000111201d00+0x0000000000000280]
J 10349 c2 net.starlark.java.eval.Eval.execIf(Lnet/starlark/java/eval/StarlarkThread$Frame;Lnet/starlark/java/syntax/IfStatement;)Lnet/starlark/java/syntax/TokenKind; (47 bytes) @ 0x00000001114eb350 [0x00000001114eb100+0x0000000000000250]
J 7846 c2 net.starlark.java.eval.Eval.execStatements(Lnet/starlark/java/eval/StarlarkThread$Frame;Ljava/util/List;Z)Lnet/starlark/java/syntax/TokenKind; (171 bytes) @ 0x000000011120213c [0x0000000111201d00+0x000000000000043c]
J 18543 c2 net.starlark.java.eval.StarlarkFunction.fastcall(Lnet/starlark/java/eval/StarlarkThread;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (127 bytes) @ 0x0000000111e388b4 [0x0000000111e38640+0x0000000000000274]
J 16297 c2 net.starlark.java.eval.Starlark.fastcall(Lnet/starlark/java/eval/StarlarkThread;Ljava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (143 bytes) @ 0x00000001114f921c [0x00000001114f9180+0x000000000000009c]
J 19980 c1 com.google.devtools.build.lib.analysis.starlark.StarlarkRuleConfiguredTargetUtil.evalRule(Lcom/google/devtools/build/lib/analysis/RuleContext;Lcom/google/devtools/build/lib/packages/RuleClass;)Ljava/lang/Object; (189 bytes) @ 0x000000010a4cf210 [0x000000010a4cef40+0x00000000000002d0]
J 17167 c1 com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(Lcom/google/devtools/build/lib/analysis/AnalysisEnvironment;Lcom/google/devtools/build/lib/packages/Rule;Lcom/google/devtools/build/lib/analysis/config/BuildConfigurationValue;Lcom/google/devtools/build/lib/skyframe/ConfiguredTargetKey;Lcom/google/devtools/build/lib/util/OrderedSetMultimap;Lcom/google/devtools/build/lib/analysis/config/ConfigConditions;Lcom/google/devtools/build/lib/analysis/ToolchainCollection;Lcom/google/devtools/build/lib/collect/nestedset/NestedSet;Lcom/google/devtools/build/lib/analysis/ExecGroupCollection$Builder;)Lcom/google/devtools/build/lib/analysis/ConfiguredTarget; (519 bytes) @ 0x000000010a6d6368 [0x000000010a6d3d40+0x0000000000002628]
J 18274 c2 com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(Lcom/google/devtools/build/lib/analysis/AnalysisEnvironment;Lcom/google/devtools/build/lib/actions/ArtifactFactory;Lcom/google/devtools/build/lib/packages/Target;Lcom/google/devtools/build/lib/analysis/config/BuildConfigurationValue;Lcom/google/devtools/build/lib/skyframe/ConfiguredTargetKey;Lcom/google/devtools/build/lib/util/OrderedSetMultimap;Lcom/google/devtools/build/lib/analysis/config/ConfigConditions;Lcom/google/devtools/build/lib/analysis/ToolchainCollection;Lcom/google/devtools/build/lib/collect/nestedset/NestedSet;Lcom/google/devtools/build/lib/analysis/ExecGroupCollection$Builder;)Lcom/google/devtools/build/lib/analysis/ConfiguredTarget; (414 bytes) @ 0x0000000111e55798 [0x0000000111e55700+0x0000000000000098]
J 18239 c2 com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(Lcom/google/devtools/build/lib/skyframe/SkyframeBuildView;Lcom/google/devtools/build/skyframe/SkyFunction$Environment;Lcom/google/devtools/build/lib/analysis/TargetAndConfiguration;Lcom/google/devtools/build/lib/skyframe/ConfiguredTargetKey;Lcom/google/devtools/build/lib/util/OrderedSetMultimap;Lcom/google/devtools/build/lib/analysis/config/ConfigConditions;Lcom/google/devtools/build/lib/analysis/ToolchainCollection;Lcom/google/devtools/build/lib/analysis/ExecGroupCollection$Builder;Lcom/google/devtools/build/lib/collect/nestedset/NestedSet;)Lcom/google/devtools/build/lib/analysis/ConfiguredTargetValue; (395 bytes) @ 0x0000000111e0486c [0x0000000111e04180+0x00000000000006ec]
j  com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(Lcom/google/devtools/build/skyframe/SkyKey;Lcom/google/devtools/build/skyframe/SkyFunction$Environment;)Lcom/google/devtools/build/skyframe/SkyValue;+414
J 16324 c2 com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run()V (1927 bytes) @ 0x00000001116f89bc [0x00000001116f7b40+0x0000000000000e7c]

or

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000001064ae91c, pid=46540, tid=14083
#
# JRE version: OpenJDK Runtime Environment Zulu21.28+85-CA (21.0+35) (build 21+35)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.28+85-CA (21+35, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0x3f291c]  G1ConcurrentMark::mark_in_bitmap(unsigned int, oopDesc*)+0x3c
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/rules_xcodeproj.noindex/build_output_base/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib:/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/embedded_tools/jdk/lib/server:/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/ -Dfile.encoding=ISO-8859-1 -Duser.country= -Duser.language= -Duser.variant= -DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 -Xmx30g /Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c/A-server.jar --max_idle_secs=10800 --shutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/Users/ci/.cache/bazel/arm64 --install_base=/Users/ci/.cache/bazel/arm64/install/f4e81c77e3f92a1e901b18f8adecf60c --install_md5=f4e81c77e3f92a1e901b18f8adecf60c --output_base=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/rules_xcodeproj.noindex/build_output_base --workspace_directory=/private/var/lib/jenkins/workspace/phantom --default_system_javabase=/Library/Java/JavaVirtualMachines/temurin-23.jdk/Contents/Home --failure_detail_out=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/rules_xcodeproj.noindex/build_output_base/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 --host_jvm_args=-Xmx30g --host_jvm_args=-Xdock:name=/Applications/Xcode16.0_16A242d.app/Contents/Developer --product_name=Bazel --option_sources=home_Urc::host_Ujvm_Uargs::output_Ubase::output_Uuser_Uroot:/Users/ci/.cache/bzl/default.451.bazelrc:shutdown_Uon_Ulow_Usys_Umem:/Users/ci/.cache/bzl/default.451.bazelrc:system_Urc::workspace_Urc:

Host: "Mac13,2" arm64, 20 cores, 128G, Darwin 23.5.0, macOS 14.5 (23F79)
Time: Wed Nov 20 08:28:59 2024 GMT elapsed time: 869.441221 seconds (0d 0h 14m 29s)

---------------  T H R E A D  ---------------

Current thread (0x0000000135a04190):  WorkerThread "G1 Conc#0"      [id=14083, stack(0x000000016baa4000,0x000000016bca7000) (2060K)]

Stack: [0x000000016baa4000,0x000000016bca7000],  sp=0x000000016bca4c30,  free space=2051k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0x3f291c]  G1ConcurrentMark::mark_in_bitmap(unsigned int, oopDesc*)+0x3c
V  [libjvm.dylib+0x3d8fc4]  G1CMTask::make_reference_grey(oopDesc*)+0x24
V  [libjvm.dylib+0x3f1da4]  void OopOopIterateDispatch<G1CMOopClosure>::Table::oop_oop_iterate<InstanceRefKlass, narrowOop>(G1CMOopClosure*, oopDesc*, Klass*)+0xe4
V  [libjvm.dylib+0x3ee6f4]  void G1CMTask::process_grey_task_entry<true>(G1TaskQueueEntry)+0x224
V  [libjvm.dylib+0x3e72f8]  G1CMTask::drain_local_queue(bool)+0x150
V  [libjvm.dylib+0x3ecf30]  G1CMTask::do_marking_step(double, bool, bool)+0x16c
V  [libjvm.dylib+0x3eeb4c]  G1CMConcurrentMarkingTask::work(unsigned int)+0xa8
V  [libjvm.dylib+0xae482c]  WorkerThread::run()+0x94
V  [libjvm.dylib+0xa28ef8]  Thread::call_run()+0xc8
V  [libjvm.dylib+0x865220]  thread_native_entry(Thread*)+0x158
C  [libsystem_pthread.dylib+0x6f94]  _pthread_start+0x88

siginfo: si_signo: 10 (SIGBUS), si_code: 1 (BUS_ADRALN), si_addr: 0x0000000936818df8

fmeum · 2024-11-20T22:34:01Z

Cc @cushon

cushon · 2024-11-20T22:43:27Z

I would try using the latest JDK update release and see if it still reproduces. If it does, and there's a repro, it would be good to report to https://bugreport.java.com/bugreport/crash.jsp

fmeum · 2024-11-21T10:22:47Z

@bazel-io fork 8.0.0

Work towards bazelbuild#23497 Work towards bazelbuild#24389

ivan-golub · 2024-11-21T17:53:47Z

Tried to update to java to zulu 21.38.21 in our fork (on top of 7.2.1)
Seeing similar errors

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000004d5eccef8, pid=61097, tid=195087
#
# JRE version: OpenJDK Runtime Environment Zulu21.28+85-CA (21.0+35) (build 21+35)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.28+85-CA (21+35, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x00000004d5eccef8
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED -Xverify:none -Djava.util.logging.config.file=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/embedded_tools/jdk/lib:/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/embedded_tools/jdk/lib/server:/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/ -Dfile.encoding=ISO-8859-1 -Duser.country= -Duser.language= -Duser.variant= -DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 -Xmx30g /Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9/A-server.jar --max_idle_secs=10800 --shutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/Users/ci/.cache/bazel/arm64 --install_base=/Users/ci/.cache/bazel/arm64/install/f188b2f86baffd6f7652f35ca90c6ae9 --install_md5=f188b2f86baffd6f7652f35ca90c6ae9 --output_base=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f --workspace_directory=/private/var/lib/jenkins/workspace/phantom --default_system_javabase=/nix/store/l57is3k03hrl5ds9vmnnzmvy1izx9fw7-zulu17.44.53-ca-jdk-17.0.8.1 --failure_detail_out=/Users/ci/.cache/bazel/arm64/200c81ba794154ad95d9a76cac3f836f/failure_detail.rawproto --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --host_jvm_args=-DBAZEL_BES_NUM_RETRIES_ON_RPC_FAILURE=14 --host_jvm_args=-Xmx30g --product_name=Bazel --option_sources=home_Urc::host_Ujvm_Uargs:/private/var/lib/jenkins/workspace/phantom/.bazelrc:output_Uuser_Uroot:/Users/ci/.cache/bzl/default.451.bazelrc:shutdown_Uon_Ulow_Usys_Umem:/Users/ci/.cache/bzl/default.451.bazelrc:system_Urc::workspace_Urc:

Host: "Mac13,2" arm64, 20 cores, 128G, Darwin 23.5.0, macOS 14.5 (23F79)
Time: Thu Nov 21 14:13:44 2024 GMT elapsed time: 49.220094 seconds (0d 0h 0m 49s)

---------------  T H R E A D  ---------------

Current thread (0x0000000b260b5000):  JavaThread "skyframe-evaluator-cpu-heavy-4" daemon [_thread_in_Java, id=195087, stack(0x0000000aff064000,0x0000000aff267000) (2060K)]

Wyverald · 2024-11-21T19:03:05Z

While looking at this, I realized that 78b8871 has not made the cut into Bazel 8 - was that intentional?

It was actually cherry-picked as 7a9e1de (on 11 Oct). If you're just looking at the GitHub UI, it doesn't tell you about cherry-picks, only whether the commit is actually on any branches.

Work towards #23497 Work towards #24389 Closes #24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd

Work towards bazelbuild#23497 Work towards bazelbuild#24389 Closes bazelbuild#24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd

Work towards #23497 Work towards #24389 Closes #24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd Commit 27a5e45 Co-authored-by: Fabian Meumertzheim <[email protected]>

meteorcloudy · 2024-11-26T15:40:26Z

Does this still happen with Bazel 8.0.0 rc4?

luispadron · 2024-11-26T17:34:42Z

@meteorcloudy We can't test with Bazel 8 for a while I imagine, there are few rule breakages that must be fixed.

Would it be possible to create a Bazel 7 release for this if we think it fixes the crash?

meteorcloudy · 2024-11-28T12:23:35Z

@luispadron Unfortunately, we are not planning 7.5 yet

luispadron · 2024-12-11T17:58:50Z

@meteorcloudy is there some workaround for folks on Bazel 7? We can't upgrade very easily to Bazel 8 and were seeing these crashes pretty frequently

meteorcloudy · 2024-12-12T10:53:03Z

@iancha1992 Let's start the 7.5 release

iancha1992 · 2024-12-12T16:56:57Z

@bazel-io fork 7.5.0

Work towards bazelbuild#23497 Work towards bazelbuild#24389 Closes bazelbuild#24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd

luispadron added type: bug untriaged labels Nov 19, 2024

luispadron assigned iancha1992 and satyanandak Nov 19, 2024

iancha1992 unassigned iancha1992 and satyanandak Nov 20, 2024

iancha1992 added the team-Rules-Java Issues for Java rules label Nov 20, 2024

bazel-io mentioned this issue Nov 21, 2024

[8.1.0] Bazel server crashing abruptly in 7.4+ #24429

Open

fmeum added a commit to fmeum/bazel that referenced this issue Nov 21, 2024

Update embedded JDK to 21.0.5

78dd643

Work towards bazelbuild#23497 Work towards bazelbuild#24389

fmeum mentioned this issue Nov 21, 2024

Update embedded JDK to 21.0.5 #24431

Closed

copybara-service bot pushed a commit that referenced this issue Nov 21, 2024

Update embedded JDK to 21.0.5

27a5e45

Work towards #23497 Work towards #24389 Closes #24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd

bazel-io pushed a commit to bazel-io/bazel that referenced this issue Nov 21, 2024

Update embedded JDK to 21.0.5

02f6d2a

Work towards bazelbuild#23497 Work towards bazelbuild#24389 Closes bazelbuild#24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd

bazel-io mentioned this issue Nov 21, 2024

[8.0.0] Update embedded JDK to 21.0.5 #24440

Merged

hvadehra added team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website and removed team-Rules-Java Issues for Java rules labels Nov 26, 2024

bazel-io mentioned this issue Dec 12, 2024

[7.5.0] Bazel server crashing abruptly in 7.4+ #24663

Open

bazel-io pushed a commit to bazel-io/bazel that referenced this issue Dec 13, 2024

Update embedded JDK to 21.0.5

b54dea2

Work towards bazelbuild#23497 Work towards bazelbuild#24389 Closes bazelbuild#24431. PiperOrigin-RevId: 698846634 Change-Id: I3fb2d46597b225833a918af9731665bb276322dd

bazel-io mentioned this issue Dec 13, 2024

[7.5.0] Update embedded JDK to 21.0.5 #24682

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel server crashing abruptly in 7.4+ #24389

Bazel server crashing abruptly in 7.4+ #24389

luispadron commented Nov 19, 2024

meisterT commented Nov 20, 2024

fmeum commented Nov 20, 2024

meisterT commented Nov 20, 2024

luispadron commented Nov 20, 2024

fmeum commented Nov 20, 2024

ivan-golub commented Nov 20, 2024

fmeum commented Nov 20, 2024

cushon commented Nov 20, 2024

fmeum commented Nov 21, 2024

ivan-golub commented Nov 21, 2024 •

edited

Loading

Wyverald commented Nov 21, 2024

meteorcloudy commented Nov 26, 2024

luispadron commented Nov 26, 2024 •

edited

Loading

meteorcloudy commented Nov 28, 2024

luispadron commented Dec 11, 2024

meteorcloudy commented Dec 12, 2024

iancha1992 commented Dec 12, 2024

Bazel server crashing abruptly in 7.4+ #24389

Bazel server crashing abruptly in 7.4+ #24389

Comments

luispadron commented Nov 19, 2024

Description of the bug:

Which category does this issue belong to?

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

meisterT commented Nov 20, 2024

fmeum commented Nov 20, 2024

meisterT commented Nov 20, 2024

luispadron commented Nov 20, 2024

fmeum commented Nov 20, 2024

ivan-golub commented Nov 20, 2024

fmeum commented Nov 20, 2024

cushon commented Nov 20, 2024

fmeum commented Nov 21, 2024

ivan-golub commented Nov 21, 2024 • edited Loading

Wyverald commented Nov 21, 2024

meteorcloudy commented Nov 26, 2024

luispadron commented Nov 26, 2024 • edited Loading

meteorcloudy commented Nov 28, 2024

luispadron commented Dec 11, 2024

meteorcloudy commented Dec 12, 2024

iancha1992 commented Dec 12, 2024

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse HEAD` ?

ivan-golub commented Nov 21, 2024 •

edited

Loading

luispadron commented Nov 26, 2024 •

edited

Loading