CPU usage is pretty high on only one node and query is slow

[njalan]

Trino version is 394. I found that one node cpu is high out of 8 workers.
Below are the top thread take the cpu:
LANG=C ps -eT -o%cpu,pid,tid,ppid,comm | grep -v CPU | sort -n -r | head -20
35.5 22151 22165 1 C2 CompilerThre
33.9 22151 27230 1 C2 CompilerThre
13.5 22151 27713 1 C1 CompilerThre
10.6 22151 22192 1 GC Thread#9
10.6 22151 22188 1 GC Thread#5
10.6 22151 22186 1 GC Thread#3
10.6 22151 22185 1 GC Thread#2
10.6 22151 22184 1 GC Thread#1
10.5 22151 22195 1 GC Thread#12
10.5 22151 22194 1 GC Thread#11
10.5 22151 22193 1 GC Thread#10
10.5 22151 22191 1 GC Thread#8
10.5 22151 22190 1 GC Thread#7
10.5 22151 22189 1 GC Thread#6
10.5 22151 22187 1 GC Thread#4
10.5 22151 22153 1 GC Thread#0
9.8 22151 26104 1 20220921_130104
9.5 22151 24133 1 20220921_130055
9.4 22151 24165 1 20220921_130053
9.3 22151 22983 1 20220921_130058

top -Hp 22151

top - 20:55:19 up 26 days, 5:47, 1 user, load average: 117.77, 113.58, 63.38
Threads: 615 total, 32 running, 583 sleeping, 0 stopped, 0 zombie
%Cpu0 : 41.4 us, 50.4 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 47.4 us, 48.9 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st
%Cpu2 : 28.8 us, 60.6 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
%Cpu3 : 27.5 us, 64.9 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 30.8 us, 58.6 sy, 0.0 ni, 7.5 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st
%Cpu5 : 31.6 us, 61.7 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 28.4 us, 59.0 sy, 0.0 ni, 10.4 id, 0.0 wa, 0.0 hi, 2.2 si, 0.0 st
%Cpu7 : 26.9 us, 65.7 sy, 0.0 ni, 7.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 32.6 us, 59.1 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st
%Cpu9 : 41.7 us, 50.8 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 31.8 us, 60.6 sy, 0.0 ni, 5.3 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu11 : 36.4 us, 56.8 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 : 28.0 us, 63.6 sy, 0.0 ni, 6.1 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu13 : 35.6 us, 59.8 sy, 0.0 ni, 4.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 43.2 us, 48.5 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st
%Cpu15 : 50.8 us, 43.9 sy, 0.0 ni, 5.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 13186292+total, 74778560 free, 39433488 used, 17650880 buff/cache
KiB Swap: 3145724 total, 3115508 free, 30216 used. 91553264 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22492 root 20 0 124.8g 35.7g 23196 R 73.8 28.4 4:00.99 C2 CompilerThre
22165 root 20 0 124.8g 35.7g 23196 D 37.7 28.4 4:01.74 C2 CompilerThre
22245 root 20 0 124.8g 35.7g 23196 D 30.0 28.4 0:56.85 20220921_125457
25745 root 20 0 124.8g 35.7g 23196 R 23.8 28.4 0:23.35 C2 CompilerThre
24170 root 20 0 124.8g 35.7g 23196 S 21.5 28.4 0:09.65 20220921_125507
22262 root 20 0 124.8g 35.7g 23196 D 19.2 28.4 0:54.17 20220921_125427
22651 root 20 0 124.8g 35.7g 23196 R 18.5 28.4 3:49.26 C2 CompilerThre
24142 root 20 0 124.8g 35.7g 23196 D 14.6 28.4 0:07.63 20220921_125507
22271 root 20 0 124.8g 35.7g 23196 D 13.8 28.4 0:35.48 task-management
22265 root 20 0 124.8g 35.7g 23196 D 13.1 28.4 0:45.00 20220921_125508
24453 root 20 0 124.8g 35.7g 23196 D 13.1 28.4 0:22.24 20220921_125508
22599 root 20 0 124.8g 35.7g 23196 D 12.3 28.4 0:12.11 http-worker-280
22275 root 20 0 124.8g 35.7g 23196 S 11.5 28.4 0:39.22 Task-20220921_1
22568 root 20 0 124.8g 35.7g 23196 D 11.5 28.4 0:14.20 Task-20220921_1
22606 root 20 0 124.8g 35.7g 23196 R 11.5 28.4 0:35.40 task-notificati
25599 root 20 0 124.8g 35.7g 23196 R 11.5 28.4 0:01.82 20220921_125508
22282 root 20 0 124.8g 35.7g 23196 D 10.8 28.4 0:12.95 http-worker-104
22537 root 20 0 124.8g 35.7g 23196 D 10.8 28.4 0:14.16 http-worker-220
22552 root 20 0 124.8g 35.7g 23196 D 10.8 28.4 0:11.49 Task-20220921_1
22609 root 20 0 124.8g 35.7g 23196 R 10.8 28.4 0:35.95 Task-20220921_1
22241 root 20 0 124.8g 35.7g 23196 D 10.0 28.4 0:57.34 20220921_125507
22572 root 20 0 124.8g 35.7g 23196 D 10.0 28.4 0:11.63 http-worker-262
22642 root 20 0 124.8g 35.7g 23196 D 10.0 28.4 0:40.82 Task-20220921_1
22449 root 20 0 124.8g 35.7g 23196 D 9.2 28.4 0:13.31 Task-20220921_1
22491 root 20 0 124.8g 35.7g 23196 D 9.2 28.4 0:11.60 Task-20220921_1
22761 root 20 0 124.8g 35.7g 23196 D 9.2 28.4 0:06.14 page-buffer-cli
22764 root 20 0 124.8g 35.7g 23196 S 9.2 28.4 0:04.82 page-buffer-cli

Below is the jstack:

jstack 22151 | grep 57dc -A30

"C2 CompilerThread2" #9 daemon prio=9 os_prio=0 cpu=249980.15ms elapsed=684.40s tid=0x00007efca44bad70 nid=0x57dc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Compiling: 58698 % 4 org.eclipse.jetty.http.HttpParser::parseLine @ 801 (2072 bytes)

"http-worker-176" #176 prio=5 os_prio=0 cpu=12681.45ms elapsed=684.40s tid=0x00007efb74006610 nid=0x57dd runnable [0x00007efabf6f4000]
java.lang.Thread.State: RUNNABLE
at java.nio.HeapByteBuffer.([email protected]/HeapByteBuffer.java:64)
at java.nio.ByteBuffer.allocate([email protected]/ByteBuffer.java:363)
at sun.nio.cs.StreamDecoder.([email protected]/StreamDecoder.java:239)
at sun.nio.cs.StreamDecoder.forInputStreamReader([email protected]/StreamDecoder.java:95)
at java.io.InputStreamReader.([email protected]/InputStreamReader.java:130)
at java.nio.file.Files.newBufferedReader([email protected]/Files.java:2922)
at java.nio.file.Files.newBufferedReader([email protected]/Files.java:2955)
at jdk.internal.platform.CgroupUtil.lambda$readStringValue$1([email protected]/CgroupUtil.java:67)
at jdk.internal.platform.CgroupUtil$$Lambda$36/0x0000000800c5f300.run([email protected]/Unknown Source)
at java.security.AccessController.executePrivileged([email protected]/AccessController.java:807)
at java.security.AccessController.doPrivileged([email protected]/AccessController.java:569)
at jdk.internal.platform.CgroupUtil.readStringValue([email protected]/CgroupUtil.java:69)
at jdk.internal.platform.CgroupSubsystemController.getStringValue([email protected]/CgroupSubsystemController.java:65)
at jdk.internal.platform.CgroupSubsystemController.getLongValue([email protected]/CgroupSubsystemController.java:124)
at jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getLongValue([email protected]/CgroupV1Subsystem.java:189)
at jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getCpuShares([email protected]/CgroupV1Subsystem.java:245)
at jdk.internal.platform.CgroupMetrics.getCpuShares([email protected]/CgroupMetrics.java:75)
at com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad([email protected]/OperatingSystemImpl.java:86)
at com.sun.management.internal.OperatingSystemImpl.getCpuLoad([email protected]/OperatingSystemImpl.java:271)
at com.sun.management.OperatingSystemMXBean.getSystemCpuLoad([email protected]/OperatingSystemMXBean.java:153)
at io.trino.server.StatusResource.getStatus(StatusResource.java:92)
at jdk.internal.reflect.GeneratedMethodAccessor371.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:568)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)

[njalan]

Since only one node cpu is high, is there any issues from server side? What can be the issues? All servers are VMs.

[njalan]

Face the same issue on trino 394

[chengcxy]

Face the same issue on trino 410 ....

[hashhar]

most of the useful discussion is under #14996, closing in favor of that one.