Some shuffle server pod crashlooping

[youjeongsue]

Hi. Uniffle team!

1. For some time now, when I start shuffle server, some pods do not run normally and go into a crashloopback state.

.

Completed pod log: The pod will be aborted after class path is. There are no other logs

Defaulted container "rss-shuffle-server" out of: rss-shuffle-server, init-data-dir (init), init-log-dir (init)
coordinator_conf: /data/rssadmin/rss/conf/coordinator.conf
server_conf: /data/rssadmin/rss/conf/server.conf
Using rss_env.sh: /data/rssadmin/rss/bin/rss-env.sh
Using Java from /opt/java/openjdk
Using Hadoop from /data/rssadmin/hadoop
Using RSS from /data/rssadmin/rss
Using RSS conf from /data/rssadmin/rss/conf
Using Hadoop conf from /data/rssadmin/hadoop/etc/hadoop
Write log file to /data/rssadmin/rss/logs
Write pid file to /data/rssadmin/rss
Shuffle Server JVM XMX size: 200G
Shuffle Server RSS_IP: xxx
Check process existence
class path is :/data/rssadmin/rss/jars/server/activation-1.1.1.jar:/data/rssadmin/rss/jars/server/animal-sniffer-annotations-1.19.jar:/data/rssadmin/rss/jars/server/annotations-4.1.1.4.jar:/data/rssadmin/rss/jars/server/checker-qual-3.12.0.jar:/data/rssadmin/rss/jars/server/commons-lang3-3.10.jar:/data/rssadmin/rss/jars/server/error_prone_annotations-2.10.0.jar:/data/rssadmin/rss/jars/server/failureaccess-1.0.1.jar:/data/rssadmin/rss/jars/server/grpc-api-
...
3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-submarine-3.2.1.jar

Running pod log: I can see normal shuffle server logs

Defaulted container "rss-shuffle-server" out of: rss-shuffle-server, init-data-dir (init), init-log-dir (init)
coordinator_conf: /data/rssadmin/rss/conf/coordinator.conf
server_conf: /data/rssadmin/rss/conf/server.conf
Using rss_env.sh: /data/rssadmin/rss/bin/rss-env.sh
Using Java from /opt/java/openjdk
Using Hadoop from /data/rssadmin/hadoop
Using RSS from /data/rssadmin/rss
Using RSS conf from /data/rssadmin/rss/conf
Using Hadoop conf from /data/rssadmin/hadoop/etc/hadoop
Write log file to /data/rssadmin/rss/logs
Write pid file to /data/rssadmin/rss
Shuffle Server JVM XMX size: 200G
Shuffle Server RSS_IP: ...
Check process existence
class path is :/data/rssadmin/rss/jars/server/activation-1.1.1.jar:/data/rssadmin/rss/jars/server/animal-sniffer-annotations-1.19.jar:/data/rssadmin/rss/jars/server/annotations-4.1.1.4.jar:/data/rssadmin/rss/jars/server/checker-qual-3.12.0.jar:/data/rssadmin/rss/jars/server/commons-lang3-3.10.jar:/data/rssadmin/rss/jars/server/error_prone_annotations-2.10.0.jar:/data/rssadmin/rss/jars/server/failureaccess-1.0.1.jar:/data/rssadmin/rss/jars/server/grpc-api-
...
3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.2.1.jar:/data/rssadmin/hadoop/share/hadoop/yarn/hadoop-yarn-submarine-3.2.1.jar
shuffle server pid:COMMAND PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java     62 rssadmin  651u  IPv6 3048802673      0t0  TCP *:19997 (LISTEN) is alive
shuffle server pid:COMMAND PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java     62 rssadmin  651u  IPv6 3048802673      0t0  TCP *:19997 (LISTEN) is alive
shuffle server pid:COMMAND PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java     62 rssadmin  651u  IPv6 3048802673      0t0  TCP *:19997 (LISTEN) is alive

2. And there's another problem: the number of shuffle servers doesn't adjust to what I want.
   In the photo above, I set replica to 10, but only 8 appeared. When I look at the rss-controller logs, it looks like all 10 have been updated.

[jerqi]

Maybe the error message is in the log of standard output. Could you check whether the config option Xmx in the rss-env.sh is right?

[youjeongsue]

Here's the rss-env.sh from the running Pod. I can't see Xmx.

#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

set -o pipefail
set -o nounset   # exit the script if you try to use an uninitialised variable
set -o errexit   # exit the script if any statement returns a non-true return value

HADOOP_HOME="/data/rssadmin/hadoop"
RUNNER="${JAVA_HOME}/bin/java"
JPS="${JAVA_HOME}/bin/jps"

# RSS_HOME, RSS home directory (Default: parent directory of the script)
# RSS_CONF_DIR, RSS configuration directory (Default: ${RSS_HOME}/conf)
# HADOOP_CONF_DIR, Hadoop configuration directory (Default: ${HADOOP_HOME}/etc/hadoop)
# RSS_PID_DIR, Where the pid file is stored (Default: ${RSS_HOME})
# RSS_LOG_DIR, Where log files are stored (Default: ${RSS_HOME}/logs)
# RSS_IP, IP address Shuffle Server binds to on this node (Default: first non-loopback ipv4)

[jerqi]

incubator-uniffle/bin/rss-env.sh

Line 26 in 5a89609

XMX_SIZE="80g" # Shuffle Server JVM XMX size

[youjeongsue]

Thanks. I'll put it in and try again! One question, why is rss-env.sh different from the file for Kubernetes below?

https://github.com/apache/incubator-uniffle/blob/5a89609dc2b6e5167f8373276d90793600559b6a/deploy/kubernetes/docker/rss-env.sh