[Bug] [Worker] RemoteShell tasks may have memory leak issue #15851

simmonn · 2024-04-15T11:48:35Z

Search before asking

I had searched in the issues and found no similar issues.

What happened

Version:3.2.0
jvm options: -Xmx3g -Xms3g -Xmn1g
jdk:amazon-corretto-11.0.19.7.1-linux-x86_64
My project has 60 RemoteShell scheduled tasks(executing php commands). After running for a while, there are frequent Full GC occurrences, causing all tasks to fail, leading to false deadlocks on the worker nodes.So I had to change remoteshell to shell task which command using ssh -i id_rsa ''.

Apart from some error logs, I also noticed WARN logs with NPE (NullPointerException) occurring every time a task is executed.
[WARN] 2024-04-10 04:01:27.782 +0800 org.apache.sshd.client.session.ClientSessionImpl:[618] - [WorkflowInstance-0][TaskInstance-0] - exceptionCaught(ClientSessionImpl[root@/172.19.23.121:22])[state=Opened] NullPointerException: No customized heartbeat handler registered

here is error log:

[ERROR] 2024-04-10 04:01:01.146 +0800 org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable:[181] - [WorkflowInstance-72475][TaskInstance-74145] - Task execute failed, due to meet an exception
org.apache.dolphinscheduler.plugin.task.api.TaskException: Execute shell task error
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteShellTask.handle(RemoteShellTask.java:110)
        at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:57)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:175)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.dolphinscheduler.plugin.task.api.TaskException: Remote shell task error
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.run(RemoteExecutor.java:101)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteShellTask.handle(RemoteShellTask.java:104)
        ... 9 common frames omitted
Caused by: org.apache.dolphinscheduler.plugin.task.api.TaskException: SSH connection failed
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.getSession(RemoteExecutor.java:83)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.runRemote(RemoteExecutor.java:208)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.getTaskPid(RemoteExecutor.java:184)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.run(RemoteExecutor.java:91)
        ... 10 common frames omitted
Caused by: org.apache.sshd.common.SshException: DefaultConnectFuture[root@/172.19.23.121:22]: Failed to get operation result within specified timeout: 5000
        at org.apache.sshd.common.future.AbstractSshFuture.formatExceptionMessage(AbstractSshFuture.java:185)
        at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:111)
        at org.apache.sshd.client.future.DefaultConnectFuture.verify(DefaultConnectFuture.java:42)
        at org.apache.sshd.client.future.DefaultConnectFuture.verify(DefaultConnectFuture.java:34)
        at org.apache.dolphinscheduler.plugin.datasource.ssh.SSHUtils.getSession(SSHUtils.java:42)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.getSession(RemoteExecutor.java:78)
        ... 13 common frames omitted
[INFO] 2024-04-10 04:01:02.874 +0800 org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteShellTask:[118] - [WorkflowInstance-72475][TaskInstance-74145] - kill remote task dolphinscheduler-remoteshell-74145
[ERROR] 2024-04-10 04:01:02.875 +0800 org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable:[140] - [WorkflowInstance-72475][TaskInstance-74145] - Cancel task failed, this will not affect the taskInstance status, but you need to check manual
org.apache.dolphinscheduler.plugin.task.api.TaskException: cancel application error
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteShellTask.cancel(RemoteShellTask.java:121)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.cancelTask(WorkerTaskExecuteRunnable.java:136)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.afterThrowing(WorkerTaskExecuteRunnable.java:118)
        at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.afterThrowing(DefaultWorkerDelayTaskExecuteRunnable.java:67)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:182)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.dolphinscheduler.plugin.task.api.TaskException: SSH connection failed
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.getSession(RemoteExecutor.java:83)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.runRemote(RemoteExecutor.java:208)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.getTaskPid(RemoteExecutor.java:184)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.kill(RemoteExecutor.java:176)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteShellTask.cancel(RemoteShellTask.java:119)
        ... 11 common frames omitted
Caused by: java.lang.IllegalStateException: SshClient not started. Please call start() method before connecting to a server
        at org.apache.sshd.client.SshClient.doConnect(SshClient.java:627)
        at org.apache.sshd.client.SshClient.doConnect(SshClient.java:616)
        at org.apache.sshd.client.SshClient.connect(SshClient.java:547)
        at org.apache.sshd.client.SshClient.connect(SshClient.java:539)
        at org.apache.sshd.client.session.ClientSessionCreator.connect(ClientSessionCreator.java:74)
        at org.apache.sshd.client.session.ClientSessionCreator.connect(ClientSessionCreator.java:57)
        at org.apache.dolphinscheduler.plugin.datasource.ssh.SSHUtils.getSession(SSHUtils.java:41)
        at org.apache.dolphinscheduler.plugin.task.remoteshell.RemoteExecutor.getSession(RemoteExecutor.java:78)
        ... 15 common frames omitted`

here is the snapshot of host's memory:

What you expected to happen

execute remoteshell tasks and has no memory leaks

How to reproduce

create remoteshell task,and schedules them in a short time

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

ruanwenjun · 2024-04-15T15:55:09Z

Should be fixed by #15348

simmonn · 2024-04-16T01:43:20Z

Should be finished by #15348

this issue you mentioned had been solved locally before. memory leak still exists

ruanwenjun · 2024-04-16T05:42:13Z

Have you close the RemoteExecutor?

       // add task close method to release resource
         try (RemoteExecutor executor = remoteExecutor) {

simmonn · 2024-04-16T05:55:03Z

executor

yes,I have
here is the code

ruanwenjun · 2024-04-16T08:58:15Z

@simmonn Could you please provide the heap dump file or stack info? Maybe this is caused by thread leak.

ruanwenjun · 2024-04-16T09:25:36Z

Should be finished by #15348

this issue you mentioned had been solved locally before. memory leak still exists

Please try to change the heartbeatType to IGNORE

session.setSessionHeartbeat(SessionHeartbeatController.HeartbeatType.IGNORE, Duration.ofSeconds(3));

simmonn · 2024-04-17T02:14:59Z

@simmonn Could you please provide the heap dump file or stack info? Maybe this is caused by thread leak.

I simulated it in the test environment. Executing the SSH command via shell works fine. However, when executing tasks via RemoteShell, the tasks will also get stuck after a while. Here are the stack info.
stack_417.jstack.gz

peak-xu · 2024-04-17T02:36:14Z

@simmonn Could you please provide the heap dump file or stack info? Maybe this is caused by thread leak.

I encountered the same problem #15812

ruanwenjun · 2024-04-17T04:47:15Z

Yes, I changed the heartbeat to IGNORE then the thread leak can be resolved. Please test. @simmonn @peak-xu

simmonn · 2024-04-17T05:49:01Z

Yes, I changed the heartbeat to IGNORE then the thread leak can be resolved. Please test. @simmonn @peak-xu

Thank you, I'll try this approach.

peak-xu · 2024-04-17T06:13:40Z

Yes, I changed the heartbeat to IGNORE then the thread leak can be resolved. Please test. @simmonn @peak-xu

Do I need to package and compile the dev branch code myself for testing?It may take some time

ruanwenjun · 2024-04-17T10:03:46Z

Yes, I changed the heartbeat to IGNORE then the thread leak can be resolved. Please test. @simmonn @peak-xu

Do I need to package and compile the dev branch code myself for testing?It may take some time

Yes, or you can directly update your code.

github-actions · 2024-05-25T00:20:25Z

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions · 2024-06-01T00:23:08Z

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

simmonn added bug Something isn't working Waiting for reply Waiting for reply labels Apr 15, 2024

ruanwenjun added Waiting for user feedback Waiting for feedback from issue/PR author and removed Waiting for reply Waiting for reply labels Apr 15, 2024

ruanwenjun self-assigned this Apr 15, 2024

ruanwenjun mentioned this issue Apr 16, 2024

Change ssh heartbeat type to IGNORE #15858

Merged

ruanwenjun added priority:high and removed Waiting for user feedback Waiting for feedback from issue/PR author labels Apr 17, 2024

github-actions bot added the Stale label May 25, 2024

github-actions bot closed this as completed Jun 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] [Worker] RemoteShell tasks may have memory leak issue #15851

[Bug] [Worker] RemoteShell tasks may have memory leak issue #15851

simmonn commented Apr 15, 2024 •

edited by ruanwenjun

Loading

ruanwenjun commented Apr 15, 2024 •

edited

Loading

simmonn commented Apr 16, 2024 •

edited

Loading

ruanwenjun commented Apr 16, 2024

simmonn commented Apr 16, 2024

ruanwenjun commented Apr 16, 2024 •

edited

Loading

ruanwenjun commented Apr 16, 2024

simmonn commented Apr 17, 2024

peak-xu commented Apr 17, 2024

ruanwenjun commented Apr 17, 2024 •

edited

Loading

simmonn commented Apr 17, 2024

peak-xu commented Apr 17, 2024

ruanwenjun commented Apr 17, 2024

github-actions bot commented May 25, 2024

github-actions bot commented Jun 1, 2024

[Bug] [Worker] RemoteShell tasks may have memory leak issue #15851

[Bug] [Worker] RemoteShell tasks may have memory leak issue #15851

Comments

simmonn commented Apr 15, 2024 • edited by ruanwenjun Loading

Search before asking

What happened

What you expected to happen

How to reproduce

Anything else

Version

Are you willing to submit PR?

Code of Conduct

ruanwenjun commented Apr 15, 2024 • edited Loading

simmonn commented Apr 16, 2024 • edited Loading

ruanwenjun commented Apr 16, 2024

simmonn commented Apr 16, 2024

ruanwenjun commented Apr 16, 2024 • edited Loading

ruanwenjun commented Apr 16, 2024

simmonn commented Apr 17, 2024

peak-xu commented Apr 17, 2024

ruanwenjun commented Apr 17, 2024 • edited Loading

simmonn commented Apr 17, 2024

peak-xu commented Apr 17, 2024

ruanwenjun commented Apr 17, 2024

github-actions bot commented May 25, 2024

github-actions bot commented Jun 1, 2024

simmonn commented Apr 15, 2024 •

edited by ruanwenjun

Loading

ruanwenjun commented Apr 15, 2024 •

edited

Loading

simmonn commented Apr 16, 2024 •

edited

Loading

ruanwenjun commented Apr 16, 2024 •

edited

Loading

ruanwenjun commented Apr 17, 2024 •

edited

Loading