[Bug] [connector-jdbc] Specify offline mode to execute discovery as a streaming task

[zengqinchris]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Specify offline mode to execute discovery as a streaming task, Perform approximately 10 tasks, which may occur once

SeaTunnel Version

v2.3.1

SeaTunnel Config

{
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "query" : " select id,use_name,use_pwd,load_ts from xxx where 1=1",
            "connection_check_timeout_sec" : 1000,
            "plugin_name" : "Jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/wcscreen"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "max_retries" : 3,
            "query" : " replace into xxx(id,use_name,use_pwd,load_ts) values (?,?,?,?) ",
            "plugin_name" : "jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/ydtest"
        }
    ]
}

Running Command

./bin/seatunnel.sh --config ./config/bb.conf

Error Exception

2023-04-24 15:41:34.041 INFO Program is substituting variables for you
2023-04-24 15:41:34.041 INFO Variables substitution ended successfully
2023-04-24 15:41:34.041 WARN The code you submit will not be limited by the limit
Job with jobId : nodeexecution_hadoop_ydstnnl_59 and execID : nodeexecution_hadoop_ydstnnl_59 submitted 
2023-04-24 15:41:34.041 INFO You have submitted a new job, script code (after variable substitution) is
************************************SCRIPT CODE************************************
{"taskId":9,"script":"/YdBigData/user/seatunnel/config/9_319_MySQL_MySQL.conf","scriptType":"prod","source":"MySQL","sink":"MySQL","taskName":"m2m4","remark":"r","createTime":"2023-04-20 20:44:34","publishTime":"2023-04-20 20:59:17","projectId":"319","flowId":336,"projectName":"lz_project","flowName":"m2m4","nodeKey":"3ea09f0d-abd8-449d-b2c1-d92040f571ad"}
************************************SCRIPT CODE************************************
2023-04-24 15:41:34.041 INFO Your job is accepted,  jobID is exec_id018014linkis-cg-entrancebigdata08:9104nodeexecution_hadoop_ydstnnl_59 and taskID is 1420 in ServiceInstance(linkis-cg-entrance, bigdata08:9104). Please wait it to be scheduled
job is scheduled.
2023-04-24 15:41:34.041 INFO Your job is Scheduled. Please wait it to run.
Your job is being scheduled by orchestrator.
2023-04-24 15:41:34.041 INFO job is running.
2023-04-24 15:41:34.041 INFO Your job is Running now. Please wait it to complete.
2023-04-24 15:41:34.041 INFO Job with jobGroupId : 1420 and subJobId : 1420 was submitted to Orchestrator.
2023-04-24 15:41:34.041 INFO Background is starting a new engine for you,execId astJob_136_codeExec_136 mark id is mark_136, it may take several seconds, please wait
2023-04-24 15:41:34.041 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, bigdata08:46624) /bigdata/datahorizon/application/linkis/dirs/hadoop/20230424/ydstnnl/14de5239-f2b5-41d5-98d1-e786df3979d1/logs
bigdata08:46624_0 >> /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/bin/seatunnel.sh --config  /appcom/tmp/ydstnnlConfDir/9_319_MySQL_MySQL.conf
JAVA_OPTS: -Dhazelcast.client.config=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast-client.yaml -Dseatunnel.config=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/seatunnel.yaml -Dhazelcast.config=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast.yaml -Dlog4j2.configurationFile=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/log4j2_client.properties -Dseatunnel.logs.path=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/logs -Dseatunnel.logs.file_name=seatunnel-starter-client -Xms128m -Xmx512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/seatunnel-6639055162999374831 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/seatunnel/

Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/seatunnel.yaml
Apr 24, 2023 3:41:35 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
INFO: seatunnel.home is /home/hadoop
Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast.yaml
Apr 24, 2023 3:41:36 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
Apr 24, 2023 3:41:36 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast-client.yaml
2023-04-24 15:41:36,678 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2023-04-24 15:41:36,753 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-04-24 15:41:36,754 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-04-24 15:41:36,777 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2023-04-24 15:41:36,780 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [bigdata04]:5801
2023-04-24 15:41:36,811 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-04-24 15:41:36,812 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [bigdata04]:5801:3ea41eb6-06cd-470a-94fe-f8c485cfa2f2, server version: 5.1, local address: /172.27.10.253:35535
2023-04-24 15:41:36,813 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-04-24 15:41:36,824 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] 
Members [2] {
	Member [bigdata03]:5801 - 438295ac-d767-43ab-ba13-e79495639ecc
	Member [bigdata04]:5801 - 3ea41eb6-06cd-470a-94fe-f8c485cfa2f2
}
2023-04-24 15:41:36,826 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [bigdata03]:5801:438295ac-d767-43ab-ba13-e79495639ecc, server version: 5.1, local address: /172.27.10.253:53423
2023-04-24 15:41:36,852 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-04-24 15:41:36,892 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/sqlite-jdbc-3.39.3.0.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/redshift-jdbc42-2.1.0.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/tablestore-jdbc-5.13.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ojdbc8-12.2.0.1.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/db2jcc-db2jcc4.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/postgresql-42.4.3.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/DmJdbcDriver18-8.1.2.141.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ali-phoenix-shaded-thin-client-5.2.5-HBase-2.x.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mssql-jdbc-9.2.1.jre8.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/terajdbc4-17.20.00.12.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mysql-connector-java-8.0.27.jar]
2023-04-24 15:41:36,898 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /appcom/tmp/ydstnnlConfDir/9_319_MySQL_MySQL.conf
2023-04-24 15:41:36,946 INFO  org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-04-24 15:41:36,989 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "query" : " select id,use_name,use_pwd,load_ts from xxx where 1=1",
            "connection_check_timeout_sec" : 1000,
            "plugin_name" : "Jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/wcscreen"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "max_retries" : 3,
            "query" : " replace into xxx(id,use_name,use_pwd,load_ts) values (?,?,?,?) ",
            "plugin_name" : "jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/ydtest"
        }
    ]
}
2023-04-24 15:41:37,008 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /appcom/tmp/ydstnnlConfDir/9_319_MySQL_MySQL.conf
2023-04-24 15:41:37,013 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "query" : " select id,use_name,use_pwd,load_ts from xxx where 1=1",
            "connection_check_timeout_sec" : 1000,
            "plugin_name" : "Jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/wcscreen"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "max_retries" : 3,
            "query" : " replace into xxx(id,use_name,use_pwd,load_ts) values (?,?,?,?) ",
            "plugin_name" : "jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/ydtest"
        }
    ]
}
2023-04-24 15:41:37,021 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel
2023-04-24 15:41:37,031 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Jdbc at: file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar
2023-04-24 15:41:37,045 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Jdbc'} from path: file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2023-04-24 15:41:38,450 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource - The partition_column parameter is not configured, and the source parallelism is set to 1
2023-04-24 15:41:38,455 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel
2023-04-24 15:41:38,456 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: jdbc at: file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar
2023-04-24 15:41:38,457 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='jdbc'} from classpath
2023-04-24 15:41:38,496 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - start submit job, job id: 702786240566525958, with plugin jar [file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/sqlite-jdbc-3.39.3.0.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/redshift-jdbc42-2.1.0.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/tablestore-jdbc-5.13.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ojdbc8-12.2.0.1.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/db2jcc-db2jcc4.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/postgresql-42.4.3.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/DmJdbcDriver18-8.1.2.141.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ali-phoenix-shaded-thin-client-5.2.5-HBase-2.x.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mssql-jdbc-9.2.1.jre8.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/terajdbc4-17.20.00.12.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mysql-connector-java-8.0.27.jar]
2023-04-24 15:41:38,666 WARN  org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2023-04-24 15:42:38,666 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:41:38
Current Statistic Time    : 2023-04-24 15:42:38
***********************************************
2023-04-24 15:43:38,660 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:42:38
Current Statistic Time    : 2023-04-24 15:43:38
***********************************************
2023-04-24 15:44:38,660 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:43:38
Current Statistic Time    : 2023-04-24 15:44:38
***********************************************
2023-04-24 15:45:38,659 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:44:38
Current Statistic Time    : 2023-04-24 15:45:38
***********************************************
2023-04-24 15:46:38,661 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:45:38
Current Statistic Time    : 2023-04-24 15:46:38
***********************************************
2023-04-24 15:46:39,825 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (702786240566525958) end with state FAILED
2023-04-24 15:46:39,826 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-04-24 15:46:39,832 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [bigdata04]:5801:3ea41eb6-06cd-470a-94fe-f8c485cfa2f2, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.27.10.253:35535->bigdata04/172.27.10.183:5801}, remoteAddress=[bigdata04]:5801, lastReadTime=2023-04-24 15:46:36.855, lastWriteTime=2023-04-24 15:46:36.855, closedTime=2023-04-24 15:46:39.828, connected server version=5.1}
2023-04-24 15:46:39,833 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [bigdata03]:5801:438295ac-d767-43ab-ba13-e79495639ecc, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/172.27.10.253:53423->bigdata03/172.27.10.152:5801}, remoteAddress=[bigdata03]:5801, lastReadTime=2023-04-24 15:46:39.812, lastWriteTime=2023-04-24 15:46:38.626, closedTime=2023-04-24 15:46:39.832, connected server version=5.1}
2023-04-24 15:46:39,833 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-04-24 15:46:39,835 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-04-24 15:46:39,835 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-04-24 15:46:39,836 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
===============================================================================
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 
2023-04-24 15:46:39,838 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:215)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:211)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$2(CheckpointCoordinator.java:201)
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1632)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: Execute given execution failed after retry 30 times
	at com.hazelcast.spi.impl.AbstractInvocationFuture.returnOrThrowWithJoinConventions(AbstractInvocationFuture.java:819)
	at com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowWithJoinConvention(AbstractInvocationFuture.java:835)
	at com.hazelcast.spi.impl.AbstractInvocationFuture.join(AbstractInvocationFuture.java:553)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:250)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
	... 3 more
Caused by: java.lang.RuntimeException: Execute given execution failed after retry 30 times
	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:75)
	at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.engine.server.exception.TaskGroupContextNotFoundException: task group TaskGroupLocation{jobId=702786240566525958, pipelineId=1, taskGroupId=1} not found.
	at org.apache.seatunnel.engine.server.TaskExecutionService.getExecutionContext(TaskExecutionService.java:168)
	at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:90)
	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
	... 9 more
	at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:119)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:174)
	... 2 more
 
2023-04-24 15:46:39,838 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
===============================================================================
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:215)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:211)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$2(CheckpointCoordinator.java:201)
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1632)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: Execute given execution failed after retry 30 times
	at com.hazelcast.spi.impl.AbstractInvocationFuture.returnOrThrowWithJoinConventions(AbstractInvocationFuture.java:819)
	at com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowWithJoinConvention(AbstractInvocationFuture.java:835)
	at com.hazelcast.spi.impl.AbstractInvocationFuture.join(AbstractInvocationFuture.java:553)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:250)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
	... 3 more
Caused by: java.lang.RuntimeException: Execute given execution failed after retry 30 times
	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:75)
	at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.engine.server.exception.TaskGroupContextNotFoundException: task group TaskGroupLocation{jobId=702786240566525958, pipelineId=1, taskGroupId=1} not found.
	at org.apache.seatunnel.engine.server.TaskExecutionService.getExecutionContext(TaskExecutionService.java:168)
	at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:90)
	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
	... 9 more
	at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:119)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:174)
	... 2 more
2023-04-24 15:46:39,839 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

Flink or Spark Version

no use flink or spark

Java or Scala Version

jdk8

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zengqinchris]

If there are more than two seatunnel cluster nodes, this problem is easy to occur. If there are only a single seatunnel node, it will occur less frequently

[EricJoy2048]

Hi, this is a bug in V2.3.1, we fixed it in #4449

You can use dev branch for test.

[zengqinchris]

Thank you for your support and assistance. The bug in dev code testing is no longer present

[zengqinchris]

The bug in dev code fixed

[zengqinchris]

Thank you for your support

Thank you for your support ，fixed

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.