[SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #32096

wangyum · 2021-04-08T14:00:54Z

What changes were proposed in this pull request?

This PR tries to fix the java.net.BindException when testing with Github Action:

[info] org.apache.spark.sql.kafka010.producer.InternalKafkaProducerPoolSuite *** ABORTED *** (282 milliseconds)
[info]   java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 100 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
[info]   at sun.nio.ch.Net.bind0(Native Method)
[info]   at sun.nio.ch.Net.bind(Net.java:461)
[info]   at sun.nio.ch.Net.bind(Net.java:453)

https://github.com/apache/spark/pull/32090/checks?check_run_id=2295418529

Why are the changes needed?

Fix test framework.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Test by Github Action.

…ervice 'sparkDriver' failed after 100 retries

SparkQA · 2021-04-08T15:04:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41659/

SparkQA · 2021-04-08T15:04:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41659/

SparkQA · 2021-04-08T17:21:53Z

Test build #137081 has finished for PR 32096 at commit 62083f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2021-04-08T20:54:37Z

.github/workflows/benchmark.yml

@@ -47,6 +47,7 @@ jobs:
      SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
      SPARK_BENCHMARK_CUR_SPLIT: ${{ matrix.split }}
      SPARK_GENERATE_BENCHMARK_FILES: 1
+      SPARK_LOCAL_IP: localhost


If it works, 127.0.0.1 would be better?

SparkQA · 2021-04-09T00:11:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41681/

SparkQA · 2021-04-09T00:11:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41681/

SparkQA · 2021-04-09T02:16:32Z

Test build #137103 has finished for PR 32096 at commit 617a1ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-09T02:22:25Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41686/

dongjoon-hyun · 2021-04-09T03:57:17Z

Hi, @wangyum . This seems to be a duplicated effort.

[DO-NOT-MERGE] Investigate loopback/InetAddress.getLocalHost #32086

cc @HyukjinKwon

HyukjinKwon · 2021-04-09T04:02:08Z

.github/workflows/benchmark.yml

@@ -47,6 +47,7 @@ jobs:
      SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
      SPARK_BENCHMARK_CUR_SPLIT: ${{ matrix.split }}
      SPARK_GENERATE_BENCHMARK_FILES: 1
+      SPARK_LOCAL_IP: 127.0.0.1


@wangyum, I think you should set HIVE_SERVER2_THRIFT_BIND_HOST too. BTW, I think this is virtually because InetAddress.getLocalHost, via DNS, returns a wrong address in specific GitHub Actions machines.

Here's the example

fv-az213-557.internal.cloudapp.net, 10.1.0.4, 10.1.0.4

It seems locaohost much better. We do not need set HIVE_SERVER2_THRIFT_BIND_HOST.

HyukjinKwon · 2021-04-09T04:03:55Z

Thanks for letting me know @dongjoon-hyun.

Please file a JIRA and go ahead with this approach @wangyum. I will try to take a deeper look and see if there's another way to fix.

SparkQA · 2021-04-09T04:08:41Z

Test build #137108 has finished for PR 32096 at commit 7999243.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-04-09T05:59:44Z

The Hive test suite failures came from another commit (landed 5 hours ago) and I reverted it. Sorry for that guys.

dongjoon-hyun

+1, LGTM.

dongjoon-hyun · 2021-04-09T06:03:19Z

Merged to master.

HyukjinKwon · 2021-04-09T06:17:22Z

@wangyum, we need to set HIVE_SERVER2_THRIFT_BIND_HOST because the problem is from InetAddress.getLocalHost

HyukjinKwon · 2021-04-09T06:17:55Z

spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java

Lines 132 to 141 in 0494dc9

    
           hiveHost = System.getenv("HIVE_SERVER2_THRIFT_BIND_HOST"); 
        
           if (hiveHost == null) { 
        
             hiveHost = hiveConf.getVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST); 
        
           } 
        
           try { 
        
             if (hiveHost != null && !hiveHost.isEmpty()) { 
        
               serverIPAddress = InetAddress.getByName(hiveHost); 
        
             } else { 
        
               serverIPAddress = InetAddress.getLocalHost(); 
        
             }

HyukjinKwon · 2021-04-09T06:24:38Z

Ah okay but we explicitly set it at

spark/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala

Line 1185 in 0494dc9

| --hiveconf ${ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST}=localhost

Okay, I agree with doing localhost.

HyukjinKwon · 2021-04-09T06:26:12Z

Thanks for merging it in any event. The issue made the developement super slowed down.

… SPARK_LOCAL_IP in GA builds ### What changes were proposed in this pull request? This PR replaces 127.0.0.1 to `localhost`. ### Why are the changes needed? - #32096 (comment) - #32096 (comment) ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? I didn't test it because it's CI specific issue. I will test it in Github Actions build in this PR. Closes #32102 from HyukjinKwon/SPARK-35002. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Yuming Wang <[email protected]>

… testing with Github Action ![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #503](https://badgen.net/badge/Preview/Closes%20%23503/blue)](https://github.com/yaooqinn/kyuubi/pull/503) ![6](https://badgen.net/badge/%2B/6/red) ![1](https://badgen.net/badge/-/1/green) ![10](https://badgen.net/badge/commits/10/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)  ### _Why are the changes needed?_ Refer apache/spark#32096, apache/spark#32102, this PR tries to fix the java.net.BindException when testing with Github Action. ``` SparkOperationSuite: *** RUN ABORTED *** java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address. at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:461) at sun.nio.ch.Net.bind(Net.java:453) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) ``` Also transfer FRONTEND_BIND_HOST by connection string to fix similar issue. ``` Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on fv-az207-19/10.1.1.0:0. at org.apache.kyuubi.service.FrontendService.initialize(FrontendService.scala:102) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40) at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:44) at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:49) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:105) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:118) at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #503 from turboFei/KYUUBI-502. Closes #502 1b10253 [fwang12] use localhost instead of 127.0.0.1 c104ce3 [fwang12] address comments 1e549c1 [fwang12] revert install shade 457ce2f [fwang12] try set frontend bind host in connection string 8bcd5a0 [fwang12] revert env KYUUBI_FRONTEND_BIND_HOST and set kyuubi.frontend.bind.host to 127.0.0.1 in scalatest-maven-plugin 717a992 [fwang12] update doc d5ba05a [fwang12] add install shaded jars in release.yml e8b2372 [fwang12] involve KYUUBI_FRONTEND_BIND_HOST 5eb7cdb [fwang12] also set KYUUBI_FRONTEND_BIND_HOST env to 127.0.0.1 7d70819 [fwang12] [KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action Authored-by: fwang12 <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

… Github Action This PR tries to fix the `java.net.BindException` when testing with Github Action: ``` [info] org.apache.spark.sql.kafka010.producer.InternalKafkaProducerPoolSuite *** ABORTED *** (282 milliseconds) [info] java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 100 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address. [info] at sun.nio.ch.Net.bind0(Native Method) [info] at sun.nio.ch.Net.bind(Net.java:461) [info] at sun.nio.ch.Net.bind(Net.java:453) ``` https://github.com/apache/spark/pull/32090/checks?check_run_id=2295418529 Fix test framework. No. Test by Github Action. Closes #32096 from wangyum/SPARK_LOCAL_IP=localhost. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 9663c40) Signed-off-by: HyukjinKwon <[email protected]>

HyukjinKwon · 2021-04-14T10:43:00Z

I merged to branch-3.1 and branch-3.0 too. Seems like it has the same issue.

… Github Action This PR tries to fix the `java.net.BindException` when testing with Github Action: ``` [info] org.apache.spark.sql.kafka010.producer.InternalKafkaProducerPoolSuite *** ABORTED *** (282 milliseconds) [info] java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 100 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address. [info] at sun.nio.ch.Net.bind0(Native Method) [info] at sun.nio.ch.Net.bind(Net.java:461) [info] at sun.nio.ch.Net.bind(Net.java:453) ``` https://github.com/apache/spark/pull/32090/checks?check_run_id=2295418529 Fix test framework. No. Test by Github Action. Closes #32096 from wangyum/SPARK_LOCAL_IP=localhost. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 9663c40) Signed-off-by: HyukjinKwon <[email protected]>

… SPARK_LOCAL_IP in GA builds This PR replaces 127.0.0.1 to `localhost`. - #32096 (comment) - #32096 (comment) No, dev-only. I didn't test it because it's CI specific issue. I will test it in Github Actions build in this PR. Closes #32102 from HyukjinKwon/SPARK-35002. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Yuming Wang <[email protected]> (cherry picked from commit a3d1e00) Signed-off-by: HyukjinKwon <[email protected]>

… Github Action This PR tries to fix the `java.net.BindException` when testing with Github Action: ``` [info] org.apache.spark.sql.kafka010.producer.InternalKafkaProducerPoolSuite *** ABORTED *** (282 milliseconds) [info] java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 100 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address. [info] at sun.nio.ch.Net.bind0(Native Method) [info] at sun.nio.ch.Net.bind(Net.java:461) [info] at sun.nio.ch.Net.bind(Net.java:453) ``` https://github.com/apache/spark/pull/32090/checks?check_run_id=2295418529 Fix test framework. No. Test by Github Action. Closes apache#32096 from wangyum/SPARK_LOCAL_IP=localhost. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 9663c40) Signed-off-by: HyukjinKwon <[email protected]>

… SPARK_LOCAL_IP in GA builds This PR replaces 127.0.0.1 to `localhost`. - apache#32096 (comment) - apache#32096 (comment) No, dev-only. I didn't test it because it's CI specific issue. I will test it in Github Actions build in this PR. Closes apache#32102 from HyukjinKwon/SPARK-35002. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Yuming Wang <[email protected]> (cherry picked from commit a3d1e00) Signed-off-by: HyukjinKwon <[email protected]>

Try to fix java.net.BindException: Cannot assign requested address: S…

62083f6

…ervice 'sparkDriver' failed after 100 retries

wangyum changed the title ~~[WIP] Try to fix java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 100 retries~~ [WIP] Try to fix the java.net.BindException when testing with Github Action Apr 8, 2021

github-actions bot added the INFRA label Apr 8, 2021

ueshin reviewed Apr 8, 2021

View reviewed changes

Test again

617a1ad

Test again

7999243

HyukjinKwon reviewed Apr 9, 2021

View reviewed changes

wangyum changed the title ~~[WIP] Try to fix the java.net.BindException when testing with Github Action~~ [SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action Apr 9, 2021

dongjoon-hyun approved these changes Apr 9, 2021

View reviewed changes

dongjoon-hyun closed this in 9663c40 Apr 9, 2021

HyukjinKwon mentioned this pull request Apr 9, 2021

[SPARK-35002][INFRA][FOLLOW-UP] Use localhost instead of 127.0.0.1 at SPARK_LOCAL_IP in GA builds #32102

Closed

yaooqinn mentioned this pull request Apr 9, 2021

Bing GitHub Action Alive apache/kyuubi#502

Closed

6 tasks

turboFei mentioned this pull request Apr 9, 2021

[KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action apache/kyuubi#503

Closed

3 tasks

pan3793 mentioned this pull request Apr 9, 2021

[BUILD] Fix the java.net.BindException when testing with Github Action apache/iceberg#2451

Merged

wangyum deleted the SPARK_LOCAL_IP=localhost branch April 10, 2021 03:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #32096

[SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #32096

wangyum commented Apr 8, 2021 •

edited

Loading

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

ueshin Apr 8, 2021

SparkQA commented Apr 9, 2021

SparkQA commented Apr 9, 2021

SparkQA commented Apr 9, 2021

SparkQA commented Apr 9, 2021

dongjoon-hyun commented Apr 9, 2021 •

edited

Loading

HyukjinKwon Apr 9, 2021

HyukjinKwon Apr 9, 2021

wangyum Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

SparkQA commented Apr 9, 2021

dongjoon-hyun commented Apr 9, 2021

dongjoon-hyun left a comment

dongjoon-hyun commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 14, 2021

[SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #32096

[SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #32096

Conversation

wangyum commented Apr 8, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

ueshin Apr 8, 2021

Choose a reason for hiding this comment

SparkQA commented Apr 9, 2021

SparkQA commented Apr 9, 2021

SparkQA commented Apr 9, 2021

SparkQA commented Apr 9, 2021

dongjoon-hyun commented Apr 9, 2021 • edited Loading

HyukjinKwon Apr 9, 2021

Choose a reason for hiding this comment

HyukjinKwon Apr 9, 2021

Choose a reason for hiding this comment

wangyum Apr 9, 2021

Choose a reason for hiding this comment

HyukjinKwon commented Apr 9, 2021

SparkQA commented Apr 9, 2021

dongjoon-hyun commented Apr 9, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 9, 2021

HyukjinKwon commented Apr 14, 2021

wangyum commented Apr 8, 2021 •

edited

Loading

dongjoon-hyun commented Apr 9, 2021 •

edited

Loading