[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48771

panbingkun · 2024-11-06T07:27:11Z

What changes were proposed in this pull request?

The pr aims to upgrade ZooKeeper from 3.9.2 to 3.9.3.
This PR is to fix potential issues with PR #48666

Why are the changes needed?

The full release notes: https://zookeeper.apache.org/doc/r3.9.3/releasenotes.html

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.
Manually check

./build/sbt -Phadoop-3 -Pkubernetes -Pkinesis-asl -Phive-thriftserver -Pdocker-integration-tests -Pyarn -Phadoop-cloud -Pspark-ganglia-lgpl -Phive -Pjvm-profiler clean package

[info] Note: Some input files use or override a deprecated API.
[info] Note: Recompile with -Xlint:deprecation for details.
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[success] Total time: 272 s (04:32), completed Nov 6, 2024, 4:29:52 PM

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗ ./python/run-tests --python-executables=python3 --testnames "pyspark.sql.tests.connect.test_connect_collection"
Running PySpark tests. Output is in /Users/panbingkun/Developer/spark/spark-community/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python tests: ['pyspark.sql.tests.connect.test_connect_collection']
python3 python_implementation is CPython
python3 version is: Python 3.9.19
Starting test(python3): pyspark.sql.tests.connect.test_connect_collection (temp output: /Users/panbingkun/Developer/spark/spark-community/python/target/097bd7e0-9311-4484-ae2d-c0f4c63fc6f9/python3__pyspark.sql.tests.connect.test_connect_collection__8dzaeio9.log)
Finished test(python3): pyspark.sql.tests.connect.test_connect_collection (14s)
Tests passed in 14 seconds

Was this patch authored or co-authored using generative AI tooling?

No.

LuciferYang · 2024-11-06T08:26:47Z

we should change to exclude

<exclusion> 
       <groupId>io.netty</groupId> 
       <artifactId>*</artifactId> 
     </exclusion>

from zookeeper instead of

<exclusion> 
       <groupId>org.jboss.netty</groupId> 
       <artifactId>netty</artifactId> 
     </exclusion>

org.jboss.netty is the groupId of netty 3.x

However, I believe it would be better to upgrade netty to 4.1.113, because zk 3.9.3 was compiled with a dependency on netty 4.1.113. Downgrading to use 4.1.110 may introduce uncertainties and potential issues.

panbingkun · 2024-11-06T10:42:18Z

we should change to exclude
<exclusion> 
       <groupId>io.netty</groupId> 
       <artifactId>*</artifactId> 
     </exclusion> 
from zookeeper instead of
<exclusion> 
       <groupId>org.jboss.netty</groupId> 
       <artifactId>netty</artifactId> 
     </exclusion> 
org.jboss.netty is the groupId of netty 3.x

However, I believe it would be better to upgrade netty to 4.1.113, because zk 3.9.3 was compiled with a dependency on netty 4.1.113. Downgrading to use 4.1.110 may introduce uncertainties and potential issues.

There is an issue with the version 4.1.113, but version 4.1.114 appears to have fixed the problem that has been present since version 4.1.112. Let's upgrade netty to version 4.1.114 first.
#46945

panbingkun · 2024-11-06T10:44:11Z

I have verified that the compilation issue with sbt has been resolved after upgrading the netty version to 1.1.114. UT ./python/run-tests --python-executables=python3 --testnames "pyspark.sql.tests.connect.test_connect_collection" has also been successful.

panbingkun · 2024-11-06T10:48:02Z

project/SparkBuild.scala

-    dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
-    dependencyOverrides += "jline" % "jline" % "2.14.6",
-    dependencyOverrides += "org.apache.avro" % "avro" % "1.11.3")
+    dependencyOverrides ++= {


@LuciferYang @dongjoon-hyun
Do you want to keep this? Although the issue was fixed when netty upgraded to version 1.1.114, similar problems may still occur next time when other libraries depend on different netty versions.

LuciferYang · 2024-11-06T14:27:47Z

project/SparkBuild.scala

+    dependencyOverrides ++= {
+      val nettyVersion = SbtPomKeys.effectivePom.value.getProperties.get(
+        "netty.version").asInstanceOf[String]
+      Seq(


I personally tend not to add this part of the configuration.

dongjoon-hyun

cc @HyukjinKwon and @zhengruifeng

HyukjinKwon · 2024-11-06T17:59:18Z

I am fine as long as the tests work :-).

dongjoon-hyun · 2024-11-06T18:17:39Z

Let's remove that part and proceed to merge cleanly, @panbingkun .

zhengruifeng · 2024-11-06T18:51:16Z

@panbingkun can you also simply check bin/pyspark --remote "local[*]"?

with simple queries like

In [1]: spark.range(10).collect()
24/11/06 10:50:00 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found
Out[1]:
[Row(id=0),
 Row(id=1),
 Row(id=2),
 Row(id=3),
 Row(id=4),
 Row(id=5),
 Row(id=6),
 Row(id=7),
 Row(id=8),
 Row(id=9)]

The whole pyspark connect was broken before, including all pyspark-connect tests and the REPL.

panbingkun · 2024-11-07T01:27:16Z

The verification process based on SBT and Maven compilation is as follows:

Based on SBT compile

./build/sbt -Phadoop-3 -Pkubernetes -Pkinesis-asl -Phive-thriftserver -Pdocker-integration-tests -Pyarn -Phadoop-cloud -Pspark-ganglia-lgpl -Phive -Pjvm-profiler clean package

[info] Note: Some input files use or override a deprecated API.
[info] Note: Recompile with -Xlint:deprecation for details.
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[success] Total time: 251 s (04:11), completed Nov 7, 2024, 9:23:18 AM

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗ ls -1 assembly/target/scala-2.13/jars/netty-*1*
assembly/target/scala-2.13/jars/netty-all-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-buffer-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-http-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-http2-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-socks-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-common-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-handler-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-handler-proxy-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-resolver-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-classes-epoll-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-classes-kqueue-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-native-epoll-4.1.114.Final-linux-aarch_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-epoll-4.1.114.Final-linux-riscv64.jar
assembly/target/scala-2.13/jars/netty-transport-native-epoll-4.1.114.Final-linux-x86_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-kqueue-4.1.114.Final-osx-aarch_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-kqueue-4.1.114.Final-osx-x86_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-unix-common-4.1.114.Final.jar

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗ ./python/run-tests --python-executables=python3 --testnames "pyspark.sql.tests.connect.test_connect_collection"
Running PySpark tests. Output is in /Users/panbingkun/Developer/spark/spark-community/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python tests: ['pyspark.sql.tests.connect.test_connect_collection']
python3 python_implementation is CPython
python3 version is: Python 3.9.19
Starting test(python3): pyspark.sql.tests.connect.test_connect_collection (temp output: /Users/panbingkun/Developer/spark/spark-community/python/target/2ed7ab5b-39f0-41f7-85c5-0fdcab0b0b2a/python3__pyspark.sql.tests.connect.test_connect_collection__61isowkl.log)
Finished test(python3): pyspark.sql.tests.connect.test_connect_collection (13s)
Tests passed in 13 seconds

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗  ./bin/pyspark --remote "sc://127.0.0.1:8888"
Python 3.9.19 (main, Mar 21 2024, 12:07:41)
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 4.0.0.dev0
      /_/

Using Python version 3.9.19 (main, Mar 21 2024 12:07:41)
Client connected to the Spark Connect server at 127.0.0.1:8888
SparkSession available as 'spark'.
>>> spark.range(10).collect()
[Row(id=0), Row(id=1), Row(id=2), Row(id=3), Row(id=4), Row(id=5), Row(id=6), Row(id=7), Row(id=8), Row(id=9)]

Based on MAVEN compile

./build/mvn -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean package -T4

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:16 min (Wall Clock)
[INFO] Finished at: 2024-11-07T09:43:02+08:00
[INFO] ------------------------------------------------------------------------

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗  ls -1 assembly/target/scala-2.13/jars/netty-*1*
assembly/target/scala-2.13/jars/netty-all-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-buffer-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-http-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-http2-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-codec-socks-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-common-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-handler-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-handler-proxy-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-resolver-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-classes-epoll-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-classes-kqueue-4.1.114.Final.jar
assembly/target/scala-2.13/jars/netty-transport-native-epoll-4.1.114.Final-linux-aarch_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-epoll-4.1.114.Final-linux-riscv64.jar
assembly/target/scala-2.13/jars/netty-transport-native-epoll-4.1.114.Final-linux-x86_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-kqueue-4.1.114.Final-osx-aarch_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-kqueue-4.1.114.Final-osx-x86_64.jar
assembly/target/scala-2.13/jars/netty-transport-native-unix-common-4.1.114.Final.jar

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗ ./python/run-tests --python-executables=python3 --testnames "pyspark.sql.tests.connect.test_connect_collection"
Running PySpark tests. Output is in /Users/panbingkun/Developer/spark/spark-community/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python tests: ['pyspark.sql.tests.connect.test_connect_collection']
python3 python_implementation is CPython
python3 version is: Python 3.9.19
Starting test(python3): pyspark.sql.tests.connect.test_connect_collection (temp output: /Users/panbingkun/Developer/spark/spark-community/python/target/7a50ed3b-267c-4891-903d-951aaf75f74c/python3__pyspark.sql.tests.connect.test_connect_collection__vqf968_3.log)
Finished test(python3): pyspark.sql.tests.connect.test_connect_collection (16s)
Tests passed in 16 seconds

(pyspark) ➜  spark-community git:(SPARK-50135_FOLLOWUP) ✗ ./bin/pyspark --remote "sc://127.0.0.1:8888"
Python 3.9.19 (main, Mar 21 2024, 12:07:41)
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 4.0.0.dev0
      /_/

Using Python version 3.9.19 (main, Mar 21 2024 12:07:41)
Client connected to the Spark Connect server at 127.0.0.1:8888
SparkSession available as 'spark'.
>>> spark.range(10).collect()
[Row(id=0), Row(id=1), Row(id=2), Row(id=3), Row(id=4), Row(id=5), Row(id=6), Row(id=7), Row(id=8), Row(id=9)]

panbingkun · 2024-11-07T01:47:14Z

@panbingkun can you also simply check bin/pyspark --remote "local[*]"?

with simple queries like
In [1]: spark.range(10).collect()
24/11/06 10:50:00 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found
Out[1]:
[Row(id=0),
 Row(id=1),
 Row(id=2),
 Row(id=3),
 Row(id=4),
 Row(id=5),
 Row(id=6),
 Row(id=7),
 Row(id=8),
 Row(id=9)]
The whole pyspark connect was broken before, including all pyspark-connect tests and the REPL.

Done, the verification process has been posted.

panbingkun · 2024-11-07T01:47:55Z

Let's remove that part and proceed to merge cleanly, @panbingkun .

https://github.com/apache/spark/pull/48771/files#r1830807688

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48771 (comment)

Done.

dongjoon-hyun

+1, LGTM. Thank you, @panbingkun and all.
Merged to master back.

panbingkun · 2024-11-07T06:08:12Z

Thanks all @LuciferYang @zhengruifeng @HyukjinKwon @dongjoon-hyun !

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3

bd0a326

github-actions bot added the BUILD label Nov 6, 2024

panbingkun mentioned this pull request Nov 6, 2024

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48666

Closed

panbingkun marked this pull request as ready for review November 6, 2024 10:44

panbingkun marked this pull request as draft November 6, 2024 10:44

panbingkun commented Nov 6, 2024

View reviewed changes

panbingkun marked this pull request as ready for review November 6, 2024 11:09

LuciferYang reviewed Nov 6, 2024

View reviewed changes

dongjoon-hyun reviewed Nov 6, 2024

View reviewed changes

restore SparkBuild.scala

d253b67

dongjoon-hyun approved these changes Nov 7, 2024

View reviewed changes

dongjoon-hyun closed this in 07301dd Nov 7, 2024

panbingkun mentioned this pull request Nov 29, 2024

[SPARK-50453][BUILD] Upgrade Netty to 4.1.115 #49008

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48771

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48771

panbingkun commented Nov 6, 2024 •

edited

Loading

LuciferYang commented Nov 6, 2024

panbingkun commented Nov 6, 2024

panbingkun commented Nov 6, 2024

panbingkun Nov 6, 2024 •

edited

Loading

LuciferYang Nov 6, 2024

HyukjinKwon Nov 6, 2024

panbingkun Nov 7, 2024

dongjoon-hyun left a comment

HyukjinKwon commented Nov 6, 2024

dongjoon-hyun commented Nov 6, 2024

zhengruifeng commented Nov 6, 2024 •

edited

Loading

panbingkun commented Nov 7, 2024 •

edited

Loading

panbingkun commented Nov 7, 2024

panbingkun commented Nov 7, 2024

dongjoon-hyun left a comment

panbingkun commented Nov 7, 2024

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48771

[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3 #48771

Conversation

panbingkun commented Nov 6, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

LuciferYang commented Nov 6, 2024

panbingkun commented Nov 6, 2024

panbingkun commented Nov 6, 2024

panbingkun Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

LuciferYang Nov 6, 2024

Choose a reason for hiding this comment

HyukjinKwon Nov 6, 2024

Choose a reason for hiding this comment

panbingkun Nov 7, 2024

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Nov 6, 2024

dongjoon-hyun commented Nov 6, 2024

zhengruifeng commented Nov 6, 2024 • edited Loading

panbingkun commented Nov 7, 2024 • edited Loading

panbingkun commented Nov 7, 2024

panbingkun commented Nov 7, 2024

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented Nov 7, 2024

panbingkun commented Nov 6, 2024 •

edited

Loading

panbingkun Nov 6, 2024 •

edited

Loading

zhengruifeng commented Nov 6, 2024 •

edited

Loading

panbingkun commented Nov 7, 2024 •

edited

Loading