From d1404e1ae7509c5903c253a5e55678853194d0cf Mon Sep 17 00:00:00 2001 From: Gabor Roczei Date: Sat, 17 Aug 2024 00:04:54 +0200 Subject: [PATCH 1/2] [SPARK-48867][BUILD] Upgrade `okhttp` to 4.12.0 and `okio` to 3.9.0 ### What changes were proposed in this pull request? This PR aims to upgrade `okhttp` to 4.12.0 and `okio` to 3.9.0 ### Why are the changes needed? okhttp depends on okio which has to be upgraded as well. The new okhttp version fixes the following vulnerabilities: 1) CVE-2023-0833 - A flaw was found in Red Hat's AMQ-Streams, which ships a version of the OKHttp component with an information disclosure flaw via an exception triggered by a header containing an illegal value. This issue could allow an authenticated attacker to access information outside of their regular permissions. CVSSv3 Score:- 5.5(Medium) https://nvd.nist.gov/vuln/detail/CVE-2023-0833 2) CVE-2021-0341 - In verifyHostName of OkHostnameVerifier.java, there is a possible way to accept a certificate for the wrong domain due to improperly used crypto. This could lead to remote information disclosure with no additional execution privileges needed. User interaction is not needed for exploitation. CVSSv3 Score:- 7.5(High) https://nvd.nist.gov/vuln/detail/CVE-2021-0341 https://github.com/square/okhttp/issues/6724 There are two places in the Spark repository where the okhttp dependency comes in as transitive dependency: 1) [INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.4.0:compile [INFO] | | +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile [INFO] | | | +- org.jdom:jdom2:jar:2.0.6:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile [INFO] | | | | +- org.ini4j:ini4j:jar:0.5.4:compile [INFO] | | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-util:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile [INFO] | | | \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile [INFO] | | \- org.codehaus.jettison:jettison:jar:1.5.4:compile [INFO] | +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.0:compile [INFO] | | \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile [INFO] | \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0:compile [INFO] | \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile [INFO] | \- com.squareup.okio:okio:jar:1.17.6:compile The Hadoop team has attempted to remove okhttp from their codebase: remove okhttp usage: https://issues.apache.org/jira/browse/HADOOP-18890 Unfortunately the hadoop-huaweicloud dependency is still there which pulls in the vulnerable okhttp 3.x version. https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml#L137C19-L137C37 3.4.0 is the only available version here: https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-huaweicloud 2) [INFO] +- org.apache.spark:spark-kubernetes_2.13:jar:4.0.0-SNAPSHOT:compile [INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.13.3:compile [INFO] | | +- io.fabric8:kubernetes-client-api:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-core:jar:6.13.3:compile [INFO] | | | | \- io.fabric8:kubernetes-model-common:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-gatewayapi:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-resource:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-rbac:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-admissionregistration:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apps:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-autoscaling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apiextensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-batch:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-certificates:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-coordination:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-discovery:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-events:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-extensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-flowcontrol:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-networking:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-metrics:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-policy:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-scheduling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-storageclass:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-node:jar:6.13.3:compile [INFO] | | | \- org.snakeyaml:snakeyaml-engine:jar:2.7:compile [INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | | | \- com.squareup.okio:okio:jar:1.17.6:compile [INFO] | | \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:compile The kubernetes-client's maintainers do not want upgrade to okhttp 4.x because it's based on Kotlin, they recommend to exclude 3.x. Related documentation: https://github.com/fabric8io/kubernetes-client/blob/main/doc/KubernetesClientWithIPv6Clusters.md My proposed solution based on the above finding: I think we should exclude the 3.x version and switch to use okhttp 4.x. It is binary backwards compatible with okhttp 3.x. More details are here: https://square.github.io/okhttp/upgrading_to_okhttp_4/ ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 10 +++- hadoop-cloud/pom.xml | 45 ++++++++++++++ pom.xml | 4 +- resource-managers/kubernetes/core/pom.xml | 58 +++++++++++++++++++ .../kubernetes/integration-tests/pom.xml | 58 +++++++++++++++++++ 5 files changed, 171 insertions(+), 4 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index c7f4431de5957..8cf59ed7d33a5 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -154,6 +154,9 @@ json4s-scalap_2.13/4.0.7//json4s-scalap_2.13-4.0.7.jar jsr305/3.0.0//jsr305-3.0.0.jar jta/1.1//jta-1.1.jar jul-to-slf4j/2.0.16//jul-to-slf4j-2.0.16.jar +kotlin-stdlib-jdk7/2.0.10//kotlin-stdlib-jdk7-2.0.10.jar +kotlin-stdlib-jdk8/2.0.10//kotlin-stdlib-jdk8-2.0.10.jar +kotlin-stdlib/2.0.10//kotlin-stdlib-2.0.10.jar kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar kubernetes-client-api/6.13.3//kubernetes-client-api-6.13.3.jar kubernetes-client/6.13.3//kubernetes-client-6.13.3.jar @@ -189,7 +192,7 @@ log4j-api/2.22.1//log4j-api-2.22.1.jar log4j-core/2.22.1//log4j-core-2.22.1.jar log4j-layout-template-json/2.22.1//log4j-layout-template-json-2.22.1.jar log4j-slf4j2-impl/2.22.1//log4j-slf4j2-impl-2.22.1.jar -logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar +logging-interceptor/4.12.0//logging-interceptor-4.12.0.jar lz4-java/1.8.0//lz4-java-1.8.0.jar metrics-core/4.2.26//metrics-core-4.2.26.jar metrics-graphite/4.2.26//metrics-graphite-4.2.26.jar @@ -223,8 +226,9 @@ netty-transport-native-kqueue/4.1.110.Final/osx-x86_64/netty-transport-native-kq netty-transport-native-unix-common/4.1.110.Final//netty-transport-native-unix-common-4.1.110.Final.jar netty-transport/4.1.110.Final//netty-transport-4.1.110.Final.jar objenesis/3.3//objenesis-3.3.jar -okhttp/3.12.12//okhttp-3.12.12.jar -okio/1.17.6//okio-1.17.6.jar +okhttp/4.12.0//okhttp-4.12.0.jar +okio-jvm/3.9.0//okio-jvm-3.9.0.jar +okio/3.9.0//okio-3.9.0.jar opencsv/2.3//opencsv-2.3.jar opentracing-api/0.33.0//opentracing-api-0.33.0.jar opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar diff --git a/hadoop-cloud/pom.xml b/hadoop-cloud/pom.xml index 9c2f21e7ab617..a1e6405f66672 100644 --- a/hadoop-cloud/pom.xml +++ b/hadoop-cloud/pom.xml @@ -171,6 +171,51 @@ org.apache.hadoop hadoop-cos + + com.squareup.okhttp3 + okhttp + + + com.squareup.okio + okio + + + + + com.squareup.okhttp3 + okhttp + ${okhttp.version} + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + + + org.jetbrains.kotlin + kotlin-stdlib + + + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + ${kotlin-stdlib.version} + + + org.jetbrains + annotations + + + + + org.jetbrains.kotlin + kotlin-stdlib + ${kotlin-stdlib.version} + + + org.jetbrains + annotations + org.fusesource.leveldbjni 6.13.3 - 1.17.6 + 3.9.0 + 4.12.0 + 2.0.10 ${java.home} diff --git a/resource-managers/kubernetes/core/pom.xml b/resource-managers/kubernetes/core/pom.xml index fa0fd454ccc44..0952737e20d32 100644 --- a/resource-managers/kubernetes/core/pom.xml +++ b/resource-managers/kubernetes/core/pom.xml @@ -79,6 +79,64 @@ io.fabric8 kubernetes-httpclient-okhttp ${kubernetes-client.version} + + + com.squareup.okhttp3 + okhttp + + + com.squareup.okhttp3 + logging-interceptor + + + + + com.squareup.okhttp3 + okhttp + ${okhttp.version} + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + + + org.jetbrains.kotlin + kotlin-stdlib + + + + + com.squareup.okhttp3 + logging-interceptor + ${okhttp.version} + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + + + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + ${kotlin-stdlib.version} + + + org.jetbrains + annotations + + + + + org.jetbrains.kotlin + kotlin-stdlib + ${kotlin-stdlib.version} + + + org.jetbrains + annotations + + io.fabric8 diff --git a/resource-managers/kubernetes/integration-tests/pom.xml b/resource-managers/kubernetes/integration-tests/pom.xml index 518c5bc217071..241805df514ce 100644 --- a/resource-managers/kubernetes/integration-tests/pom.xml +++ b/resource-managers/kubernetes/integration-tests/pom.xml @@ -68,6 +68,64 @@ io.fabric8 kubernetes-client ${kubernetes-client.version} + + + com.squareup.okhttp3 + okhttp + + + com.squareup.okhttp3 + logging-interceptor + + + + + com.squareup.okhttp3 + okhttp + ${okhttp.version} + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + + + org.jetbrains.kotlin + kotlin-stdlib + + + + + com.squareup.okhttp3 + logging-interceptor + ${okhttp.version} + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + + + + + org.jetbrains.kotlin + kotlin-stdlib-jdk8 + ${kotlin-stdlib.version} + + + org.jetbrains + annotations + + + + + org.jetbrains.kotlin + kotlin-stdlib + ${kotlin-stdlib.version} + + + org.jetbrains + annotations + + org.apache.spark From 2509e77ee10946d40de670156dd939140b2cc3b8 Mon Sep 17 00:00:00 2001 From: Gabor Roczei Date: Mon, 19 Aug 2024 18:48:00 +0200 Subject: [PATCH 2/2] test kotlin-stdlib --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 -- hadoop-cloud/pom.xml | 11 ----------- resource-managers/kubernetes/core/pom.xml | 11 ----------- .../kubernetes/integration-tests/pom.xml | 11 ----------- 4 files changed, 35 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 8cf59ed7d33a5..e6b9f812b6229 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -154,8 +154,6 @@ json4s-scalap_2.13/4.0.7//json4s-scalap_2.13-4.0.7.jar jsr305/3.0.0//jsr305-3.0.0.jar jta/1.1//jta-1.1.jar jul-to-slf4j/2.0.16//jul-to-slf4j-2.0.16.jar -kotlin-stdlib-jdk7/2.0.10//kotlin-stdlib-jdk7-2.0.10.jar -kotlin-stdlib-jdk8/2.0.10//kotlin-stdlib-jdk8-2.0.10.jar kotlin-stdlib/2.0.10//kotlin-stdlib-2.0.10.jar kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar kubernetes-client-api/6.13.3//kubernetes-client-api-6.13.3.jar diff --git a/hadoop-cloud/pom.xml b/hadoop-cloud/pom.xml index a1e6405f66672..d73d6e51bc129 100644 --- a/hadoop-cloud/pom.xml +++ b/hadoop-cloud/pom.xml @@ -196,17 +196,6 @@ - - org.jetbrains.kotlin - kotlin-stdlib-jdk8 - ${kotlin-stdlib.version} - - - org.jetbrains - annotations - - - org.jetbrains.kotlin kotlin-stdlib diff --git a/resource-managers/kubernetes/core/pom.xml b/resource-managers/kubernetes/core/pom.xml index 0952737e20d32..64406b41bf701 100644 --- a/resource-managers/kubernetes/core/pom.xml +++ b/resource-managers/kubernetes/core/pom.xml @@ -116,17 +116,6 @@ - - org.jetbrains.kotlin - kotlin-stdlib-jdk8 - ${kotlin-stdlib.version} - - - org.jetbrains - annotations - - - org.jetbrains.kotlin kotlin-stdlib diff --git a/resource-managers/kubernetes/integration-tests/pom.xml b/resource-managers/kubernetes/integration-tests/pom.xml index 241805df514ce..a82a083f87c62 100644 --- a/resource-managers/kubernetes/integration-tests/pom.xml +++ b/resource-managers/kubernetes/integration-tests/pom.xml @@ -105,17 +105,6 @@ - - org.jetbrains.kotlin - kotlin-stdlib-jdk8 - ${kotlin-stdlib.version} - - - org.jetbrains - annotations - - - org.jetbrains.kotlin kotlin-stdlib