Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded #4426
Conversation
|
Do you think it's valuable cherry-pick to branch-4.16? |
Yes, however there will be the need to solve the grpc incompatibility issue: https://lists.apache.org/thread/ysr7041jjdgqo0f0bghm9rswzmsddf3v .
|
|
@lhotari Okay, my colleague @zhiheng123 is trying to solve branch-4.16 problem. For me, I would like to finish the 4.17.1 release first. |
|
@shoothzj @zhiheng123 Pulsar CI is also failing in grpc with 4.1.111.Final. So I'm currently investigating similar issues. |
|
@lhotari @zhiheng123 has do trying in #4420 and also his fork repo. I think we can share some information about trying upgrade gRPC later. |
|
Results on Pulsar side: apache/pulsar#22892 (comment) |
|
It seems that grpc-netty dependency will have to be excluded and replaced by grpc-netty-shaded since grpc is compatible only with specific Netty versions. |
|
@lhotari Nice work! I think it might be the right way. It can solve both master branch error and |
|
Unfortunately jetcd-core isn't compatible with grpc-netty-shaded. There are conflicts with |
|
I created etcd-io/jetcd#1370 to jetcd project about the issue. |
|
Custom shading of jetcd-core and jetcd-test will slow down the build which is unfortunate. One possibility would be to exclude the modules from the main build (or move them to another repository) and publish them only periodically when the version of jetcd gets updated. Not great. It's too bad that this is causing additional headaches. |
|
In Pulsar, I found a reasonable solution by shading jetcd-core in a way where vertx is relocated and included, but grpc is I'll apply a similar solution to Bookkeeper soon. |
lhotari
force-pushed
the
lh-netty-4.1.111
branch
from
d2f33b1 to
4c7aaad
Compare
lhotari
force-pushed
the
lh-netty-4.1.111
branch
from
4c7aaad to
527dc8d
Compare
- fixes the issue with transient dependencies
lhotari
changed the title
|
@shoothzj @zhiheng123 @dlg99 Please review now. I was finally able to find a solution where grpc-netty is replaced with grpc-netty-shaded. That fixes the compatibility issues. One of the challenges was to preserve IntelliJ support and provide correct metadata for the shaded jetcd library that has a direct dependency on grpc-netty and doesn't work with grpc-netty-shaded. The shading handles the relocation so that jetcd-core and it's transient dependency vertx-grpc will be relocated to use grpc-netty-shaded packages without needing to shade and relocate grpc libraries. |
|
I didn't see |
@shoothzj I didn't add that exclusion in this PR, I just moved it and made it consistent. It's possible that it is obsolete. |
|
@shoothzj I removed the obsolete exclusion of bcpkix-jdk15on. PTAL |
…4426) [Netty 4.1.111.Final](https://netty.io/news/2024/06/11/4-1-111-Final.html) contains important fixes: - netty/netty#14086 - netty/netty#14093 On the Pulsar side these address Broker stability when TLS with Bookkeeper V2 protocol is used between Broker and Bookies. The Bookkeeper client didn't contain a workaround for the Netty SslHandler "feature" like Pulsar did. That's why Bookkeeper client was impacted. Netty 4.1.111.Final will address those stability issues with the Bookkeeper client when using V2 protocol over TLS. - Upgrade Netty to 4.1.111.Final - Switch to use grpc-netty-shaded instead of grpc-netty - grpc-java is not compatible with Netty 4.1.111.Final. [grpc-java supports only specific Netty versions](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty). The solution is to use grpc-netty-shaded instead of grpc-netty dependency. - Add module to shade jetcd-core partially so that it can work with grpc-netty-shaded. - jetcd-core library and it's transient dependency vertx-grpc has a direct dependency on grpc-netty and it doesn't work with grpc-netty-shaded without this solution. - this solution avoids the need to shade grpc libraries completely which would cause a lot of duplication and increase build times. (cherry picked from commit a95f698)
…4426) [Netty 4.1.111.Final](https://netty.io/news/2024/06/11/4-1-111-Final.html) contains important fixes: - netty/netty#14086 - netty/netty#14093 On the Pulsar side these address Broker stability when TLS with Bookkeeper V2 protocol is used between Broker and Bookies. The Bookkeeper client didn't contain a workaround for the Netty SslHandler "feature" like Pulsar did. That's why Bookkeeper client was impacted. Netty 4.1.111.Final will address those stability issues with the Bookkeeper client when using V2 protocol over TLS. - Upgrade Netty to 4.1.111.Final - Switch to use grpc-netty-shaded instead of grpc-netty - grpc-java is not compatible with Netty 4.1.111.Final. [grpc-java supports only specific Netty versions](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty). The solution is to use grpc-netty-shaded instead of grpc-netty dependency. - Add module to shade jetcd-core partially so that it can work with grpc-netty-shaded. - jetcd-core library and it's transient dependency vertx-grpc has a direct dependency on grpc-netty and it doesn't work with grpc-netty-shaded without this solution. - this solution avoids the need to shade grpc libraries completely which would cause a lot of duplication and increase build times. (cherry picked from commit a95f698)
…pache#4426) ### Motivation [Netty 4.1.111.Final](https://netty.io/news/2024/06/11/4-1-111-Final.html) contains important fixes: - netty/netty#14086 - netty/netty#14093 On the Pulsar side these address Broker stability when TLS with Bookkeeper V2 protocol is used between Broker and Bookies. The Bookkeeper client didn't contain a workaround for the Netty SslHandler "feature" like Pulsar did. That's why Bookkeeper client was impacted. Netty 4.1.111.Final will address those stability issues with the Bookkeeper client when using V2 protocol over TLS. ### Changes - Upgrade Netty to 4.1.111.Final - Switch to use grpc-netty-shaded instead of grpc-netty - grpc-java is not compatible with Netty 4.1.111.Final. [grpc-java supports only specific Netty versions](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty). The solution is to use grpc-netty-shaded instead of grpc-netty dependency. - Add module to shade jetcd-core partially so that it can work with grpc-netty-shaded. - jetcd-core library and it's transient dependency vertx-grpc has a direct dependency on grpc-netty and it doesn't work with grpc-netty-shaded without this solution. - this solution avoids the need to shade grpc libraries completely which would cause a lot of duplication and increase build times.
### Motivation There is a potential jar shading issue introduced in #4426 that causes `NoClassDefFoundError` when connecting to an etcd metadata store. The `jetcd-core-shaded` module was introduced in #4426 to address the compatibility issues between jetcd-core’s grpc-java dependency and Netty. You can find more details [here][1] and in the [grpc-java documentation][2]. [1]: #4426 (comment) [2]: https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty Currently, we use `unpack-shaded-jar` execution unpacks the shaded jar produced by `maven-shade-plugin:shade` into the `jetcd-core-shaded/target/classes` directory. However, the classes in this directory conflict with its dependencies. If the `maven-shade-plugin:shade` runs again without cleaning this directory, it can produce an incorrect shaded jar. You can replicate and verify this issue with the following commands: ```shell # Step 1: Clean the build directory mvn clean # Step 2: Perform an install and unpack the shaded jar into a directory. # Verify the import statement for `io.netty.handler.logging.ByteBufFormat` in # `org/apache/pulsar/jetcd/shaded/io/vertx/core/net/NetClientOptions.class`. # The correct import should be: # `import io.grpc.netty.shaded.io.netty.handler.logging.ByteBufFormat;`. mvn install unzip $M2_REPO/org/apache/bookkeeper/metadata/drivers/jetcd-core-shaded/4.18.0-SNAPSHOT/jetcd-core-shaded-4.18.0-SNAPSHOT-shaded.jar \ -d metadata-drivers/jetcd-core-shaded/target/first-classes # Step 3: Run the install command again without cleaning. # The unpacked jar from the previous step will persist in `target/classes`. # Unpack the shaded jar into a different directory (e.g., second-classes) and check the import. # The incorrect import will be: # `import io.grpc.netty.shaded.io.grpc.netty.shaded.io.netty.handler.logging.ByteBufFormat;`. mvn install unzip $M2_REPO/org/apache/bookkeeper/metadata/drivers/jetcd-core-shaded/4.18.0-SNAPSHOT/jetcd-core-shaded-4.18.0-SNAPSHOT-shaded.jar \ -d metadata-drivers/jetcd-core-shaded/target/second-classes # Step 4: Use IntelliJ IDEA's "Compare Directories" tool to compare the `first-classes` # and `second-classes` directories. The differences in imports should become apparent. ``` We can remove the attach and unpack configurations, and it should work fine. This issue has already been [reported][3] in apache/pulsar, and a similar [patch][patch] has addressed it. [3]: apache/pulsar#23513 [patch]: apache/pulsar#23604
Motivation
Netty 4.1.111.Final contains important fixes:
On the Pulsar side these address Broker stability when TLS with Bookkeeper V2 protocol is used between Broker and Bookies. The Bookkeeper client didn't contain a workaround for the Netty SslHandler "feature" like Pulsar did. That's why Bookkeeper client was impacted. Netty 4.1.111.Final will address those stability issues with the Bookkeeper client when using V2 protocol over TLS.
Changes