Create hoodie-presto bundle jar. fixes #567 #571
Conversation
|
|
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.hadoop</groupId> |
There was a problem hiding this comment.
I am wondering if we should just reuse the hadoop jars from Presto runtime as-is?
They are not shaded below and thus could bring in new jars that could conflict with ones already presto uses?
There was a problem hiding this comment.
+1 Additionally, even if the current presto versions runs fine, upgrades may cause issues since the runtime dependencies may change.
There was a problem hiding this comment.
It looks like the org.apache.hadoop* jars are excluded from shading already - cd78935#diff-9b589d19949c0d8c4a89834d2d4d80b2R169 but I will check for other jars too.
Exclude common dependencies that are available in Presto
bhasudha
force-pushed
the
presto-bundle
branch
from
cd78935 to
1021f2f
Compare
| <exclude>org.apache.hive:*</exclude> | ||
| <exclude>org.apache.derby:derby</exclude> | ||
| <!--Already a dependency in presto-hive connector--> | ||
| <exclude>org.apache.thrift:*</exclude> |
There was a problem hiding this comment.
Excluded thrift since its available in Presto.
| <!--Already a dependency in presto-hive connector--> | ||
| <exclude>org.apache.thrift:*</exclude> | ||
| <!--Provided by aws-java-sdk-core dependency in presto-hive connector--> | ||
| <exclude>org.apache.httpcomponents:*</exclude> |
There was a problem hiding this comment.
same. This is also available in Presto. So excluding it here.
|
I ensured either the dependencies are relocated in this bundle or are excluded (in which case Presto will bring in these dependencies). I was able to re-run the same queries with this new jar. |
|
lgtm.. thanks for the contribution.. |
|
@bhasudha can you please update the docs as well after you successfully deploy the bundle |
Added hoodie-presto-bundle module under packaging/ . Tested this jar for presto queries on both copy-on-write and merge-on-read tables. Please let me know what you think!