[WIP] Make pyarrow and msgpack optional #8218
Conversation
bkyryliuk
force-pushed
the
bogdan/make_pyarrow_optional
branch
from
1f2169a to
87591f7
Compare
bkyryliuk
changed the title
bkyryliuk
force-pushed
the
bogdan/make_pyarrow_optional
branch
from
54b4d1a to
8a14030
Compare
|
Based on the performance gains of msgpack + pyarrow over raw JSON, it would be ideal to standardize on this serialization approach going forward, pending community testing. I'm not familiar with bazel, but can you expand a bit on the difficulties there? |
bkyryliuk
changed the title
|
@robdiciuccio pyarrow has only binary distributions for the 0.14.1 and https://github.com/apache/arrow/blob/master/python/setup.py#L284 shells out to the cmake during install. There is a workaround via cloning, defining bazel rules and building it internally. I am all in for the standardization, my only preference is to have it optional during the community |
bkyryliuk
force-pushed
the
bogdan/make_pyarrow_optional
branch
from
73f63d8 to
54fbef0
Compare
bkyryliuk
force-pushed
the
bogdan/make_pyarrow_optional
branch
2 times, most recently
from
0b5f6d3 to
605692d
Compare
Move pyarrow back for unit tests Disable msgpack by default Fix tests
bkyryliuk
force-pushed
the
bogdan/make_pyarrow_optional
branch
from
605692d to
ee98f94
Compare
Codecov Report
@@ Coverage Diff @@
## master #8218 +/- ##
=========================================
+ Coverage 66.8% 66.8% +<.01%
=========================================
Files 449 449
Lines 22703 22705 +2
Branches 2366 2366
=========================================
+ Hits 15166 15168 +2
Misses 7399 7399
Partials 138 138
Continue to review full report at Codecov.
|
CATEGORY
Choose one
SUMMARY
Makes msgpack and pyarrow dependency optional.
Follow up to the #8069
Some context, we use bazel @ dropbox and bazelyfing pyarrow is quite challenging due to the a bit messy build there e.g. shelling out to cmake.
It would be great to keep not only functionality but also pip dependency optional as well. Please let me know what you think.
REVIEWERS