-
Notifications
You must be signed in to change notification settings - Fork 834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support building object_store
and parquet
on wasm32-unknown-unknown target
#2896
Conversation
object_store
and parquet
on wasm32-unknown-unknown target
.github/workflows/parquet.yml
Outdated
@@ -119,6 +119,9 @@ jobs: | |||
- name: Check compilation --all-targets --no-default-features --features json | |||
run: | | |||
cargo check -p parquet --all-targets --no-default-features --features json | |||
- name: Check compilation wasm32-unknown-unknown | |||
run: | | |||
cargo check -p parquet --no-default-features --target wasm32-unknown-unknown |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI compiling with lz4 appears to fail
Trying to compile object_store with non-default features returns an error about getrandom which appears related to https://docs.rs/getrandom/latest/getrandom/#webassembly-support I'm not familiar enough with WASM to know if this is an issue or not? |
* Added cargo check step to parquet workflow for wasm32-unknown-unknown * Added compile-time warning for unsupported cloud features when compiling with wasm32 * Added cargo check features to the parquet github workflow. * Added a section to the README.md for parquet
da40737
to
b141ded
Compare
Yep, |
Benchmark runs are scheduled for baseline = 4d5d10d and contender = f812d2c. f812d2c is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
As a note on this, I've gotten zstd and lz4-flex to both compile to wasm, so it's possible to have better wasm parquet support if you're interested. For zstd I think it should work out of the box with arrow-rs IIRC (you may need to make sure you're building against the right clang, especially if on Mac, see https://github.com/kylebarron/parquet-wasm/pull/42/files). For lz4, we'd need to update arrow-rs to switch between lz4 and lz4-flex depending on whether building for wasm or not. See jorgecarleitao/parquet2#124 for the relevant code in parquet2. As of that, parquet2 builds to |
Which issue does this PR close?
Closes #180.
Rationale for this change
My goal is to get
datafusion
building and running on thewasm32-unknown-unknown
target, and gettingobject_store
andparquet
are the first steps.What changes are included in this PR?
This PR adds some conditional compilation in
object_store
(mostly around disabling local filesystem access), as well as shifting some dependencies to dev, and removing un-used features in deps (i.e. tokio).Are there any user-facing changes?
None expected, this is purely a build-support change.