zstd-sys: impl wasm32-unknown-unknown support #139

quininer · 2022-01-26T18:00:45Z

This PR adds a simple wasm shim to zstd-sys to support wasm32-unknown-unknown arch.

The legacy and dict_builder cannot be compiled on this shim because they depend on stdio.h, so I added a new feature cfg to turning off dict_builder code.

You can test it with wasmtime

> cd zstd-safe/zstd-sys/
> cargo build --target wasm32-unknown-unknown --no-default-features --example it_work
> wasmtime target/wasm32-unknown-unknown/debug/examples/it_work.wasm --invoke test_compress

gyscos · 2022-01-26T20:42:39Z

Hi, and thanks for the work!

I think without dictBuilder, the functions from zdict don't have a definition anymore, and should not be used.

This means that the bindings file needs to depend on this. In practice, the zstd.h file used to generate the bindings should not include zdict.h when the dict builder feature is not enabled.

Rather than adding yet another dimension to the selection of the bindings file, we could split this bindings into the main zstd part and the dict part, and include each file conditionally. I pushed something like that to the bindings_split branch if you want to have a look.
If it looks fine, I'll merge the split, and rebase this PR to not include the zdict bindings without the feature.

We may also want to forward this dict_builder feature from zstd and zstd-safe, and there too conditionally enable the relevant functions.

quininer · 2022-01-27T03:15:08Z

Yes, it would be nice to split them. Thanks for your quick work.

zstd-safe/zstd-sys/build.rs

gyscos · 2022-01-28T15:49:01Z

Looking good! One last thing: we'll want to add dict_builder features to the zstd-safe and zstd crates, and forward it downstream.

We'll also want to conditionally hide the relevant types in these crates depending on this feature.

gyscos · 2022-01-28T16:22:43Z

examples/train.rs

+    #[cfg(feature = "zdict_builder")]
+    {
+        let dict = zstd::dict::from_files(&files, size).unwrap();

-    let mut dict_reader: &[u8] = &dict;
-    io::copy(&mut dict_reader, &mut io::stdout()).unwrap();
+        let mut dict_reader: &[u8] = &dict;
+        io::copy(&mut dict_reader, &mut io::stdout()).unwrap();
+    }


You can specify that an example requires some features. In Cargo.toml:

[[example]] name = "train" required-features = ["zdict_builder"]

This way, in the example itself, you can just assume the feature is set.

gyscos · 2022-01-28T18:54:58Z

Thanks again for the work!

inspired by zstd-rs (gyscos/zstd-rs#139)

quininer added 2 commits January 28, 2022 21:57

impl wasm32-unknown-unknown support

ea28d3d

Add zdict feature for zstd-sys

fe55976

quininer force-pushed the wasm-support branch from e7a7c98 to fe55976 Compare January 28, 2022 14:31