Wasm libzim package should include precompiled library.

[mgautierfr]

Until now, precompiled libzim packages include libzim.so.
This libzim.so include its dependencies as it is linked statically with them.

But on wasm it seems it is not the case. So we have to:

• Check how wasm's libzim.a could be linked statically with dependencies (if possible) or
• Include dependencies .a in the package

[Jaifroid]

Do you mean the way the dependencies are linked in https://github.com/openzim/javascript-libzim/blob/master/Makefile#L42 to make libzim.a? If the libzim.a produced here contains those dependencies, then it should simply be a matter of compiling it with the JS wrapper using em++ on openzim/javascript-libzim. However, it's not working, see post in #548 (comment).

[mgautierfr]

Do you mean the way the dependencies are linked in openzim/javascript-libzim@master/Makefile#L42 to make libzim.a?

The pointed line say to make that it need to build other dependencies before building libzim.a.
It is "just" an ordering instruction.
The fact that libzim.a actually need other dependencies is independent of that.
kiwix-build already does that (ordering builds).

If the libzim.a produced here contains those dependencies, then it should simply be a matter of compiling it with the JS wrapper using em++ on openzim/javascript-libzim. However, it's not working, see post in #548 (comment).

libzim.a doesn't contains those dependencies (both with kiwix-build and javascript-libzim's makefile).
This is why you specify all the dependencies at compilation of demo_file_api.cpp here : https://github.com/openzim/javascript-libzim/blob/master/Makefile#L52

This issue is about either make libzim.a contains those dependencies or package the dependencies along with libzim.a.
I will probably do the latter.

[Jaifroid]

OK, thanks @mgautierfr. That sounds good. When you have a libzim.a with dependencies built in, I'll be very happy to test compiling it to WASM + bindings with Emscripten, and then test the WASM itself in our test implementation. It should then be relatively easy to integrate this libzim.a binary into a nightly CI at openzim/javascript-libzim.

[Jaifroid]

@mgautierfr I compiled an Emscripten WASM from the libzim.a and dependencies in /data/openzim/release/libzim/libzim_wasm-emscripten-8.1.0.tar.gz (date-stamp: 01/12/22 21:42) from master.download.kiwix.org). I compiled by making a clean build directory at /src/build/ and placing the lib and include directories from the package in it. I moved the contents of lib/x86_64-linux-gnu so that it is directly under lib, because that is how the libs are referenced in the em++ command below. I then compiled with this command, which is the command that produces a working WASM and wrapper when I compile everything from source on kiwix/javascript-libzim:

em++ -o libzim-wasm.js --bind libzim_bindings.cpp -I/src/build/include -L/src/build/lib -lzim -llzma -lzstd -lxapian -lz -licui18n -licuuc -licudata -lpthread -lm -fdiagnostics-color=always -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror --pre-js prejs_file_api.js --post-js postjs_file_api.js -s WASM=1 -s DISABLE_EXCEPTION_CATCHING=0 -s "EXPORTED_RUNTIME_METHODS=['ALLOC_NORMAL','printErr','ALLOC_STACK','print']" -s DEMANGLE_SUPPORT=1 -s TOTAL_MEMORY=83886080 -s ALLOW_MEMORY_GROWTH=1 -lworkerfs.js

The good news is that this appears to compile the WASM and wrapper without complaining (they are both produced, and no error is shown). The not-so-good news is that this WASM is producing fatal exceptions when used within the test implementation on javascript-libzim. It initializes but throws an obscure exception after initialization. Then, attempting to load an article or do a full-text search produces a more informative exception chain, which is a mixture of JS and WebAssembly mimicking C++.The errors are clearly coming from the WASM, not the wrapper (text in white in the console below the red JS errors):

---

I'm afraid I haven't got much of a clue about how to fix this. I suspect that the dependencies are not being compiled with the right switches or configuration overrides for Emscripten, but I really don't know much more about all this than what appears in the Makefile on openzim/javascript-libzim. I sure hope you have more of a clue than I do... But it seems possible that the issue is in the precompiled binaries. What Emscripten configuration are you doing to compile these binaries?

I've had to learn a lot just to get to the stage of being able to test-compile this, but I really don't think I can fix this by myself. I can commit the built files and modified demo to a branch on the repo if that would be useful. Note I built this using the branch in my workflow PR: openzim/javascript-libzim#14, so the compiled WASM and wrapper are now named libzim-wasm.* (and are correctly linked in the demo), rather than a.out.*.

Feel free to move this comment to a new issue if you prefer, or to re-open this one if you agree that the issue is with the binaries and not with the binding.

[mgautierfr]

@Jaifroid I've open the issue openzim/libzim#751 for this