-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Apple M1 architecture #153
Comments
Thanks for confirming there are not problems on M1. The current issue is the lack of M1 support on Github actions - I use the build artifacts from the GH actions (and GCC compile farm) and include them in the final jars. Let's hope the M1 build problems get resolved sooner: actions/runner-images#2187, actions/runner#805 Another option is to cross-compile it. I have to research a little bit about this, though the latest installed xcode (12.2) should support it. Edit: it looks it support creating |
Seems like cross-compile is going to be the only option for a while, even after Github fixes that one - sbt (and most things JVM) are mostly broken. Same thing for golang and therefore Docker. |
I have a build script that uses just the cc and produces the binary, I use it for cross-compiling and building on more exotic architectures. Downside is that I don't run the test suite on them. |
Could you put it up on a new branch? My exotic architecture is a 2015 MBP (sdkman -> JVMs, SBTs, scala-langs...) and a 2020 MBA ("native" homebrew -> cmake zstd)) Could it be adapted to run the sbt tests on an sbt-container image prior to packaging up the dylibs from an as-yet-nonexistent m1 action? |
@jasonnerothin , looks I was able to cross-compile it with Xcode using github actions, here is the result:
One question, can you check the architecture folder in the JAR you compiled on M1? It should be |
This program prints this output: OS: mac_os_x arch: aarch64 Note: I'm running Azul Systems' JVM. It is the only one that runs, last time I checked. |
Great, thanks. So next release should support M1. I will be releasing later today with v1.4.7 upstream. |
Hmm, I will wait for facebook/zstd#2428 to be resolved as it looks dangerous. |
@jasonnerothin pushed v1.4.7-1 to maven-central, it also contains the fix for the above bug. Can you test it? |
Better try 1.4.7-3 |
Thank you for releasing 1.4.7-3, but Facebook seems to release a hot fix 1.4.8. |
@dongjoon-hyun , yes I already have the same fix. Doesn't hurt to release also 1.4.8 |
Nice! Thanks for confirming, @luben ! |
@dongjoon-hyun , I just pushed v1.4.8-1 the only functional change is adding an assert for the buffer alignment. |
### What changes were proposed in this pull request? This PR aims to upgrade Zstd library to 1.4.8. ### Why are the changes needed? This will bring Zstd 1.4.7 and 1.4.8 improvement and bug fixes and the following from `zstd-jni`. - https://github.com/facebook/zstd/releases/tag/v1.4.7 - https://github.com/facebook/zstd/releases/tag/v1.4.8 - luben/zstd-jni#153 (Apple M1 architecture) ### Does this PR introduce _any_ user-facing change? This will unblock Apple Silicon usage. ### How was this patch tested? Pass the CIs. Closes #30848 from dongjoon-hyun/SPARK-33843. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
### What changes were proposed in this pull request? This PR aims to upgrade Zstd library to 1.4.8. ### Why are the changes needed? This will bring Zstd 1.4.7 and 1.4.8 improvement and bug fixes and the following from `zstd-jni`. - https://github.com/facebook/zstd/releases/tag/v1.4.7 - https://github.com/facebook/zstd/releases/tag/v1.4.8 - luben/zstd-jni#153 (Apple M1 architecture) ### Does this PR introduce _any_ user-facing change? This will unblock Apple Silicon usage. ### How was this patch tested? Pass the CIs. Closes #30848 from dongjoon-hyun/SPARK-33843. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 00642ee) Signed-off-by: Dongjoon Hyun <[email protected]>
This is what I'm seeing: Unsupported OS/arch, cannot find /darwin/aarch64/libzstd-jni.dylib or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in your system. |
@jasonnerothin , There should be other error strings on the previous lines. Can you tell me what they say. Also which line is the exception thrown? Double checked the native artifacts packed in the jar and there is nothing suspicious there:
|
Another thing that may help is to extract the library from the jar, e.g. 'jar -xf zstd-jni.jar' and find if there is some break in the linking with |
created: darwin/ |
|
@jasonnerothin , any help with the error message? It should at least have line number, but also there may be another error messages before the |
^^ two comments back is partial output of ^ last comment says that the |
I don't think the
I am not sure about the link to "/User/runner/..." : I don't think this is an issue as 1.4.5 was build using the old gcc toolchain and it's working. With 1.4.7 I switched the toolchain to use the xcode compiler in order to support the cross-compilation to M1. Can you try with v1.4.8-1? |
Okay, so it looks like the issue is that Apple has moved its "User Cache" to somewhere else. otool is reporting that the darwin dylib is looking for However, installing XCode installs one for each of tvOS, iOS, watchOS. I am symlinking to see if I can get the scala tests to run from IntelliJ. |
Seems to be working with latest |
Thanks for confirming. I am not sure where the fix came from - most probably something changed on the GH workers. @jasonnerothin , can you confirm |
We only had this as an enterprise dependency. The server could use it through a transititive dependency of kafka-client. This however was version 1.4.5-6 which has an issue on M1 Macs: luben/zstd-jni#153
We only had this as an enterprise dependency. The server could use it through a transititive dependency of kafka-client. This however was version 1.4.5-6 which has an issue on M1 Macs: luben/zstd-jni#153
Zstd builds and seems stable on Apple M1. I'd like to provide a PR.
What would the logistics need to be? (No Actions for M1 yet on Github.)
The text was updated successfully, but these errors were encountered: