[ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 #4822

tbonelee · 2024-09-10T11:15:25Z

This PR is reopening #4806

What is this PR for?

Previous pty4j dependency version does not support Silicon OSX
So %sh.terminal interpreter did not work on Silicon OSX.
To fix this, I bumped pty4j version to 0.12.35

The previous pty4j dependency version does not support macOS on Apple Silicon.
As a result, the %sh.terminal interpreter did not work on Apple Silicon macOS.
To fix this issue, I have updated the pty4j verison to 0.12.35

What type of PR is it?

Bug Fix

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/6071

How should this be tested?

Run %sh.terminal interpreter
- Main focus : Test on a Mac with Apple Silicon to ensure it works correctly.
- The native files for other OS/architectures have also changed to match the library version downgrade. It would be helpful to verify that it functions properly on these other OS/architectures as well.

Screenshots (if appropriate)

Before

The interpreter was not working, and these error logs were generated:

ERROR [2024-09-01 14:10:18,305] ({Thread-11} PtyHelpers.java[<clinit>]:231) - Cannot load implementation of interface com.pty4j.unix.PtyHelpers$OSFacade
java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/bt/g5zw1jfn36783j9xvwl6pmg80000gn/T/jna-738948474/jna9752198374376500371.tmp
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2638)
	at java.base/java.lang.Runtime.load0(Runtime.java:768)
	at java.base/java.lang.System.load(System.java:1854)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922)
	at com.sun.jna.Native.<clinit>(Native.java:190)
	at com.pty4j.unix.macosx.OSFacadeImpl.<clinit>(OSFacadeImpl.java:91)
	at com.pty4j.unix.PtyHelpers$1.call(PtyHelpers.java:200)
	at com.pty4j.unix.PtyHelpers$1.call(PtyHelpers.java:196)
	at com.pty4j.util.LazyValue.getValue(LazyValue.java:25)
	at com.pty4j.unix.PtyHelpers.getOsFacade(PtyHelpers.java:244)
	at com.pty4j.unix.PtyHelpers.<clinit>(PtyHelpers.java:228)
	at com.pty4j.unix.Pty.ptyMasterOpen(Pty.java:141)
	at com.pty4j.unix.Pty.openMaster(Pty.java:193)
	at com.pty4j.unix.Pty.<init>(Pty.java:56)
	at com.pty4j.unix.UnixPtyProcess.<init>(UnixPtyProcess.java:83)
	at com.pty4j.PtyProcessBuilder.start(PtyProcessBuilder.java:111)
	at com.pty4j.PtyProcess.exec(PtyProcess.java:78)
	at com.pty4j.PtyProcess.exec(PtyProcess.java:49)
	at org.apache.zeppelin.shell.terminal.service.TerminalService.initializeProcess(TerminalService.java:88)
	at org.apache.zeppelin.shell.terminal.service.TerminalService.lambda$onTerminalReady$0(TerminalService.java:62)
	at java.base/java.lang.Thread.run(Thread.java:829)
 INFO [2024-09-01 14:10:18,324] ({Thread-11} ExtractedNative.java[doInit]:127) - Extracted pty4j native in 17 ms
ERROR [2024-09-01 14:10:18,325] ({Thread-11} PtyHelpers.java[<clinit>]:237) - Cannot load native pty executor library
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
	at com.pty4j.unix.NativePtyExecutor.<init>(NativePtyExecutor.java:14)
	at com.pty4j.unix.PtyHelpers$2.call(PtyHelpers.java:222)
	at com.pty4j.unix.PtyHelpers$2.call(PtyHelpers.java:218)
	at com.pty4j.util.LazyValue.getValue(LazyValue.java:25)
	at com.pty4j.unix.PtyHelpers.getPtyExecutor(PtyHelpers.java:254)
	at com.pty4j.unix.PtyHelpers.<clinit>(PtyHelpers.java:234)
	at com.pty4j.unix.Pty.ptyMasterOpen(Pty.java:141)
	at com.pty4j.unix.Pty.openMaster(Pty.java:193)
	at com.pty4j.unix.Pty.<init>(Pty.java:56)
	at com.pty4j.unix.UnixPtyProcess.<init>(UnixPtyProcess.java:83)
	at com.pty4j.PtyProcessBuilder.start(PtyProcessBuilder.java:111)
	at com.pty4j.PtyProcess.exec(PtyProcess.java:78)
	at com.pty4j.PtyProcess.exec(PtyProcess.java:49)
	at org.apache.zeppelin.shell.terminal.service.TerminalService.initializeProcess(TerminalService.java:88)
	at org.apache.zeppelin.shell.terminal.service.TerminalService.lambda$onTerminalReady$0(TerminalService.java:62)
	at java.base/java.lang.Thread.run(Thread.java:829)
ERROR [2024-09-01 14:10:18,325] ({Thread-11} TerminalService.java[lambda$onTerminalReady$0]:64) - Cannot load implementation of interface com.pty4j.unix.PtyHelpers$OSFacade
java.lang.RuntimeException: Cannot load implementation of interface com.pty4j.unix.PtyHelpers$OSFacade
	at com.pty4j.unix.PtyHelpers.getOsFacade(PtyHelpers.java:247)
	at com.pty4j.unix.PtyHelpers.getInstance(PtyHelpers.java:262)
	at com.pty4j.unix.Pty.ptyMasterOpen(Pty.java:141)
	at com.pty4j.unix.Pty.openMaster(Pty.java:193)
	at com.pty4j.unix.Pty.<init>(Pty.java:56)
	at com.pty4j.unix.UnixPtyProcess.<init>(UnixPtyProcess.java:83)
	at com.pty4j.PtyProcessBuilder.start(PtyProcessBuilder.java:111)
	at com.pty4j.PtyProcess.exec(PtyProcess.java:78)
	at com.pty4j.PtyProcess.exec(PtyProcess.java:49)
	at org.apache.zeppelin.shell.terminal.service.TerminalService.initializeProcess(TerminalService.java:88)
	at org.apache.zeppelin.shell.terminal.service.TerminalService.lambda$onTerminalReady$0(TerminalService.java:62)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/bt/g5zw1jfn36783j9xvwl6pmg80000gn/T/jna-738948474/jna9752198374376500371.tmp
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2638)
	at java.base/java.lang.Runtime.load0(Runtime.java:768)
	at java.base/java.lang.System.load(System.java:1854)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922)
	at com.sun.jna.Native.<clinit>(Native.java:190)
	at com.pty4j.unix.macosx.OSFacadeImpl.<clinit>(OSFacadeImpl.java:91)
	at com.pty4j.unix.PtyHelpers$1.call(PtyHelpers.java:200)
	at com.pty4j.unix.PtyHelpers$1.call(PtyHelpers.java:196)
	at com.pty4j.util.LazyValue.getValue(LazyValue.java:25)
	at com.pty4j.unix.PtyHelpers.getOsFacade(PtyHelpers.java:244)
	at com.pty4j.unix.PtyHelpers.<clinit>(PtyHelpers.java:228)
	... 10 more

After

The interpreter is now working as expected, just like any ordinary shell.

Questions:

Does the license files need to update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

tbonelee · 2024-09-10T14:34:29Z

@pan3793 Could you please take a loot at this PR again?

pan3793 · 2024-09-10T14:38:59Z

nice!

a quick question, I saw you updated some so/dll files, why should we ship those files in the codebase? could they be retrieved through third-parity dep approach (e.g. include some jars)

tbonelee · 2024-09-10T15:33:38Z

@pan3793
So, are you suggesting a way to make it work automatically using the native files included in the pty4j jar file?
Honestly, I'm not entirely sure what the original intention was.
I tried looking through the feature PR (#3359) to see if there were any related comments, but I couldn't find any.
Maybe they wanted more control over which directory the native files get copied to?
That's just a guess though.
Since I wasn't sure about the exact reason, I didn't feel comfortable removing them. What do you think?

pan3793 · 2024-09-11T05:00:22Z

I saw another member also concerns about the binary files including #3359 (comment)

.so, .exe, .dll etc cannot be in the git source repo

as we are touching this part, let's address this issue.

tbonelee · 2024-09-11T09:40:39Z

@pan3793
I checked that pty4j library copies the native files when no pty4j.preferred.native.folder properties are given.
So I remove native files in the repo and codes related to these and checked that it works on my local environment. (M1 MacBook)

It would be great if we could check whether it works well on other environments too, and if anyone could point out the reason why the binary files are included, that would be helpful as well.

pan3793 · 2024-09-11T09:42:43Z

@tbonelee Good job! I will find time to test it in a Linux environment in a few days.

pan3793 · 2024-09-17T21:02:41Z

Verified on Linux arm64

shell/PTY4J.md

pan3793 · 2024-09-19T06:06:16Z

Thanks, merged to master for 0.12.0

tbonelee added 2 commits September 10, 2024 20:10

Bump pty4j version to 0.12.35

93cb8c5

Downgrade pty4j version to 0.12.10-jdk8

f36e562

Remove native files in Zeppelin repo

972672c

pan3793 reviewed Sep 17, 2024

View reviewed changes

shell/PTY4J.md Outdated Show resolved Hide resolved

pan3793 changed the title ~~[ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 (reopening of #4806)~~ [ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 Sep 17, 2024

Remove unnecessary docs

68f9171

pan3793 approved these changes Sep 18, 2024

View reviewed changes

pan3793 merged commit 876d1dc into apache:master Sep 19, 2024
26 of 28 checks passed

tbonelee deleted the update-pty4j branch September 19, 2024 06:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 #4822

[ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 #4822

tbonelee commented Sep 10, 2024 •

edited

Loading

tbonelee commented Sep 10, 2024

pan3793 commented Sep 10, 2024 •

edited

Loading

tbonelee commented Sep 10, 2024

pan3793 commented Sep 11, 2024 •

edited

Loading

tbonelee commented Sep 11, 2024

pan3793 commented Sep 11, 2024

pan3793 commented Sep 17, 2024

pan3793 commented Sep 19, 2024

[ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 #4822

[ZEPPELIN-6071] Bump pty4j version to 0.12.10-jdk8 #4822

Conversation

tbonelee commented Sep 10, 2024 • edited Loading

What is this PR for?

What type of PR is it?

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Before

After

Questions:

tbonelee commented Sep 10, 2024

pan3793 commented Sep 10, 2024 • edited Loading

tbonelee commented Sep 10, 2024

pan3793 commented Sep 11, 2024 • edited Loading

tbonelee commented Sep 11, 2024

pan3793 commented Sep 11, 2024

pan3793 commented Sep 17, 2024

pan3793 commented Sep 19, 2024

tbonelee commented Sep 10, 2024 •

edited

Loading

pan3793 commented Sep 10, 2024 •

edited

Loading

pan3793 commented Sep 11, 2024 •

edited

Loading