Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rhino 1.7.14 not usable on Android any more due to javax.lang.model.SourceVersion #1149

Closed
b3nn0 opened this issue Jan 7, 2022 · 23 comments · Fixed by #1385
Closed

Rhino 1.7.14 not usable on Android any more due to javax.lang.model.SourceVersion #1149

b3nn0 opened this issue Jan 7, 2022 · 23 comments · Fixed by #1385
Labels
Android Issues related to running Rhino on Android Regression

Comments

@b3nn0
Copy link

b3nn0 commented Jan 7, 2022

Hi,
while previous versions worked just fine on Android in interpreted mode, Rhino 1.7.14 does not work any more due to this commit: 0fcfb09
because javax.lang is not available on Android: import javax.lang.model.SourceVersion;

As a workaround I created my own class in javax.lang.model:

public class SourceVersion {
    public static SourceVersion latestSupported() {
        return new SourceVersion();
    }
    public final int ordinal() {
        return 8;
    }
}

It seems to work fine, but also feels very hacky and I'm not sure if it's even correct.
Is there any hope that this is fixed upstream despite Android not being supported?

@b3nn0 b3nn0 changed the title Rhino 1.7.14 not usable on Android any more Rhino 1.7.14 not usable on Android any more due to javax.lang.model.SourceVersion Jan 7, 2022
@gbrail
Copy link
Collaborator

gbrail commented Jan 7, 2022 via email

@msridhar
Copy link
Contributor

msridhar commented Jan 7, 2022

FWIW, Error Prone does have a static checker for these types of issues:

http://errorprone.info/bugpattern/AndroidJdkLibsChecker

@gbrail
Copy link
Collaborator

gbrail commented Jan 7, 2022 via email

@gbrail
Copy link
Collaborator

gbrail commented Jan 7, 2022

Interestingly, ErrorProne (which is hard to make work on Java 8, BTW, which is another reason I'd love to upgrade) found over 200 things to fix, so we'd have our work cut out for us to clean up the code and also suppress spurious warnings.

The "AndroidJdkLibsChecker" found problems with:

BigInt support (new in 1.7.14)
the java.util.function package in general, which is used all over (and why isn't this in Android?)
StampedLock, which we only use if you are enabling a rarely-used synchronized object feature

It did not, however, complain about using javax.lang.SourceVersion.

Is there some sort of an official list of what is supported and not supported in various Android versions? Thanks!

@msridhar
Copy link
Contributor

msridhar commented Jan 7, 2022

Not super well documented, but it looks like there is a flag -XepOpt:Android:Java8Libs you can pass to Error Prone to allow certain Java 8 functionalities like java.util.function:

https://github.com/google/error-prone/releases/tag/v2.3.2

Those allowed functionalities should be desugared or otherwise handled by modern Android build tools for software targeting older Android versions. I think it's reasonable for Rhino to use those functionalities.

I'm not sure about why javax.lang.model.SourceVersion was missed. I'll look around a bit.

@msridhar
Copy link
Contributor

msridhar commented Jan 7, 2022

I opened google/error-prone#2817 for the issue with not finding the use of SourceVersion

@gbrail
Copy link
Collaborator

gbrail commented Jan 7, 2022

Back to "how to fix it" -- does this show up as a ClassNotFoundException, or something else?

I wonder if we were to surround that call with a try...catch for CNFE, that we could make this same code work on Android- and non-Android platforms.

@b3nn0
Copy link
Author

b3nn0 commented Jan 8, 2022

Stack trace:

java.lang.NoClassDefFoundError: Failed resolution of: Ljavax/lang/model/SourceVersion;
    at org.mozilla.javascript.JavaMembers.<clinit>(JavaMembers.java:37)
    at org.mozilla.javascript.JavaMembers.lookupClass(JavaMembers.java:769)
    at org.mozilla.javascript.NativeJavaClass.initMembers(NativeJavaClass.java:50)
    at org.mozilla.javascript.NativeJavaObject.<init>(NativeJavaObject.java:53)
    at org.mozilla.javascript.NativeJavaClass.<init>(NativeJavaClass.java:44)
    at org.mozilla.javascript.NativeJavaClass.<init>(NativeJavaClass.java:40)

So I guess, try-catch might be an option.

@p-bakker p-bakker added the Android Issues related to running Rhino on Android label Jan 12, 2022
@gbrail
Copy link
Collaborator

gbrail commented Jan 13, 2022

Thanks for all the attention!

Is there anyone in the community who works with Android and Rhino regularly who can create a reproducer for this? It could be a GitHub repo or a Gist that includes code that reproduces it, or an existing app that we can build and test, or just a detailed set of instructions.

Without that, we're waiting for someone in the Rhino community, or me, to install Android Studio, learn to build an app, learn how to get Rhino into that app, test it, and fix the problem. No one here actually is paid to work on Rhino full time or even close to it so this second option will probably take a while.

@b3nn0
Copy link
Author

b3nn0 commented Jan 14, 2022

I have now tried to prepare something like this: an easy way to run the unit tests against the Android SDK (on desktop, so no Android device or emulator is needed).
It will (hopefully) automatically download the respective SDK/Build tools via gradle.

Since I'm not very familiar with the Rhino code, I simply excluded a few tests in build.gradle to get it working initially. Some stuff is correct to be excluded (like debugger GUI), some stuff I don't know about...

All changes related to the android tests are in a separate directory "android" - only build.gradle and AndroidManifest.xml is needed, really:
b3nn0@27e7276

However, there were some compilation problems. One due to the already mentioned SourceVersion, but also two more:
b3nn0@ba24d7f
However, I don't know what these are or how to change them correctly. I just wanted everything to compile properly for now.

So with my fork, you should be able to just run
cd android && ../gradlew test
and everything should work. With these tests, right now 98 tests are failing all over the place, while 3047 tests are passing.
Not sure how much of that is to be expected with my obviously breaking changes, since I'm not familiar with the codebase.

EDIT:
This will automatically accept the android SDK licenses! Only execute if you agree to them.

EDIT2:
One problem seems to be the working dir if done like this.. In the beginning, I had the build.gradle/AndroidManifest in the project root, resulting in 98 test fails. When I have it in the "android" subdirectory though, test seem to fail when trying to load script files, because the working dir is now inside the android subdirectory and I wasn't able to find a way to change that in gradle, so that the working directory is ".." when running tests.. :-/

EDIT3:
Alright, figured it out finally. It was related to the sytemProperties set by the original build.gradle. Now it can run correctly from the "android" subdirectory, with almost all tests succeeding. The only failing ones are these:

org.mozilla.javascript.tests.Test262SuiteTest > initializationError FAILED
    java.lang.RuntimeException at Test262SuiteTest.java:609

org.mozilla.javascript.tests.StringifyJavaObjectsTest > rhinoTestNoOpt FAILED
    java.lang.AssertionError at Assert.java:89

org.mozilla.javascript.tests.StringifyJavaObjectsTest > rhinoTestOpt0 FAILED
    java.lang.AssertionError at Assert.java:89

org.mozilla.javascript.tests.ShellTimerTest > testNestedTimers[0, opt=-1] FAILED
    org.mozilla.javascript.JavaScriptException at ShellTimerTest.java:136

org.mozilla.javascript.tests.StringifyJavaObjectsTest > rhinoTestOpt9 FAILED
    java.lang.AssertionError at Assert.java:89

which looks like it's related to the changes I had to do to make it compile.

@gbrail How to proceed with this? Should I create a cleaned up pull request to add this self-contained "android" subdirectory?
The only things that are left then are the actual issues with android compatibility (SourceVersion, java.beans and XMLConstants). However, I have no idea how to fix them properly...
See here for full changes I had to do -- breaking stuff obviously: master...b3nn0:master

@gbrail
Copy link
Collaborator

gbrail commented Jan 19, 2022

Thanks a ton for pulling this together -- I'm currently wondering whether it'd be easier to incorporate this into the regular repository, or have a separate repo that just does what you've done here to build Rhino with Android tools, and then run that as part of the CI.

@ghost
Copy link

ghost commented Jan 19, 2022

I saw the commit log of src/org/mozilla/javascript/JavaMembers.java, I think the commit that occurred was committed to solve this issue at #462, if so, I think it is very foolish to simply comment or fix the STRICT_REFLECTIVE_ACCESS value as true.

Why don't we leave the SourceVersion as it is and make a separate function to return true if it is Android, and use SourceVersion if it is not?

@ghost
Copy link

ghost commented Jan 21, 2022

First, in order to solve the javax.lang.model.SourceVersion related issue that occurs in the org.mozilla.javascript.JavaMembers class, you must first check whether the device running rhino is Android.
I made a class to check if the device is android (you can also get the SDK_INT value).
The source code can be found here

@gbrail
Copy link
Collaborator

gbrail commented Jan 26, 2022

Just an udpate -- I haven't had a chance to set up an Android environment to test this, and I'm not sure when I will.

If someone can produce a PR that works in the current environment, and can assure us that it also works in Android, then that's a start, and we can quickly move forward. Otherwise we're waiting for someone to have time to learn how to test Android stuff.

When we get a fix, I'm willing to create a 1.7.14.1 release that includes this fix. But in the meantime there are other things pending so I plan to merge them in to "master" while we work.

@p-bakker
Copy link
Collaborator

p-bakker commented Jan 31, 2022

How to proceed with this? Should I create a cleaned up pull request to add this self-contained "android" subdirectory?

@b3nn0 yes, a cleaned-up PR with all the test infra structure would be the way forward I think.

Personally I don't think we should wire up things to run the android tests automatically when running gradle test, as (correct me if I'm wrong) that would entail for downloading a whole bunch of Android tooling for everyone developing Rhino, no?

We should have the option to trigger running the Android tests locally and, once all the kinks are ironed out, add a job to our CI(s) to run the tests against Android on every commit.

One question though (not being well-versed in Android development): there are many Andoid versions/API levels: how do those correlate to the test infrastructure you set up? Would that test against all versions/levels? Or just the last one? Or...? Looking at your branch, it looks like it is only testing against API version 31? Also see #1156 which refers to api level 21.

Once the testing infra structure is available, work on fixing the issues at hand can commence.

@archethic I agree we should have a utility function to test whether running on Android or not. Currently in 2 places in the codebase we have "Dalvik".equals(System.getProperty("java.vm.name"), which should be refactored to use such function. The best location for such function would be on the ScriptRuntime class I think. Care to create a PR just for that to get started? Not sure if we need the getCurrentAPILevel() method as well: #1156 however reports an issue on api level 21 which may or may not exist with the latest API level

I think it is very foolish to simply comment or fix the STRICT_REFLECTIVE_ACCESS value as true

@archethic I think @b3nn0 just did that to make the tests pass in his branch (just like some of the other things he commented out)

If I'm not mistaken, the check on STRICT_REFLECTIVE_ACCESS has to do with modular Java (correct @carlosame?). If so, how relevant is that when running on Android?

@b3nn0 The stuff you commented out in XmlProcessor.java: isnt the catch on IllegalArgumentException sufficient? What kind of exception is thrown then?

As for the stuff you commented out in JavaToJSONConverters: Looks like Android doesn't support the full java.beans package api, just a small subset. There is a 3rd party library it seems to fills in the missing bits: https://github.com/melix/openbeans. Not exactly sure how to go about fixing this, whether we could adjust out implementation to not rely on that specific api from the java.beans package or if we could somehow instead use https://github.com/melix/openbeans opn Android is its available on the classpath. Hope others have idea's on that

@b3nn0
Copy link
Author

b3nn0 commented Jan 31, 2022

OK, I will prepare a pull request in the next few days. I agree that it makes sense to have the android build separately, since yes, it will download the Android SDK and install it in a temporary subdirectory.

wrt. SDK Version: While the "more advanced"/"highlevel" Android features are sometimes not 100% compatible in newer SDK version, the "basic java library" that is used by Rhino is usually only extended, not reduced. So IMO it should be enough to pick a "minimum supported version" and test against that. Let's say something around Android 5.x (API Level 21) should be old enough (?).

Regarding exceptions for non-available library features (e.g. XmlProcessor.java):
That's a bit of a tricky one and I'm not sure how to handle it in a good way.
If you simply use a precompiled rhino jar, everything will work fine (except for the SourceVersion error), because many people will not use the affected classes (XmlProcessor). And if they do, the respective exception is thrown, hopefully handled, and doesn't cause too many issues down the line.
However, what my PR does is compile rhino itself against the Android SDK. That means that these errors occur during compile time, not runtime. Catching exception won't do the job then, since it won't compile in the first place.
I'm not sure if that's an upside or downside of the approach I chose. It certainly makes things a bit harder, because we also need to be able to compile against the Android SDK, not just run against it.
A different approach might be compiling rhino against the local JVM, then use the resulting jar and only compile tests against the android SDK with the help of the previously compiled rhino.jar. That way we would only catch runtime errors on Android. The downside is that I'm not sure if I'm familiar enough with gradle to accomplish this.

@p-bakker
Copy link
Collaborator

wrt. SDK Version: While the "more advanced"/"highlevel" Android features are sometimes not 100% compatible in newer SDK version, the "basic java library" that is used by Rhino is usually only extended, not reduced. So IMO it should be enough to pick a "minimum supported version" and test against that. Let's say something around Android 5.x (API Level 21) should be old enough (?).

k, sounds good to me

Regarding your choice of compiling against the Android SDK: how would Android developers use Rhino? I would think they take the Rhino jar file that we provide here and use that, not (try to) compile the Rhino source against the Android SDK, correct? If so, I think the approach to testing Rhino against Android should be the same: take the Rhino jar as provided by us and they, using the Android SDK run the tests against that jar.

Given asuch approach, I wonder if we couldn't 'just' add some android task to our existing build.gradle file that we can trigger conditionally and has dependsOn compileJava to get the compiled rhino.jar. Or maybe, based on https://www.titanwolf.org/Network/q/c552b9a9-9557-43e2-972b-1eb014075ef6/y, sort of externalize android.gradle into its own thing.

I'm not well-versed in gradle either, so no idea how to for example dynamically load the andoid plugin only when executing the task to run the testsuite against andoid

@carlosame
Copy link
Contributor

carlosame commented Jan 31, 2022

If I'm not mistaken, the check on STRICT_REFLECTIVE_ACCESS has to do with modular Java (correct @carlosame?). If so, how relevant is that when running on Android?

To fix this issue @p-bakker you could try using a different way to determine whether one is on a modular JDK, like attempting a reflective access to a method that only a modular JDK would have. Something like the following (or any variation of it):

@@ -21,25 +21,33 @@ import java.security.Permission;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
-import javax.lang.model.SourceVersion;
 
 /**
  * @author Mike Shaver
  * @author Norris Boyd
  * @see NativeJavaObject
  * @see NativeJavaClass
  */
 class JavaMembers {
 
-    private static final boolean STRICT_REFLECTIVE_ACCESS =
-            SourceVersion.latestSupported().ordinal() > 8;
+    private static final boolean STRICT_REFLECTIVE_ACCESS;
 
     private static final Permission allPermission = new AllPermission();
 
+    static {
+        boolean getModule;
+        try {
+            getModule = Class.class.getMethod("getModule") != null;
+        } catch (NoSuchMethodException e) {
+            getModule = false;
+        }
+        STRICT_REFLECTIVE_ACCESS = getModule;
+    }
+
     JavaMembers(Scriptable scope, Class<?> cl) {
         this(scope, cl, false);
     }
 
     JavaMembers(Scriptable scope, Class<?> cl, boolean includeProtected) {

I haven't actually tested it but should work. It is less computationally efficient than the SourceVersion code but still good.

P.S.:
I see that now the OSGi bundle declares an Import-Package of javax.lang.model which should be addressed when fixing this issue, given that the package would no longer be used.

"Import-Package": "javax.lang.model,javax.script"

In fact you should remove the entire Import-Package as it is declaring a dependency on javax.script for the main rhino and rhino-runtime, but neither of those bundles use that package (rhino-engine would be the right one).

Moreover, I'm not sure why Rhino needs to explicitly declare an Import-Package: the default value is * which means "import automatically whatever is needed". Explicitly declaring one adds a maintenance burden.

@p-bakker
Copy link
Collaborator

tnx for the input @carlosame

@carlosame
Copy link
Contributor

tnx for the input @carlosame

Unfortunately the code snippet in my previous post would be flaky. Turns out that the static block initializing STRICT_REFLECTIVE_ACCESS could be executed after other static methods, with the weird result that a static final field can first evaluate to false and then to true.

I got it to work in my fe-protected-access branch (rebased) with the help of a static method:

https://github.com/carlosame/rhino/blob/0db93dc7ce522c5b6660f4bd0d849a6a5c3c9312/src/org/mozilla/javascript/VMBridge.java#L16

@ouachman
Copy link

Hello. Any news on this? I downgraded to the previous version for now. Thanks.

@p-bakker
Copy link
Collaborator

p-bakker commented Nov 2, 2023

Should be resolved by #1385

@p-bakker p-bakker closed this as completed Nov 2, 2023
@hyy920109
Copy link

Thanks for catching this! I think that these kinds of things keep happening because the few people who are regularly contributing to Rhino aren't Android developers and don't know how to tell whether Rhino is compatible with certain versions of Android (and, in fact, what versions of Android Rhino should target). I'm willing to spend some time fixing this, but do you (or anyone else) have the ability to help come up with some sort of test plan that would allow us to regularly test for Android compatibility so that future changes can be discovered before it's too late?

On Fri, Jan 7, 2022 at 7:16 AM Adrian Batzill @.> wrote: Hi, while previous versions worked just fine on Android in interpreted mode, Rhino 1.7.14 does not work any more due to this commit: 0fcfb09 <0fcfb09> because javax.lang is not available on Android: import javax.lang.model.SourceVersion; As a workaround I created my own class in javax.lang.model: public class SourceVersion { public static SourceVersion latestSupported() { return new SourceVersion(); } public final int ordinal() { return 8; } } It seems to work fine, but also feels very hacky and I'm not sure if it's even correct. Is there any hope that this is fixed upstream despite Android not being supported? — Reply to this email directly, view it on GitHub <#1149>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD7I2Y7CDHBZ5K7IN3AKODUU37VLANCNFSM5LPAASLA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.>

I‘m android developer too, I think many people work on android need this lib. too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Android Issues related to running Rhino on Android Regression
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants