Dependence on sun.misc.Unsafe getting troublesome now. #273
Comments
|
Sorry I've been swamped and falling really far behind. I usually work on this project during my weekends, but I haven't had any time or been too exhausted. So lots of things need to be taken care of. It's odd that they require that now, given that Unsafe still works fine. Maybe it to prepare one LTS before the 2021 LTS that will likely remove Unsafe for good? For v3, I would like to rewrite the JCache implementation to use the native per-entry expiration support, since that wasn't implemented when the adapter was written. Instead it has to use an uglier approach, but also used by the spec leads, of relying on maximum size with dead entries. Since this would change the configuration format, I want to make all of the breaking changes together. It might be better to update the config format and punt on the rewrite work, though, given my struggles to keep up with things. For VarHandles, my only concern is how to port |
|
Like other cache/map implementations using |
|
I'm not so sure that VarHandles has better performance than Unsafe. Static VarHandles maybe but that's not really suitable for the use case. Just to note that in Java 10 there's also jdk.internal.misc.Unsafe which has *unaligned access methods which may enable even more performance (but it's hidden and restricted with required --add-exports / --add-opens). sun.misc.Unsafe is a wrapper around the internal version that lacks a few methods. |
|
Have we considered using multi-version jar? (implemented in Java 9) |
|
Just hit this issue moving a project to Java 11. |
|
This shouldn't impact moving to Java 11, as I'm using that without any special flags. What issue did you run into? There doesn't seem to be a rush by the JDK team to remove Unsafe and JCTools still prefers it for performance reasons. Since it requires a major release and hasn't seemed pressing, I've been working small scoped improvements instead that are more amenable to weekend / commute time. |
|
I've been moving to use the module system and must admit my ignorance. I thought Unsafe was gone in Java 11. Took me a while to find the VM argument --add-modules=jdk.unsupported to get access to Unsafe again. Adding here for other people who hit this issue. :) Thanks for indirectly sending me in the right direction. |
|
This was brought to me recently by one of our customers, now that Solr uses Caffeine - their security team flagged this as a concern. In general, I think commercial users are wary (usually for good reasons) of adding any flags with "unsupported" in their names... |
|
I believe you only need |
Well, I'm using jlink with I'm not sure how common this is or how good of an idea it is. I only started doing this recently with one application. (Since my application and dependencies are not modular, coming up with the list of modules for jlink was a process of trial and error. I would start the application and look for |
|
Oh thanks. I didn't realize jlink also works with the classpath and had assumed it required opting in all of the way. Moving to a custom runtime jvm does simplify deployments for newer projects, so I'm sure it will become more popular. It would probably make more sense to gather a list of changes for 3.0, implement any API facing, and bump the base JDK requirements. Then removing Unsafe or other enhancements can be done incrementally rather than needing a big bang release. |
|
13 looks a bit more blocked - byte buddy 1.10 |
|
Is that related though? The stacktrace doesn't include this library and we do not dynamically generate classes ( |
|
I have the initial start on this in the v3 branch. The main difficulty is Java reuses the ThreadLocalRandom probe value stashed in Thread as The other major case to review our usage of JCTools which hasn't migrated due to Unsafe being much faster in current JDKs. Similarly we'll probably use Unsafe and a fallback to VarHandles if unavailable. No timeline for v3 but those are the caveats with Unsafe. |
|
Sorry for being slow here. In my first reply, I mentioned a possibly backwards incompatible change to the JCache module. That isn't used much, but I had wanted to improve its implementation and that seemed to require a lot of effort to redo. I took advantage of the holidays being extra quiet this year, got far with a rewrite before discovering a blocker, and took that learning to make a surgical change that resolved my concerns. So that problem is done and has no incompatibilities to worry about. The other issues were resolved in that v3 branch. That work needs to be resurrected, but hopefully I can start making progress again. I believe all remaining tasks are small and doable as I find the time. JCTools is likely the only blocker and our use is limited, so hopefully converting code that isn't too difficult (they rightfully prefer Unsafe for performance reasons, but our case is less sensitive so its okay). |
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
By default Unsafe will be used if available, due to the performance benefits as discussed in [1]. If not available then access will fallback to VarHandles. The queue sanity tests run against both variants. [1] JCTools/JCTools#157
|
A 3.0-SNAPSHOT is now available if anyone wants to test disallowing This branch is ready for release, so I plan on letting it bake for a week in case anything else comes up before making an official cut. |
|
Released in 3.0 |
@sigram You might want to upgrade Solr to v3 to resolve your concerns. I notice that project is JDK11 on master, so this should be fine. There are not incompatible changes that should affect you. The new release will use Unsafe if available, but it is not required. If the flag is not included it will work fine via VarHandles, which has similar performance. I am not sure how Solr manages version upgrades in the new Gradle build. You might consider setting up something like the gradle-update-checker Github Action, which uses my gradle-versions-plugin. Obviously you don't want to upgrade too rapidly but you might want to track that updates are available for bug fixes and such things. |
|
Further improvements allowed for removal of |
|
Released 3.0.2 with removal of |
|
@ben-manes any chances of getting this backported to 2.x? I tried to use 3.x in this PR, but as the project needs to target Java 8, there is lots of test failures. I'm happy to help if that's something you're interested in and you give me some direction. |
|
Sorry, but this won't be backported. We use Unsafe for slimmer entries (no Atomic wrappers), which is now possible using VarHandles. The only alternative is to use AtomicReferenceFieldUpdater, but this uses reflection making it generally disfavored. The code generator and various other direct usages would have to migrate to either approach and I don't think it would be preferable for slow or memory bloated atomic operations. The other usages of Unsafe were replaced by algorithmic changes. It seems wiser to upgrade JDKs, e.g. Java 8 is EOL in six months (March 2022). |
|
Fair enough, thanks for the reply! |
In #122 you said when Java 9 was out you'd probably do a version 3.0 using VarHandles instead of depending on the Unsafe stuff. Now Java11 is out and I'm having to jlink my server jre with jdk.unsupported just so caffeine can work. (tbh I'm not entirely sure why I didn't have to do this in Java10.) Any sign of that 3.0 version using VarHandles? :-)
The text was updated successfully, but these errors were encountered: