Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2.13.9] Can't reference sun. package classes with -release option #12643

Closed
xuwei-k opened this issue Sep 21, 2022 · 13 comments · Fixed by scala/scala#10543
Closed

[2.13.9] Can't reference sun. package classes with -release option #12643

xuwei-k opened this issue Sep 21, 2022 · 13 comments · Fixed by scala/scala#10543
Assignees
Milestone

Comments

@xuwei-k
Copy link

xuwei-k commented Sep 21, 2022

Reproduction steps

class A {
  classOf[sun.misc.Unsafe]
}

Problem

Java 8 without -release option

Success

$ java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Zulu 8.62.0.19-CA-macos-aarch64) (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Zulu 8.62.0.19-CA-macos-aarch64) (build 25.332-b09, mixed mode)
$ scala -version
Scala code runner version 2.13.9 -- Copyright 2002-2022, LAMP/EPFL and Lightbend, Inc.
$ scalac A.scala 
A.scala:2: warning: a pure expression does nothing in statement position; multiline expressions may require enclosing parentheses
  classOf[sun.misc.Unsafe]
         ^
1 warning

Java 8 with -release:8 option

Success

$ scalac -release:8 A.scala
A.scala:2: warning: a pure expression does nothing in statement position; multiline expressions may require enclosing parentheses
  classOf[sun.misc.Unsafe]
         ^
1 warning

Java 11 without -release option

Success

$ java -version
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment Zulu11.58+15-CA (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.58+15-CA (build 11.0.16+8-LTS, mixed mode)
$ scala -version
Scala code runner version 2.13.9 -- Copyright 2002-2022, LAMP/EPFL and Lightbend, Inc.
$ scalac A.scala           
A.scala:2: warning: a pure expression does nothing in statement position; multiline expressions may require enclosing parentheses
  classOf[sun.misc.Unsafe]
         ^
1 warning

Java 11 with -release:11 option

Success

$ java -version 
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment Zulu11.58+15-CA (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.58+15-CA (build 11.0.16+8-LTS, mixed mode)
$ scala -version
Scala code runner version 2.13.9 -- Copyright 2002-2022, LAMP/EPFL and Lightbend, Inc.
$ scalac -release:11 A.scala
A.scala:2: warning: a pure expression does nothing in statement position; multiline expressions may require enclosing parentheses
  classOf[sun.misc.Unsafe]
         ^
1 warning

Java 11 with -release:8 option

Failure !?

$ java -version            
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment Zulu11.58+15-CA (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.58+15-CA (build 11.0.16+8-LTS, mixed mode)
$ scala -version            
Scala code runner version 2.13.9 -- Copyright 2002-2022, LAMP/EPFL and Lightbend, Inc.
$ scalac -release:8 A.scala 
A.scala:2: error: not found: value sun
  classOf[sun.misc.Unsafe]
          ^
1 error

Java 11 with -target:jvm-1.8 option

Success

$ java -version                
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment Zulu11.58+15-CA (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.58+15-CA (build 11.0.16+8-LTS, mixed mode)
$ scala -version                
Scala code runner version 2.13.9 -- Copyright 2002-2022, LAMP/EPFL and Lightbend, Inc.
$ scalac -target:jvm-1.8 A.scala
A.scala:2: warning: a pure expression does nothing in statement position; multiline expressions may require enclosing parentheses
  classOf[sun.misc.Unsafe]
         ^
warning: 1 deprecation; re-run with -deprecation for details
2 warnings
@lrytz
Copy link
Member

lrytz commented Sep 21, 2022

It's the same with Scala 2.13 and Scala 3. When using Java 11 with -release:8, the ct.sym file that's put on the classpath (instead of rt.jar) contains Java 8 symbols, but only the official JDK. So sun is missing. (That's what I remember, maybe someone has a good reference here.)

@lrytz
Copy link
Member

lrytz commented Sep 21, 2022

It's the same with Scala 2.13

Sorry, that didn't make sense. But the behavior is already in 2.13.8

➜ sandbox sv 2.13.8 -release 8 -e "classOf[sun.misc.Unsafe]"
/var/folders/8d/8tnybb497mbgq2v08vgcr5c80000gn/T/scalacmd9807037385199816981.scala:1: error: not found: value sun
classOf[sun.misc.Unsafe]
        ^

@som-snytt
Copy link

Maybe the point is to undeprecate -target? Or is there a convenient -classpath idiom?

@SethTisue
Copy link
Member

Does javac -release behave the same way?

@SethTisue
Copy link
Member

Maybe the point is to undeprecate -target?

Huh?

@som-snytt
Copy link

-target says "emit class file of version N, but I want to use arbitrary classes from the JDK and take my chances".

If that is a reasonable use case, then either undeprecate the compiler option or make it easy to put certain packages of compelling utility on the class path when using -release.

@lrytz
Copy link
Member

lrytz commented Sep 21, 2022

Does javac -release behave the same way?

yes, with javac --release 8, sun.misc.Unsafe is not visible

@SethTisue
Copy link
Member

If javac is the same, then I think we should just close this ticket.

@SethTisue
Copy link
Member

make it easy to put certain packages of compelling utility on the class path when using -release

If anyone figures out what incantation is currently needed, please share!

(What do javac users do in this case? I did a bit of Googling for prior art but failed to find something, but maybe I didn't try hard enough.)

@He-Pin
Copy link
Contributor

He-Pin commented Oct 11, 2022

@SethTisue @som-snytt cc

https://github.com/akka/akka/blob/a6863e9ed5378d52cb16f386db4e137f0a02c170/project/JdkOptions.scala#L47-L58

 selectOptions( 
   targetSystemJdk, 
   jdk8home, 
   fullJavaHomes, 
   Seq(if (scalaVersion.startsWith("3.")) "-Xtarget:8" else "-target:jvm-1.8"), 
   // '-release 8' is not enough, for some reason we need the 8 rt.jar 
   // explicitly. To test whether this has the desired effect, compile 
   // akka-remote and check the invocation of 'ByteBuffer.clear()' in 
   // EnvelopeBuffer.class with 'javap -c': it should refer to 
   //""java/nio/ByteBuffer.clear:()Ljava/nio/Buffer" and not 
   // "java/nio/ByteBuffer.clear:()Ljava/nio/ByteBuffer". Issue #27079 
   (java8home: File) => Seq("-release", "8", "-javabootclasspath", java8home + "/jre/lib/rt.jar")) 

akka/akka#27079

@som-snytt
Copy link

The inliner needs actual classfiles and accommodates -release, maybe there could be a fallback option in the case of missing symbols (or a list of allowed symbols).

dongjoon-hyun pushed a commit to apache/spark that referenced this issue Sep 13, 2023
### What changes were proposed in this pull request?
This pr downgrade `scala-maven-plugin` to version 4.7.1 to avoid it automatically adding the `-release` option as a Scala compilation argument.

### Why are the changes needed?
The `scala-maven-plugin` versions 4.7.2 and later will try to automatically append the `-release` option as a Scala compilation argument when it is not specified by the user:

1. 4.7.2 and 4.8.0: try to add the `-release` option for Scala versions 2.13.9 and higher.
2. 4.8.1: try to append the `-release` option for Scala versions 2.12.x/2.13.x/3.1.1, and append `-java-output-version` for Scala 3.1.2.

The addition of the `-release` option has caused issues mentioned in SPARK-44376 | #41943 and #40442 (comment). This is because the `-release` option has stronger compilation restrictions than `-target`, ensuring not only bytecode format, but also that the API used in the code is compatible with the specified version of Java. However, many APIs in the `sun.*` package are not `exports` in Java 11, 17, and 21, such as `sun.nio.ch.DirectBuffer`, `sun.util.calendar.ZoneInfo`, and `sun.nio.cs.StreamDecoder`, making them invisible when compiling across different versions.

For discussions within the Scala community, see scala/bug#12643, scala/bug#12824, scala/bug#12866,  but this is not a bug.

I have also submitted an issue to the `scala-maven-plugin` community to discuss the possibility of adding additional settings to control the addition of the `-release` option: davidB/scala-maven-plugin#722.

For Apache Spark 4.0, in the short term, I suggest downgrading `scala-maven-plugin` to version 4.7.1 to avoid it automatic adding the `-release` option as a Scala compilation argument. In the long term, we should reduce use of APIs that are not `exports` for compatibility with the `-release` compilation option due to `-target` already deprecated after Scala 2.13.9.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions
- Manual check

run `git revert 656bf36` to revert to using Scala 2.13.11 and run `dev/change-scala-version.sh 2.13` to change Scala to 2.13

1. Run `build/mvn clean install -DskipTests -Pscala-2.13 -X` to check the Scala compilation arguments.

Before

```
[[DEBUG] [zinc] Running cached compiler 1992eaf4 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
  -unchecked
  -deprecation
  -feature
  -explaintypes
  -target:jvm-1.8
  -Wconf:cat=deprecation:wv,any:e
  -Wunused:imports
  -Wconf:cat=scaladoc:wv
  -Wconf:cat=lint-multiarg-infix:wv
  -Wconf:cat=other-nullary-override:wv
  -Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
  -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
  -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
  -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
  -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
  -Wconf:msg=method without a parameter list overrides a method with a single empty one:s
  -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
  -Wconf:cat=unchecked&msg=outer reference:s
  -Wconf:cat=unchecked&msg=eliminated by erasure:s
  -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
  -Wconf:msg=Implicit definition should have explicit type:s
  -release
  8
  -bootclasspath
...
```

After

```
[DEBUG] [zinc] Running cached compiler 72dd4888 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
  -unchecked
  -deprecation
  -feature
  -explaintypes
  -target:jvm-1.8
  -Wconf:cat=deprecation:wv,any:e
  -Wunused:imports
  -Wconf:cat=scaladoc:wv
  -Wconf:cat=lint-multiarg-infix:wv
  -Wconf:cat=other-nullary-override:wv
  -Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
  -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
  -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
  -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
  -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
  -Wconf:msg=method without a parameter list overrides a method with a single empty one:s
  -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
  -Wconf:cat=unchecked&msg=outer reference:s
  -Wconf:cat=unchecked&msg=eliminated by erasure:s
  -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
  -Wconf:msg=Implicit definition should have explicit type:s
  -target:8
  -bootclasspath
  ...
```

After downgrading the version, the `-release` option should no longer appear in the compilation arguments.

2. Maven can build the project with Java 17 without the issue described in #41943. And after this pr, we can re-upgrade Scala 2.13 to Scala 2.13.11.

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #42899 from LuciferYang/SPARK-45144.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
@SethTisue SethTisue added this to the 2.13.13 milestone Jan 24, 2024
@SethTisue SethTisue reopened this Jan 24, 2024
@som-snytt
Copy link

We're back in business, baby!

@He-Pin
Copy link
Contributor

He-Pin commented Jan 24, 2024

Rocks the Scala, 动起来,让我们一起摇摆。

szehon-ho pushed a commit to szehon-ho/spark that referenced this issue Aug 7, 2024
This pr downgrade `scala-maven-plugin` to version 4.7.1 to avoid it automatically adding the `-release` option as a Scala compilation argument.

The `scala-maven-plugin` versions 4.7.2 and later will try to automatically append the `-release` option as a Scala compilation argument when it is not specified by the user:

1. 4.7.2 and 4.8.0: try to add the `-release` option for Scala versions 2.13.9 and higher.
2. 4.8.1: try to append the `-release` option for Scala versions 2.12.x/2.13.x/3.1.1, and append `-java-output-version` for Scala 3.1.2.

The addition of the `-release` option has caused issues mentioned in SPARK-44376 | apache#41943 and apache#40442 (comment). This is because the `-release` option has stronger compilation restrictions than `-target`, ensuring not only bytecode format, but also that the API used in the code is compatible with the specified version of Java. However, many APIs in the `sun.*` package are not `exports` in Java 11, 17, and 21, such as `sun.nio.ch.DirectBuffer`, `sun.util.calendar.ZoneInfo`, and `sun.nio.cs.StreamDecoder`, making them invisible when compiling across different versions.

For discussions within the Scala community, see scala/bug#12643, scala/bug#12824, scala/bug#12866,  but this is not a bug.

I have also submitted an issue to the `scala-maven-plugin` community to discuss the possibility of adding additional settings to control the addition of the `-release` option: davidB/scala-maven-plugin#722.

For Apache Spark 4.0, in the short term, I suggest downgrading `scala-maven-plugin` to version 4.7.1 to avoid it automatic adding the `-release` option as a Scala compilation argument. In the long term, we should reduce use of APIs that are not `exports` for compatibility with the `-release` compilation option due to `-target` already deprecated after Scala 2.13.9.

No

- Pass GitHub Actions
- Manual check

run `git revert 656bf36` to revert to using Scala 2.13.11 and run `dev/change-scala-version.sh 2.13` to change Scala to 2.13

1. Run `build/mvn clean install -DskipTests -Pscala-2.13 -X` to check the Scala compilation arguments.

Before

```
[[DEBUG] [zinc] Running cached compiler 1992eaf4 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
  -unchecked
  -deprecation
  -feature
  -explaintypes
  -target:jvm-1.8
  -Wconf:cat=deprecation:wv,any:e
  -Wunused:imports
  -Wconf:cat=scaladoc:wv
  -Wconf:cat=lint-multiarg-infix:wv
  -Wconf:cat=other-nullary-override:wv
  -Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
  -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
  -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
  -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
  -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
  -Wconf:msg=method without a parameter list overrides a method with a single empty one:s
  -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
  -Wconf:cat=unchecked&msg=outer reference:s
  -Wconf:cat=unchecked&msg=eliminated by erasure:s
  -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
  -Wconf:msg=Implicit definition should have explicit type:s
  -release
  8
  -bootclasspath
...
```

After

```
[DEBUG] [zinc] Running cached compiler 72dd4888 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
  -unchecked
  -deprecation
  -feature
  -explaintypes
  -target:jvm-1.8
  -Wconf:cat=deprecation:wv,any:e
  -Wunused:imports
  -Wconf:cat=scaladoc:wv
  -Wconf:cat=lint-multiarg-infix:wv
  -Wconf:cat=other-nullary-override:wv
  -Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
  -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
  -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
  -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
  -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
  -Wconf:msg=method without a parameter list overrides a method with a single empty one:s
  -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
  -Wconf:cat=unchecked&msg=outer reference:s
  -Wconf:cat=unchecked&msg=eliminated by erasure:s
  -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
  -Wconf:msg=Implicit definition should have explicit type:s
  -target:8
  -bootclasspath
  ...
```

After downgrading the version, the `-release` option should no longer appear in the compilation arguments.

2. Maven can build the project with Java 17 without the issue described in apache#41943. And after this pr, we can re-upgrade Scala 2.13 to Scala 2.13.11.

No

Closes apache#42899 from LuciferYang/SPARK-45144.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
mamccorm pushed a commit to wolfi-dev/os that referenced this issue Jan 17, 2025
)

**Context and implications of the changes:**

The inability to update `org.scala-lang:scala-library` to version
`2.13.9` or higher in `spark-3.5.x` arises from a critical build issue
documented in
[SPARK-44376](https://issues.apache.org/jira/browse/SPARK-44376). This
problem stems from the transition to Scala `2.13.11`
([SPARK-40497](https://issues.apache.org/jira/browse/SPARK-40497)) and
the deprecation of the `-target` argument in favor of `-release` in the
[scala-maven-plugin](davidB/scala-maven-plugin#722).
This change introduces stricter compatibility checks, breaking builds
when using Java 11 or later.

The key errors include inaccessible `sun.*` classes like `DirectBuffer`
and `Unsafe`, which are not exported in Java versions above 8
([source](scala/bug#12643)). Despite manual
attempts to adjust Maven configurations, such as switching
`-target:jvm-1.8` to `-release:8`, compilation failures persist due to
inherent restrictions imposed by the `-release` argument. This issue is
compounded by the Scala Maven plugin's automatic addition of the
`-release` argument for Scala `2.13.9` and above, leading to
incompatibilities when targeting Java 8 compatibility while running on
newer Java versions.

While upstream remediated by upgrading to `2.13.11`, even `2.13.9` is
not possible with `scala-maven-plugin` version `4.8.0`. Currently,
`scala.version` is hardcoded to `2.13.8`, and fixing this requires
implementation of the following PRs:
1. https://github.com/apache/spark/pull/41626/files
2. https://github.com/apache/spark/pull/42899/files

**The only reason remediation is achievable is due to the following
conditions:**
1. It is already merged upstream and awaiting a `4.0.0` release.
2. The commits are not intertwined with other more complicated
initiatives or functional changes.
3. Support for Java 8 runtime dependency is not dropped with this
implementation.
4. This is a critical CVE.
5. Thorough package and image-level testing is in place.
6. Downgrading `scala-maven-plugin` to `4.7.1` does not introduce any
new CVEs beyond those already existing in `4.8.0`.

**This was a more involved remediation due to:**
1. The dynamic setting of this version property via
[change-scala-version.sh](https://github.com/apache/spark/blob/3b0ac45391708642ce6a1779e3c234bab0e40b66/dev/change-scala-version.sh#L72C1-L81C21).
2. The malformed `scala.version` property, which does not follow usual
`pom.xml` conventions, with the same property value defined twice in the
`pom.xml`, causing maven/pombump to not function.
3. The incredibly careful and diligent investigation required to gain
confidence in the fix, alongside the experience required from working on
previous spark-3.5 issues.

Bonus: Was able to tack on an additional remediation for
ivy/GHSA-2jc4-r94c-rp7h

---------

Signed-off-by: jamie-albert <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants