-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot use experimental language features in @experimental code #13392
Comments
It is expected to have a compilation error on |
Yes, but I think in terms of language features that's wrong. We should check on use, not import. |
Can we wait with enforcing @experimental until 3.2? I don't think it's baked yet. The problem shown by |
The current version is conservative, if we relax it we will need to add custom logic for each experimental feature on a case-by-case basis.
|
This is too dangerous in general. For |
Oh no, but then we would not be able to bootstrap on a stable compiler. The simples change seems to be to special case |
Yes, I think at least so far |
But in fact that will probably not work either. Since the parser does not know about experimental contexts. |
Why is it dangerous? That's what I am not getting. |
The idea would be to not fail on the import of |
I just had another idea. We could delay the experimental warning on experimental imports to typer. Then, allow an experimental import if one of the following is true:
|
That might be a bit more complex. My version only requires the delay of the |
Let me try it anyway. I prefer not to need special treatments and exceptions. It's very awkward documenting them, to start with. |
Overview of the current solutionsThere are two PRs containing different parts of the solution (#13394 and #13396). The solutions in those PRs are half-complementary half-overlapping. There are 3 new rules that can be extracted from those PRs
Rule (1)
This rule is not controversial. It allows experimental imports in any experimental scope such as in a nightly build of the compiler or within an experimental definition. ExampleAssuming that we are on a stable release of the compiler or added the import annotation.experimental
@experimental
object Foo:
import language.experimental.erasedDefinitions // ok, Foo is @experimental
def foo = ...
erased def bar = ...
object Bar:
import language.experimental.erasedDefinitions // error
def foo = ...
erased def bar = ... This rule alone does not allow the use of This rule can be complemented by rules (2) or (3). Rule (2)
This rule aims to allow experimental erased top-level definitions in a file. ExampleAssuming that we are on a stable release of the compiler or added the import language.experimental.erasedDefinitions // ok, because we only have experimental definitions
import annotation.experimental
@experimental erased class Foo import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental class Bar.
import annotation.experimental
@experimental erased class Foo
class Bar import annotation.experimental
class Foo:
import language.experimental.erasedDefinitions // ok, because all statements are experimental
@experimental erased def foo = 1 import annotation.experimental
class Foo:
import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental method bar.
@experimental erased def foo = 1
def bar = 1 This allows any experimental features to be used a top-level in a source file if all definitions are experimental. It also allows the use of experimental in several other contexts (hard to describe which). Currently, it suffers from UX problems that may confuse the user or make them think there is a bug where there is non. There might also be some bugs in the implementation that make this judgment harder. Here are a few examples that will cause confusion or any users: Problematic casesAssuming that we are on a stable release of the compiler or added the Adding a new object will cause a compilation error - import language.experimental.erasedDefinitions
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental class Bar.
import annotation.experimental
@experimental erased class Foo
+ object Bar but the following does work import language.experimental.erasedDefinitions
import annotation.experimental
@experimental erased class Foo
+ object Foo Adding some stable member breaks code import annotation.experimental
class Foo:
- import language.experimental.erasedDefinitions
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental method bar.
@experimental erased def foo = 1
+ def bar = 1 Strangely the following also breaks even though there is already had a non experimental constructor. import annotation.experimental
class Foo:
- import language.experimental.erasedDefinitions
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental constructor Foo.
@experimental erased def foo = 1
+ def this(i: Int) = this() And adding code inside the constructor that was already allowed breaks the code import annotation.experimental
class Foo:
- import language.experimental.erasedDefinitions
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental expression println(1).
@experimental erased def foo = 1
+ println(1) It behaves strangely in term scopes such as iin a method bodies import annotation.experimental
def foo: Unit = {
- import language.experimental.erasedDefinitions
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental expression println().
@experimental erased def foo = 1
println()
+ println()
} Why does it fail or why didn't it? Interaction with @experimental definition and @experimental erased experimental differs for no good reason. import annotation.experimental
class Foo:
// import language.experimental.erasedDefinitions
def foo = 1
+ @experimental def bar = 1 // ok
class Bar:
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental method foo.
def foo = 1
+ @experimental erased def bar = 1 // not ok It is impossible to define erased sealed classes. import language.experimental.erasedDefinitions // error: ....
import annotation.experimental
sealed trait Foo
@experimental erased class Foo extends Foo I've hit more strange interactions and limitations, but I do not have the time to list them all. Therefore I stop with this small subset of issues that users will encounter and report. It is unclear if it is really useful to be able to import a language feature inside a definition but not make that definition experimental (i.e. use rule 1). If we would not support this use case we would only need a rule that allow us to import experimental definition at the top level of a file. This is equivalent to forcing all top level definitions of a file to be experimental. We could instead create a simpler targeted mechanism to achieve this purpose. Rule (3)
This rule special cases the imports of ExampleAssuming that we are on a stable release of the compiler or added the import language.experimental.erasedDefinitions // ok
import annotation.experimental
object Foo:
def foo = ... // ok
@experimental erased def bar = ... // ok
erased def baz = ... // error: `erased` definitions can only be used in an experimental scope. Consider making `baz` experimental.
@experimental erased class Bar // ok
@experimental erased def baz = ... // ok Like rule (3), it also supports top-level erased definitions. Unlike rule (3), this allows stable definitions to be defined along with @experimental erased definitions. When we stabilize the ConclusionTo fix #13392 we need (2) or (3). (1) can be added to either to enhance the user experience. All rules complement each other and overlap in some use-cases. If we use rule (3) we will be able to remove it once If we use rule (2) we will have to support it from here on even if it is broken as changes to it might affect non-experimental parts of sources. I would favor (3) for |
(3) means that we cannot use any of the other experimental language imports in a stable compiler. For instance, I cannot write import language.experimental.fewerBraces
@experimental
class Test(xs: List[Int]:
xs.map x =>
x + 1 and try that out in a standard Scala 3 distribution. I don't find that acceptable. This leaves (2), or else we turn experimental checking off until 3.2. |
We need (1) to allow experimental language features to be used in stable compiler. Extra use case can be added later. @experimental
class Test(xs: List[Int]:
import language.experimental.fewerBraces
xs.map x =>
x + 1 This works for any of the experimental language features we have except for erasedDefinitions. In it's current state (2) looks like a special case for us to be able to compile some code while it make it hard for users to use the feature. The current version has a rule that is already complex and also leaves some underspecified cases (or has many bugs). For the sake of the users we should not have this one and find a good way to achieve the same. Rule happens to work for the experimental language features we have now. There is the possibility that at some point we will want to add a new experimental feature that is incompatible with this rule. This would be a blocker for that feature. On the other hand (3) was designed to be easily usable by anyone that wants to try the erasedDefinitions feature. It is not an exception for us but it also allows us to compile the use case we care about. It also us keeps the experimental scopes rules simple. |
I disagree and I think you have to be more specific to make that allegation. Also, any underspecification simply does not matter! We are not talking a type soundness theorem here but a pragmatic feature that should prevent the common misuses and allow the comon use cases. That is all. The solution to put I don't think we will make progress discussing this further. Let's get both fixes in and call it a day. |
Rule (4): an alternative to rule (3) that would work for users as wellThe idea is to strip the strange corner cases out of (3) and solve specifically the problem it tries to solve. Also, remove inference of experimental to avoid bad UX.
This implies that all top classes must be marked as experimental as in normal experimental rules. We might also require top-level methods and values to be experimental. It is intended to be used with rule (1) to allow all the use cases in the tests of #13396 The question then is how to state that a source file is experimental. Here are some options:
|
I believe any of these conventions is too complicated and not necessary. it's still in the spirit that we should do our utmost to prevent usability rather than allow common use cases without hassles. |
I agree it would be good to have tests but I personally have spent way too much time on this already. So somebody else should add tests. And I insist that #13396 will go into 3.1, or experimental checking as a whole goes out. |
If we get #13396 we will have a flood of issues from users. I already have 5+ issues that I could immediately report. This would not be good from the user's perspective. To disable experimental checks we would need to make On the other hand, we have #13394 which does not touch non-experimental features, has proper tests, is useful for users, and is ready. |
We agreed that more precise version of (2) that only allows top-level definitions is the solution.
|
I don't know why this had been done. Seems like the enforcement rules make it's practically impossible to write libraries relying on "experimental" features, because all the client code needs to be annotated with this. |
sounds intentional to me... otherwise the concept of experimental has no teeth |
I guess I am a bit late to the "experimental" train, but am I correct in my understanding is that its not possible to avoid to the usage of the If so its somewhat annoying, I am actually currently developing a macro annotation that is designed portable across Scala 2.12, Scala 2.13 and Scala 3.3 (i.e. the whole point of the macro annotation is that you have a single source that is cross compiled against those Scala versions) and since macro annotations are considered experimental in Scala 3 this basically means that users of that macro annotation would have to mark |
In most cases it is possible to work around with various tricks, in worst case by relying on java reflection. |
Is there a reference (i.e. code sample) for these tricks? Also in case its not clear, I am not so worried about using import scala.annotation.MacroAnnotation
import scala.quoted.Quotes
import scala.annotation.experimental
@experimental class getInline extends MacroAnnotation Tne problem is that it appears to pollute/propogate the call site of the macro, i.e. to use import scala.annotation.experimental
object Test {
@experimental @getInline final def myVal = 5
} Which is what I meant beforehand about the annoying part. |
The whole point of @experimental is that it is transitive. You cannot have a definition that's @experimental and have the users not be @experimental. I also find that mechanism a bit heavy-handed since we don't want to sprinkle @experimental on all user code. But there is an alternative: we can also access experimental definition in a snapshot compiler release. This would mean you can cross compile using @experimental Scala 3 features, but the Scala 3 compiler would have to be a snapshot compiler. Question to @Kordyjan , @nicolasstucki , @smarter: What is currently the best way to specify snapshot release? Are we happy with that, or would we like to have something simpler or more robust? |
I would question the transitivity feature specifically for macro annotations here because unless I am missing something, unlike other experimental features (i.e. direct style) whether the macro annotation is experimental or not is not relevant for users (i.e. callers) of the macro. Macros just transform code so even if the implementation of how macro annotations is designed is marked as experimental thats a problem for the macro author, i.e. if the design of macro annotations changes then the author has to deal with it but the user doesn't have to do anything, i.e. for a user they will always just do I mean I guess there are ways around this, i.e. putting
At least for the project I am talking about (Pekko), using a snapshot compiler would very likely not cut it although I can see how this is a good general improvement for other cases. Note that there is a scala contributors thread with more context on the problem I am dealing with https://contributors.scala-lang.org/t/make-scala-2-inline-annotation-work-as-inline-keyword-for-scala-3-with-source-3-0-migration/6286 |
I quickly tried this suggestion and while it does work for Scala 3, it doesn't work for Scala 2 because the |
@odersky Should I make a discussion thread on Scala contributors on the more general point of whether |
Using a nightly release is probably the simplest. |
It would be good to see concrete incantations to achieve that, with consideration how it would work in the compiler, REPL, and the IDE. |
I think for macro annotations @experimental is currently unavoidable until they are fully stabilized. The question how to deal with @experimental in general could be interesting in a contributors thread. EDIT: One thing we probably need is to backport @experimental to the Scala 3 library. It's really high time we get that moving again, it was frozen for too long and this state is holding everyone back. |
True, but to clarify I am not suggesting making macro annotations fully stable. I am making a distinction between users of the macro and the implementation/design of the macro and my argument is that the former is stable (unless you are thinking of hypothetically removing macro annotations completely in the future at some point) where as the latter is unstable. Put differently, it really doesn't matter how macro annotations are designed/implemented, for a user the code will always be
Okay ill make a thread, better to talk about it there rather than derailing this issue thread. |
At this point the |
Until they are stabilized there's a theoretical chance that they might be removed or changed in a way that some functionality is no longer expressible. I don't expect it, but there's no formal guarantee it won't happen. So yes, we need users to be aware of that. |
The fact that there is a reference to |
This to me is more of a technical implementation detail because
Understood
Aside from the hypothetical macro annotations can be removed at some future point, would this case happen if users of the macro stick with a specific Scala 3 version? If not, then aside from the severe changes @odersky alluded to I don't see how it would effect users, i.e. if they upgrade their Scala version to 3.4 then that would bring in a new artifact which has a new implementation against a stable macro annotation API (not sure if this is the case, genuine question). |
If they use the same patch version (3.x.y) it will be OK, but not necessarily if they use the same minor release (3.x). |
In Scala 3.3.1, this issue becomes more prominent. Any usage of the Especially, @pshirshov (izumi-reflect) and I (airframe-surface) have created libraries based on Scala 3 macros. I believe we share the same concerns and troubles in dealing with this experimental mechanism. It is unfortunate that, even though some features are experimental, they are good enough to be used in production. Changing or dropping the API design in the future is acceptable, but the only currently available workaround is to depend on the snapshot compiler version, which will create an additional burden for library maintainers. |
If there are specific experimental features or APIs that the community considers high priority to be stabilized (not be experimental anymore), I think we should have specific tickets about those features. I doubt that any general expression of dissatisfaction about experimental stuff in general is going to have any effect — especially not on this ticket, as it's already been requested to move general discussion to https://contributors.scala-lang.org.
I'm curious: could or should the open community build have caught this...? That might be a subject for its own ticket or discussion thread, also. |
Is this new to Scala 3.3.1? I was experiencing this already in 3.3.0, at least with the prototype macro I am writing in https://github.com/mdedetrich/get-inline the users calling that macro annotation also had to add
I should get around to doing this, kinda fell of the table. |
@mdedetrich Yes. Since 3.3.1-RC1. Another user also reported the behavior change:
|
@SethTisue Yes. ClassDef.apply, Symbol.newClass are essential to produce some type of code, which was impossible to generate in Scala 3.1 #14124. Those are marked as experimental for some reason, probably testing the usability or feasibility: Libraries (izumi, zio, airframe) are already using these reflect methods, so the community build will be able to track the usage. Yeah. I should have opened another community thread instead of messing up this ticket. |
As requested, created a thread at https://contributors.scala-lang.org/t/behaviour-of-experimental-in-scala-3/6309 . @xerial feel free to post/add there. |
Compiler version
3.1.0
Minimized example
Output
Expectation
Should compile.
Otherwise, we cannot use experimental language features in @experimental code.
I think what happens is that we check the compiler version when we analyze the import statement. We should wait with it until we need to check whether the language import is enabled. If all these tests are from experimental scopes, this should be permitted.
The text was updated successfully, but these errors were encountered: