Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrating the main compiler to explicit nulls #14032

Merged
merged 12 commits into from
Mar 6, 2022

Conversation

noti0na1
Copy link
Member

@noti0na1 noti0na1 commented Dec 3, 2021

This PR tries to migrating the main compiler to explicit nulls.

Changes from #13975, #13976, and #14411 are included in this PR.

The explicit nulls currently is only enabled for bootstrap compile. We can test the changes using scala3-bootstrapped/compile.
We enable unsafeNulls in files to migrate step by step.

The first step of migration is the core parts (core, ast, typer, and tranform). The backend and some other modules are kept with unsafeNulls enabled, because they have a lot of interaction with Java libraries.


/** A map from (name-) offsets of all local variables in this compilation unit
* that can be tracked for being not null to the list of spans of assignments
* to these variables.
*/
def assignmentSpans(using Context): Map[Int, List[Span]] =
if myAssignmentSpans == null then myAssignmentSpans = Nullables.assignmentSpans
myAssignmentSpans
myAssignmentSpans.nn
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing this is needed because the flow typing only applies for local variables, not for class fields.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Perhaps we could do something for private fields, but it would still be unsound in the presence of threads.

An alternative would be

if myAssignmentSpans == null then
  val ret = Nullables.assignmentSpans
  myAssignmentSpans = ret
  ret
else
  myAssignmentSpans

But that's more verbose than the .nn.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could try this one

import scala.util.chaining.*
if myAssignmentSpans == null then Nullables.assignmentSpans.tap(myAssignmentSpans = _)
else myAssignmentSpans

@noti0na1
Copy link
Member Author

noti0na1 commented Dec 17, 2021

One of the difficulty during migration is: some fields are treated as non-nullable at a lot of places, but they are compared with null value at several locations (for detecting uninitialization or invalid value).
Because these fields could contain null value, the correct type should be nullable. However, if I change their definitions to nullable, then I have to add .nn to almost all places they are used. Since they are not local variables, flow typing doesn't work for them, which makes the migration more difficult.

One example is compilationUnit in Contexts. In many causes, we select members of compilationUnit directly; only in several places, we check whether it is null first. There is a TODO comment in code, which says we could add a NoCompilationUnit to avoid comparing with null.

However, adding a default value will not always work. For types of symbols, NoType or NoSymbol could have different meaning to null value.

Some examples are:

defn.TupleType(arity)
paramInfos
ctx.run

@noti0na1 noti0na1 force-pushed the en-migrate-main branch 2 times, most recently from f0e4a8e to fc1eca4 Compare January 11, 2022 21:13
@noti0na1
Copy link
Member Author

noti0na1 commented Jan 13, 2022

Currently there are some errors in picklingWithCompiler about pickling differences in Symbols.scala, ProtoTypes.scala and LazyVals.scala.

diff before-pickling.txt after-pickling.txt for LazyVals.scala shows:

$ diff before-pickling.txt after-pickling.txt
8213,8215c8213,8214
<                                 ):dotty.tools.dotc.core.Symbols.Symbol{ThisName = dotty.tools.dotc.core.Names.TermName}>
<                                   @
<                                 compiler/src/dotty/tools/dotc/transform/LazyVals.scala<15721..15736>
---
>                                 ):dotty.tools.dotc.core.Symbols.TermSymbol>@
>                                   compiler/src/dotty/tools/dotc/transform/LazyVals.scala<15721..15736>
8499c8498
<                               ):dotty.tools.dotc.core.Symbols.Symbol{ThisName = dotty.tools.dotc.core.Names.TermName}>@
---
>                               ):dotty.tools.dotc.core.Symbols.TermSymbol>@
8756c8755
<                           ):dotty.tools.dotc.core.Symbols.Symbol{ThisName = dotty.tools.dotc.core.Names.TermName}>@
---
>                           ):dotty.tools.dotc.core.Symbols.TermSymbol>@
9014c9013
<                         ):dotty.tools.dotc.core.Symbols.Symbol{ThisName = dotty.tools.dotc.core.Names.TermName}>@
---
>                         ):dotty.tools.dotc.core.Symbols.TermSymbol>@
9490c9489
<                   ):dotty.tools.dotc.core.Symbols.Symbol{ThisName = dotty.tools.dotc.core.Names.TermName}>@
---
>                   ):dotty.tools.dotc.core.Symbols.TermSymbol>@

It seems the difference is caused by .nn and TermSymbol. TermSymbol should be the same type as Symbol { ThisName = TermName }, I'm not sure why they become different after pickling?

@noti0na1
Copy link
Member Author

@smarter Do you have any idea about the pickling difference errors? Thanks

@smarter
Copy link
Member

smarter commented Jan 19, 2022

No precise idea, I think you'd have to minimize it to figure it out: somehow an expression of type TermSymbol has its type dealiased before pickling but on unpickling the recomputed type preserves the alias. Usually we try to preserve type aliases but there is no guarantee we do so if we can't figure this out we could also change the pickling tests to normalize by always dealiasing (we already do various normalizations like this in https://github.com/lampepfl/dotty/blob/2849aed4b1a14755a5fde3ea417f4b8bfa16685f/compiler/src/dotty/tools/dotc/printing/PlainPrinter.scala#L48)

@noti0na1 noti0na1 force-pushed the en-migrate-main branch 2 times, most recently from 366bea8 to b9c7e75 Compare January 21, 2022 01:32
@noti0na1
Copy link
Member Author

@smarter I can reproduce the error with only mutable variable and .nn.

compiler/src/dotty/tools/Test.scala

class Test {
  var x: String = null
  def f = x.nn
}

I add a simple test in compiler/test/dotty/tools/dotc/BootstrappedOnlyCompilationTests.scala:

@Test def myPicklingTest: Unit = {
  implicit val testGroup: TestGroup = TestGroup("testPicklingWithCompiler")
  aggregateTests(
    compileFile("compiler/src/dotty/tools/Test.scala", picklingWithCompilerOptions)
  ).limitThreads(4).checkCompile()
}

scala3-bootstrapped/testOnly -- *myPicklingTest

Fatal compiler crash when compiling: compiler/src/dotty/tools/Test.scala:
pickling difference for class Test in compiler/src/dotty/tools/Test.scala, for details:

  diff before-pickling.txt after-pickling.txt
        at dotty.tools.dotc.report$.error(report.scala:63)
        at dotty.tools.dotc.transform.Pickler.testSame(Pickler.scala:145)
        at dotty.tools.dotc.transform.Pickler.testUnpickler$$anonfun$2(Pickler.scala:135)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
        at dotty.tools.dotc.transform.Pickler.testUnpickler(Pickler.scala:136)
        at dotty.tools.dotc.transform.Pickler.runOn(Pickler.scala:119)
        at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:259)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:270)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:278)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:287)
        at dotty.tools.dotc.Run.compileSources(Run.scala:220)
        at dotty.tools.dotc.Run.compile(Run.scala:204)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:39)
        at dotty.tools.dotc.Driver.process(Driver.scala:199)
        at dotty.tools.dotc.Driver.process(Driver.scala:167)
        at dotty.tools.vulpix.ParallelTesting$Test.compile(ParallelTesting.scala:494)
        at dotty.tools.vulpix.ParallelTesting$CompilationLogic.compileTestSource$$anonfun$1(ParallelTesting.scala:213)
        at scala.util.Try$.apply(Try.scala:210)
        at dotty.tools.vulpix.ParallelTesting$CompilationLogic.dotty$tools$vulpix$ParallelTesting$CompilationLogic$$compileTestSource(ParallelTesting.scala:223)
        at dotty.tools.vulpix.ParallelTesting$$anon$5.checkTestSource$$anonfun$1(ParallelTesting.scala:269)
        at dotty.tools.vulpix.ParallelTesting$Test.tryCompile(ParallelTesting.scala:437)
        at dotty.tools.vulpix.ParallelTesting$$anon$5.checkTestSource(ParallelTesting.scala:272)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run(ParallelTesting.scala:340)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run$(ParallelTesting.scala:322)
        at dotty.tools.vulpix.ParallelTesting$$anon$5.run(ParallelTesting.scala:267)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
class dotty.tools.dotc.reporting.Diagnostic$Error at ?: pickling difference for class Test in compiler/src/dotty/tools/Test.scala, for details:

  diff before-pickling.txt after-pickling.txt while compiling compiler/src/dotty/tools/Test.scala

before-pickling.txt

<package <empty>.type {
  @SourceFile(
    <"compiler/src/dotty/tools/Test.scala":("compiler/src/dotty/tools/Test.scala" : String)>@
      compiler/src/dotty/tools/Test.scala<no position>
  ) class Test() extends <
    <<new Object:Object>@compiler/src/dotty/tools/Test.scala<6..6>:((): Object)>@
      compiler/src/dotty/tools/Test.scala<6..6>
  ():Object>@compiler/src/dotty/tools/Test.scala<6..6> {
    var x: scala.Predef.String@compiler/src/dotty/tools/Test.scala<22..28> = 
      <null:scala.Null>@compiler/src/dotty/tools/Test.scala<31..35>
    @compiler/src/dotty/tools/Test.scala[15..19..35]@@(x=compiler/src/dotty/tools/Test.scala:<19..19>)
    def x_=(
      x$1: scala.Predef.String@compiler/src/dotty/tools/Test.scala<15..15>@@(x$1=
        compiler/src/dotty/tools/Test.scala:<15..15>
      )
    ): scala.Unit = <():scala.Unit>@compiler/src/dotty/tools/Test.scala<15..15>@
      compiler/src/dotty/tools/Test.scala[15..19..35]
    @@(x_==compiler/src/dotty/tools/Test.scala:<19..19>)
    def f: scala.Predef.String = 
      <
        <<scala.Predef.nn:([T](x: T | scala.Null): x.type & T)>@compiler/src/dotty/tools/Test.scala<48..50>[String]:
          ((x: String): x.type)
        >@compiler/src/dotty/tools/Test.scala<48..50>
      (
        <<this:(Test.this : Test)>@compiler/src/dotty/tools/Test.scala<46..46>.x:scala.Predef.String>@
          compiler/src/dotty/tools/Test.scala<46..47>
      ):String>@compiler/src/dotty/tools/Test.scala<46..50>
    @compiler/src/dotty/tools/Test.scala[38..42..50]@@(f=compiler/src/dotty/tools/Test.scala:<42..42>)
  }@compiler/src/dotty/tools/Test.scala[0..6..52]@@(Test=compiler/src/dotty/tools/Test.scala:<6..6>)
}:<empty>.type>@compiler/src/dotty/tools/Test.scala<0..52>

after-pickling.txt

<package <empty>.type {
  @SourceFile(
    <"compiler/src/dotty/tools/Test.scala":("compiler/src/dotty/tools/Test.scala" : String)>@
      compiler/src/dotty/tools/Test.scala<no position>
  ) class Test() extends <
    <<new Object:Object>@compiler/src/dotty/tools/Test.scala<6..6>:((): Object)>@
      compiler/src/dotty/tools/Test.scala<6..6>
  ():Object>@compiler/src/dotty/tools/Test.scala<6..6> {
    var x: scala.Predef.String@compiler/src/dotty/tools/Test.scala<22..28> = 
      <null:scala.Null>@compiler/src/dotty/tools/Test.scala<31..35>
    @compiler/src/dotty/tools/Test.scala[15..19..35]@@(x=compiler/src/dotty/tools/Test.scala:<19..19>)
    def x_=(
      x$1: scala.Predef.String@compiler/src/dotty/tools/Test.scala<15..15>@@(x$1=
        compiler/src/dotty/tools/Test.scala:<15..15>
      )
    ): scala.Unit = <():scala.Unit>@compiler/src/dotty/tools/Test.scala<15..15>@
      compiler/src/dotty/tools/Test.scala[15..19..35]
    @@(x_==compiler/src/dotty/tools/Test.scala:<19..19>)
    def f: scala.Predef.String = 
      <
        <<scala.Predef.nn:([T](x: T | scala.Null): x.type & T)>@compiler/src/dotty/tools/Test.scala<48..50>[String]:
          ((x: String): x.type)
        >@compiler/src/dotty/tools/Test.scala<48..50>
      (
        <<this:(Test.this : Test)>@compiler/src/dotty/tools/Test.scala<46..46>.x:scala.Predef.String>@
          compiler/src/dotty/tools/Test.scala<46..47>
      ):scala.Predef.String>@compiler/src/dotty/tools/Test.scala<46..50>
    @compiler/src/dotty/tools/Test.scala[38..42..50]@@(f=compiler/src/dotty/tools/Test.scala:<42..42>)
  }@compiler/src/dotty/tools/Test.scala[0..6..52]@@(Test=compiler/src/dotty/tools/Test.scala:<6..6>)
}:<empty>.type>@compiler/src/dotty/tools/Test.scala<0..52>

diff before-pickling.txt after-pickling.txt

27c27
<       ):String>@compiler/src/dotty/tools/Test.scala<46..50>
---
>       ):scala.Predef.String>@compiler/src/dotty/tools/Test.scala<46..50>

We don't have the error when x is val or using .uncheckedNN.

@smarter
Copy link
Member

smarter commented Jan 21, 2022

This is likely because the type seen before pickling contains a skolem (when I tweak the printer to not strip prefix and print after posttyper I see (<?1:scala.Predef.String> : scala.Predef.String) & java.lang.String) but skolems aren't preserved by pickling and so when unpickling and simplifying the type we end up printing it in a slightly different way. Besides the change to the printer I mentioned above the only way to avoid that would be to avoid inferring a type with a skolem when it's not useful like here, but I don't know what would be required to do that.

compiler/src/dotty/tools/dotc/ast/TreeInfo.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/ast/Trees.scala Show resolved Hide resolved
compiler/src/dotty/tools/dotc/core/Contexts.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/core/Contexts.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/typer/Implicits.scala Outdated Show resolved Hide resolved
@@ -303,7 +306,7 @@ object Implicits:
*/
class ContextualImplicits(
val refs: List[ImplicitRef],
val outerImplicits: ContextualImplicits,
val outerImplicits: ContextualImplicits | Null,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this ever null other than for NoContext? If not, is there anything cleaner we could do?

compiler/src/dotty/tools/dotc/typer/ImportInfo.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/typer/Typer.scala Outdated Show resolved Hide resolved
@noti0na1 noti0na1 force-pushed the en-migrate-main branch 2 times, most recently from 29148d2 to aa39739 Compare February 2, 2022 08:45
@noti0na1 noti0na1 changed the title [WIP] Migrating the main compiler to explicit nulls Migrating the main compiler to explicit nulls Feb 2, 2022
@noti0na1 noti0na1 marked this pull request as ready for review February 2, 2022 08:46
@noti0na1 noti0na1 force-pushed the en-migrate-main branch 2 times, most recently from 2f8d297 to e08d48b Compare February 10, 2022 01:19
@bishabosha
Copy link
Member

test performance please

@dottybot
Copy link
Member

performance test scheduled: 1 job(s) in queue, 0 running.

@dottybot
Copy link
Member

Performance test finished successfully:

Visit https://dotty-bench.epfl.ch/14032/ to see the changes.

Benchmarks is based on merging with main (29f9d33)

@noti0na1 noti0na1 requested a review from odersky February 19, 2022 03:02
@noti0na1 noti0na1 assigned odersky and unassigned noti0na1 Feb 19, 2022
@noti0na1
Copy link
Member Author

@odersky @sjrd

To enable explicit nulls and disable unsafe nulls globally, I think we need to modify some code in scala-js, because the code of scala-js is included during bootstrapped compiling.

scalajs-ir-src/org/scalajs/ir/

Names.scala
OriginalName.scala
Position.scala
SHA1.scala
ScalaJSVersions.scala
Serializers.scala
UTF8String.scala

(We can simply add import scala.lang.unsafeNulls to each file to avoid errors for now)

@sjrd
Copy link
Member

sjrd commented Feb 25, 2022

Hum, but that's not so easy to do. These files still cross-compile all the way back to Scala 2.11.12. So there's no way we can add that import upstream.

I see two paths forward:

  • patch up the sources from the build that extracts those sources, or
  • compile those sources in a separate project, and then reintegrate the .class and .tasty files produced in the compiler jar.

@odersky
Copy link
Contributor

odersky commented Mar 3, 2022

Hi Yaoyu, I am trying to get a clean changeset for the whole PR. Can you rebase over current main? Then I could squash all commits and see all changes together. Thanks!

@noti0na1
Copy link
Member Author

noti0na1 commented Mar 3, 2022

@odersky I have rebased and squashed my commits.

@noti0na1
Copy link
Member Author

noti0na1 commented Mar 4, 2022

test performance please

@dottybot
Copy link
Member

dottybot commented Mar 4, 2022

performance test scheduled: 1 job(s) in queue, 0 running.

@dottybot
Copy link
Member

dottybot commented Mar 4, 2022

Performance test finished successfully:

Visit https://dotty-bench.epfl.ch/14032/ to see the changes.

Benchmarks is based on merging with main (727395c)

Copy link
Contributor

@odersky odersky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise looking good now. I suggest to apply the final changes, leave notes where this cannot be done, and merge. We can improve the codebase incrementally from there.

I am going to leave some notes about painpoints of null checking in a separate issue.

while e != null do
if isEqual(e, x) then return e
if isEqual(e.uncheckedNN, x) then return e
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have thought that flow typing would work here so that uncheckedNN is not needed? But we can resolve that later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for other uses of uncheckedNN in HashSet and HashMap.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment. We will doule-check every uncheckedNN once we update the reference compiler.

compiler/src/dotty/tools/package.scala Show resolved Hide resolved
@@ -338,7 +337,7 @@ object Implicits:
if monitored then record(s"check uncached eligible refs in irefCtx", refs.length)
val ownEligible = filterMatching(tp)
if isOuterMost then ownEligible
else combineEligibles(ownEligible, outerImplicits.uncachedEligible(tp))
else combineEligibles(ownEligible, outerImplicits.nn.uncachedEligible(tp))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the rationale of using sometimes uncheckedNN and sometimes nn on outerImplicits?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used uncheckedNN in def level because I know outerImplicits is non-nullable immediately from the if condition. I should change uncheckedNN to nn in other places.

compiler/src/dotty/tools/dotc/core/TyperState.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/core/TyperState.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/core/Types.scala Outdated Show resolved Hide resolved
compiler/src/dotty/tools/dotc/core/Uniques.scala Outdated Show resolved Hide resolved
@@ -129,77 +135,71 @@ abstract class WeakHashSet[A <: AnyRef](initialCapacity: Int = 8, loadFactor: Do
tableLoop(0)
}

def lookup(elem: A): A | Null = elem match {
case null => throw new NullPointerException("WeakHashSet cannot hold nulls")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was this changed? I think it's better if WeakHashSets don't contains nulls.

Copy link
Member Author

@noti0na1 noti0na1 Mar 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since A <: AnyRef, the type can ensure the value is not null. I can revert the change for now, and add a TODO to delete the case when we can enable explicit nulls for regular compiling.

@noti0na1
Copy link
Member Author

noti0na1 commented Mar 6, 2022

@odersky Some of the uncheckedNN is just to make sure the code can be compiled in regular "compile", because the explicit nulls is only enabled for bootstrapped compiling.

Once the reference compiler is updated, we can drop these uncheckedNN .

@noti0na1
Copy link
Member Author

noti0na1 commented Mar 6, 2022

@odersky The changes have been applied

Copy link
Contributor

@odersky odersky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good now! Thanks for all the hard work. This is a big step towards full null checking.

@noti0na1 noti0na1 merged commit a05ff76 into scala:main Mar 6, 2022
@noti0na1 noti0na1 deleted the en-migrate-main branch March 6, 2022 13:36
@SethTisue
Copy link
Member

Very cool to see this landing 👏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants