Add Scala 3 support via dotty compatibility

[alexarchambault]

So this PR adds Scala 3 support, as a follow up of #1144.

Its changes successively consist in:

• minor refactorings, easing subsequent developments (from 0dd150b up to 6208f80, 12 commits as of now),
• actually adding Scala 3 support (8c22bfa up to 6a815b0, 10 commits)
• updating the tests, so that they run fine for Scala 3 (a4e2db9 up to the end, 28 commits).

All commits but one don't touch more than approx. 100 lines each, which should make it easy to review those. The only sizable commit is the one adding the Scala 3 implementation of the compiler module, under amm/compiler/src/main/scala-3 (8c22bfa). This commit mainly:

• adds new sources under amm/compiler/src/main/scala-3,
• updates build.sc so that using just 3 as cross version builds and runs Ammonite as a mix of Scala 2.13 / 3 modules (that run fine together thanks to dotty compatibility)

Many of the sources this commit adds under amm/compiler/src/main/scala-3 are adapted either from dotty sources, or from Ammonite Scala 2 things.

Some features don't work yet in Scala 3, or don't work as fine as in Scala 2 yet. The commits updating the tests should make clear what has been disabled for Scala 3. Most notably:

• entrypoints: this would require mainargs to be ported over to Scala 3, its use of Scala 2 macros complicates that,
• line numbers in diagnostics and stack traces: an equivalent of the LineNumberModifier plugin for Scala 3 would fix both at the same time, else ammonite.interp.script.PositionOffsetConversion could be used to fix at least the former
• deep completions: there's currently a candidate implementation for these (in amm/compiler/src/main/scala-3/dotty/ammonite/compiler/Completion.scala), but it's awfully slow, so it's disabled by default (look for the commented out line calling addDeepCompletions)
• import $ivy completion: there's a candidate implementation too (in Completion.scala), but it has issues with backticks. This required tinkering in Scala 2, which seemed less straightforward to do in Scala 3 (IIRC, dotty doesn't easily hand us over backticks around what's being completed)
• source and desugar macros: they need to be ported over to Scala 3 (using Tasty reflection-based macros I guess)

There are also limitations in the Scala 3 specific code, most notably:

• the script blocks splitter (splitting around @) is regex based for now (I didn't try to have the dotty parser handle those yet)

[alexarchambault]

@lihaoyi This is ready for review, if you wish. Don't fear the number of commits or the number of changed lines… All commits but one touch less than or around 100 lines. The only sizeable commit is the one adding the Scala 3 source of the compiler module - it mostly only adds code, but for its changes in build.sc.

That said, I can always split that to make it easier to review it:

• all the commits until the one adding Scala 3 code in compiler in a first PR, then
• adding Scala 3 code per se, then
• adding CI jobs and adjusting the tests for Scala 3.

The tests now pass in #1151, that switches to 3.0.0-M3. Its developments could also be folded here.

Lastly, some things could be cleaned-up in the biggest commit (Scala 3 code in compiler), I could update that, but that shouldn't hamper review… (or even merging).

[lihaoyi]

@alexarchambault I skimmed this, I think it looks good to me. I trust both that you know your way around the codebase, and also our test suite to catch any regressions in the old code paths.

Since the bulk of the code here is in Scala 3, could it be worth getting someone more familiar with Scala3 to review those sections? Even if you need to explain what the code is meant to do, a second perspective from the Scala3 side of things may be valuable to make sure what we're doing sounds reasonable to them

[alexarchambault]

(Last push force: folded #1151 here.)

[alexarchambault]

Since the bulk of the code here is in Scala 3, could it be worth getting someone more familiar with Scala3 to review those sections? Even if you need to explain what the code is meant to do, a second perspective from the Scala3 side of things may be valuable to make sure what we're doing sounds reasonable to them

I agree it would be nice to have more Scala 3 expert eyes have a look, yes.

@smarter @anatoliykmetyuk Don't know if you guys would be interested in having a look at the commit named Add Scala 3 stuff, currently at ecf2928. This commit adds Scala 3 implementations for parsing and compiler related stuff.

In more detail, it adds the following classes:

• AmmonitePhase: a Scala 3 equivalent to AmmonitePlugin, which tracks new variables, but also new imports, during compilation
• Compiler: as the name says, going from source to byte code, but also some completion-related things
• DottyParser and Parsers: parsing-related stuff
• Preprocessor; generating additional code (printing the type and value of new vals, etc.) from an AST of the input code (most of it adapted from its Scala 2 counterpart)
• SyntaxHighlighting: adapted from dotty sources
• Completion: adapted from dotty sources too, so that we can add support for import $ivy completions, deep completion (committed but disabled for now, as my current implem is too slow), but also for a better handling of back-quoted identifiers (that could be interesting for the dotty REPL too)

(I might push force some clean-ups of that commit, as I just found some duplication or formatting related issues…)

[smarter]

Looks like there's quite a lot of code adapted from existing code in dotty, my advice would be to instead change dotty to make the existing code reusable for your purposes when possible.

[alexarchambault]

Looks like there's quite a lot of code adapted from existing code in dotty, my advice would be to instead change dotty to make the existing code reusable for your purposes when possible.

I agree. That's mostly syntax-highlighting and completion-related code. I'll look into submitting a PR to dotty.

[mallman]

Hi @alexarchambault. I was just working with Spark SQL in Jupyter thanks to Almond, and I'm curious how you're handling the removal of scala.Symbol and its related ' literal syntax from Scala 3 in this patch. Have you provided some kind of substitute? Unfortunately I don't understand this patch well enough to answer this question for myself. FWIW, my inquiry is partly motivated by the extensive use of the symbol literal syntax in the documentation for Ammonite at ammonite.io. However, I must admit I'm also rather partial to that syntax for use in Spark SQL, so I'm hopeful you have found a pleasant alternative.

Cheers.

[alexarchambault]

@mallman In the Ammonite codebase, most symbol literals (maybe even all of them, I think) are related to os-lib paths, in code like os.pwd / 'foo / 'something. Luckily, os-lib also accepts string literals, so I just changed such code to os.pwd / "foo" / "something".

I'm not sure how spark-sql will handle the end of symbol literals, but IIRC, it also accepts string literals in most places, right? Using string literals would be the way to go then.

(By the way, this kind of question might be more suited to the Gitter channel, although I have to admit I should check my Gitter notifications more often 😬)