Generate IR definitions by annotation processor - 1st step #11770

Akirathan · 2024-12-04T21:02:08Z

This PR is the initial step for migrating all the IR elements implemented as Scala case classes to Java classes generated by an annotation processor.

This PR was created from the abandoned #11267 that tried to generated subclasses.

Pull Request Description

The overall description is the same as in #11267, but this approach tries to generate super classes as suggested in https://github.com/enso-org/enso/pull/11267/files#r1869342527

References

New project runtime-parser-dsl contains new annotations for IR elements. The docs for these annotations are the most important. See org.enso.runtime.parser.dsl package
New project runtime-parser-processor contains the implementation of the annotation processor. The entry point to annotation processing is in org.enso.runtime.parser.processor.IRProcessor.
New project runtime-parser-processor-tests depends on aforementioned projects as well as on runtime-parser.
- The main test is TestIRProcessorInline.java that tests the annotation processor by compiling snippets of code with the help of com.google.testing.compile:compile-testing library.
- Another test is GeneratedIRTest.scala that test the usage of generated code.

Important Notes

Apart from the annotation processor implementation and tests, CallArgument.Specified was migrated to the new approach from Scala case class. I uploaded its generated class code in this gist so you can see it without compilation.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

Benchmarks are OK
The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

This will be important for mapExpressions method implementation

This is the only way to retain 100% backward compatibility.

…processor" This reverts commit d2bb8c3.

This reverts commit 68a070b.

From LazyDiagnostics trait.

JaroslavTulach

I am surprised there is hand written implementation of copy method in CallArgument.Specified - it'd be better if it was generated. The amount of arguments sent to super constructor is surprising.

JaroslavTulach · 2024-12-31T04:53:52Z

engine/runtime-parser/src/main/java/module-info.java

@@ -2,6 +2,8 @@
  requires org.enso.syntax;
  requires scala.library;
  requires org.enso.persistance;
+  requires static org.enso.runtime.parser.dsl;
+  requires static org.enso.runtime.parser.processor;


Requires static is correct.

JaroslavTulach · 2024-12-31T04:55:58Z

engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/CallArgument.java

+    }
+
+    public Specified copy(Expression value) {
+      if (value != this.value()) {


Looks like boiler plate code. I was hoping this code would be generated.

JaroslavTulach · 2024-12-31T04:57:06Z

engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/Empty.java

+public final class Empty extends EmptyGen {
+  @GenerateFields
+  public Empty(IdentifiedLocation identifiedLocation, MetadataStorage passData) {
+    super(null, passData, identifiedLocation, null);


What does it mean super(null, ..., null)? Why two null parameters?

IRNodeClassGenerator generates two constructors - the "main" constructor which has all user-defined and meta fields as parameters, and another one that has only user-defined fields as parameters. This super calls the constructor which has meta fields. There are four meta fields. It is possible, and probably better, to generate constructor in the super class that exactly matches the constructor here though.

JaroslavTulach · 2024-12-31T04:59:47Z

engine/runtime-parser/src/test/java/org/enso/compiler/core/ParserDependenciesTest.java

+  @Test
+  public void parserProcessorIsAvailable() {
+    try {
+      var clazz = Class.forName("org.enso.runtime.parser.processor.IRProcessor");


Strange test. Why are we testing this class can be loaded? Of course, because it on module path of runtime-parser project as provided and that implies it is visible in the tests of the runtime-parser project.

I'd rather have a test in runtime-integration-tests that IRProcessor class cannot be loaded. Because we don't want this class in the final JARs that we deploy.

hubertp

I feel like processor would become much more readable if we used String templates which are now available. A combination of StringBuilders and text replacing can be sometimes hard to read.

build.sbt

engine/runtime-parser-dsl/src/main/java/org/enso/runtime/parser/dsl/GenerateIR.java

hubertp · 2024-12-31T10:08:21Z

...or-tests/src/main/java/org/enso/runtime/parser/processor/test/gen/ir/core/JCallArgument.java

+/** Call-site arguments in Enso. */
+public interface JCallArgument extends IR {
+  /** The name of the argument, if present. */
+  Option<Name> name();


Why Scala option?

To retain backward compatibility. We use scala.Option all over the place in IR types.

hubertp · 2024-12-31T10:13:40Z

...-processor-tests/src/test/scala/org/enso/runtime/parser/processor/test/GeneratedIRTest.scala

+  "JSpecifiedGen" should "have generated parameter names with javac compiler" in {
+    val lit     = Literal.Text("foo", null, new MetadataStorage())
+    val callArg = new JSpecified(isSynthetic = true, value = lit, name = None)
+    callArg should not be null


that's always guaranteed to be true, no?

It is. This test is here just to ensure that new JSpecified can be used with named arguments.

...r-processor/src/main/java/org/enso/runtime/parser/processor/GenerateIRAnnotationVisitor.java

hubertp · 2024-12-31T13:41:28Z

...cessor/src/main/java/org/enso/runtime/parser/processor/methodgen/BuilderMethodGenerator.java

+
+    var code =
+        """
+        public static final class Builder {


Could use string templates since we are on latest jdk?

Unfortunately, Frgaal still does not support string templates.

hubertp · 2024-12-31T13:46:07Z

...src/main/java/org/enso/runtime/parser/processor/methodgen/MapExpressionsMethodGenerator.java

+public final class MapExpressionsMethodGenerator {
+  private final ExecutableElement mapExpressionsMethod;
+  private final GeneratedClassContext ctx;
+  private static final String METHOD_NAME = "mapExpressions";


Probably should use came case, as everywhere else

I don't know about "everywhere else". For example, in Constants.java we use upper case, as well as in Builtins.java

hubertp · 2024-12-31T13:52:36Z

...ne/runtime-parser-processor/src/main/java/org/enso/runtime/parser/processor/utils/Utils.java

+              // it.
+              // This is because runtime-parser-processor project does not depend on runtime-parser
+              // and
+              // so the org.enso.compiler.core.IR interface is not available in the classpath.


I assume the sad face is there because we have not yet separated the runtime-parser project into a separate project that would contain the IR interface.

...ne/runtime-parser-processor/src/main/java/org/enso/runtime/parser/processor/utils/Utils.java

hubertp · 2024-12-31T14:04:08Z

Is there any impact on benchmarks with all this machinery?

Akirathan added 30 commits October 6, 2024 21:19

Add parser-dsl and parser-processor projects

6e33120

Add JModule

69f9247

Fix module declaration - no compiler module dependency

0e4ceb0

Add annotation processor testing.

9891eb7

Rewrite some IR elements to IRNode annotated interfaces

a3bf203

IRProcessor does some sanity checks

08b0571

IRNode does not have name parameter

51796ba

IRProcessor generates source file for record

3ea7a90

Add imports and overriden IR methods to generated record

f6bf733

Add field processing

1870ef2

IRProcessor generates classes, not records

826245d

IRProcessor generates overrides for user-defined parameterless methods

fc89b3d

Add Field.isExpression method

1fc0497

This will be important for mapExpressions method implementation

sbt picks up Scala case classes generated by Java annotation processor

d2bb8c3

IRProcessor generates Scala classes.

68a070b

This is the only way to retain 100% backward compatibility.

Revert "sbt picks up Scala case classes generated by Java annotation …

cf08781

…processor" This reverts commit d2bb8c3.

Revert "IRProcessor generates Scala classes."

f930e53

This reverts commit 68a070b.

IRProcessor handles primitive fields

298c883

Remove unused methods

bc1e9c9

IRProcessor fails fast

ef05c64

IRProcessor can collect fields from super interfaces

aa0a5b6

Fix collecting fields from super interfaces.

c6def78

Prepare IRNodeProcessor for processing multiple nested interfaces

8cea491

IRNodeProcessor handles multiple nested interfaces

f5cb331

runtime-parser-processor does not depend on runtime-parser

2393e18

Move parser processor tests to a separate project

cf1d079

runtime-parser-processor does not depend on runtime-parser

f63bda1

runtime-parser depends on runtime-parser-processor

158a4f1

IRProcessor adds package

1289374

ParserDependenciesTest ensures that IRNodeProcessor is on class path

73a9d31

Akirathan added 17 commits December 30, 2024 15:41

Duplicate method has specific return type

94c40d0

Generate also parameterless duplicate method

aaad0c7

Generated class can already exist - will be rewritten

c062b90

Generate diagnosticsCopy.

140d7b5

From LazyDiagnostics trait.

Replace CallArgument Scala code with the generated class.

a5420c1

Fix duplicate method - passData and diagnostics are copied.

0708535

Generate toString method

e2c3c3b

Fix expected error message

3086f52

FIx ensureExtendsGeneratedSuperclass

24b28ad

Fix NPE in duplicate

7479925

mapExpressions duplicates this iff one of the mapped children changed

f8c9e12

mapExpressions do not recurse on Expression children

198ce88

Specified.copy only copies if parameter is different

70b1be3

Pass null to meta field in super ctor

a25ad6f

Fix whitespaces in formatting of generated classes

80509a1

No "copy constructor" for Builder

df0a001

imports are sorted

bd4b0b2

Akirathan changed the title ~~[Poc] Generate IR definitions - superclass approach~~ Generate IR definitions by annotation processor - 1st step Dec 30, 2024

Akirathan marked this pull request as ready for review December 30, 2024 20:21

Akirathan requested review from 4e6 and hubertp as code owners December 30, 2024 20:21

JaroslavTulach approved these changes Dec 31, 2024

View reviewed changes

hubertp reviewed Dec 31, 2024

View reviewed changes

Akirathan added 6 commits January 1, 2025 13:27

Replace Supplier with private method

412d466

Add docs

3d865b9

fix typos

fab106f

Rename minus to diff

fe0a378

Introduce constants

42a988f

passData is never null

cbaa6f7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate IR definitions by annotation processor - 1st step #11770

Generate IR definitions by annotation processor - 1st step #11770

Akirathan commented Dec 4, 2024 •

edited

Loading

JaroslavTulach left a comment

JaroslavTulach Dec 31, 2024

JaroslavTulach Dec 31, 2024

JaroslavTulach Dec 31, 2024

Akirathan Jan 1, 2025

JaroslavTulach Dec 31, 2024

hubertp left a comment

hubertp Dec 31, 2024

Akirathan Jan 1, 2025

hubertp Dec 31, 2024

Akirathan Jan 1, 2025

hubertp Dec 31, 2024

Akirathan Jan 1, 2025

hubertp Dec 31, 2024

Akirathan Jan 1, 2025

hubertp Dec 31, 2024

Akirathan Jan 1, 2025

hubertp commented Dec 31, 2024

Generate IR definitions by annotation processor - 1st step #11770

Are you sure you want to change the base?

Generate IR definitions by annotation processor - 1st step #11770

Conversation

Akirathan commented Dec 4, 2024 • edited Loading

Pull Request Description

References

Important Notes

Checklist

JaroslavTulach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hubertp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hubertp commented Dec 31, 2024

Akirathan commented Dec 4, 2024 •

edited

Loading