Improved support for large repositories (#3118)

* Large repository handling

* rewrite-xml tests passing

* AddGradleEnterpriseTest passing

* UpgradeGradleWrapper passing

* License headers

* Reject version of rocksdb which is broken on Windows

* Fix CreateTextFile. Move Groovy adaptability tests into a package where they wont name collide with java tests

* Remove visitJavaSourceFile

* Move org.openrewrite.java.cleanup to rewrite-static-analysis

* Move a few commonly used cleanup recipes back

* Move some recipes back from static-analysis that are actually formatting recipes

* Move groovy cleanup tests to rewrite-static-analysis, fix other compilation issues

* Fix UnnecessaryParentheses

* Fix SelectRecipeExamplesTest

* Fix more tests, move java-best-practices to rewrite-static-analysis

* Fix remaining tests

* Remove deprecated gradle parser builders

* Remove some incubating annotations, order imports

* Remove select deprecations

* Remove `JavaParser#[gs]etSourceSet()` overrides

* Make precondition of `ChangeMethodTargetToStatic` optional

Also add `stopAfterPreVisit()` call to `RemoveImport#preVisit()` and fix failing `UseJavaParserBuilderInJavaTemplateTest` test.

* Make sure `Preconditions.check()` also works with other source types

* Add a few more `stopAfterPreVisit()` directives

* Change return type of `ScanningRecipe#generate()`

Use `Collection<? extends SourceFile>` rather than `Collection<SourceFile>`.

* Update call site for `ScanningRecipe#generate()`

* Don't attempt to instantiate abstract recipe classes

For example the `ScanningRecipe` type.

* Change top-level detection logic in `TreeVisitor#visit()`

Using `cursor.getParent() == root` doesn't work when a visitor is called with a client-supplied cursor (i.e. `visit(Tree, P, Cursor)`, as then the top-level for that visitor is not properly detected and as a result the after visitors don't get applied.

* Reset `TreeVisitor#visitCount` after exiting top-level

This way the visitor can be reused.

* Add `stopAfterPreVisit()` call to `AddImport#preVisit()`

* Update signatures of `Recipe#run()` methods

* Resolve some conflicts after rebasing

* Changeset

* SourceSet -> LargeSourceSet

* Polish

* Polish

* Polish

* Fix constructor calls in `InMemoryLargeSourceSet`

Also add generated and deleted sources to changeset.

* FindCallGraph public getVisitor

* Missing expected generated file check was in wrong place in RewriteTest

* Recipe#generate can observe other files generated in the same cycle

* Add guard for loading yaml source, gracefully handling the loading error. (#3224)

* Add guard for loading yaml source, gracefully handling the loading error.

* Fix PlainText#withText

* Iteration on cachable, context-free java templating.

* Cleanup

* Refactor to `JavaTemplate.Builder#requiresContext(boolean)`

The `JavaTemplate` currently also requires access to the `Cursor` in order to be able to format the code (using `JavaVisitor#autoFormat()`) before returning it to the caller. The alternative would be to require the calling visitor to apply the formatting itself after updating the LST.

* Add `Cursor` parameter to `J#withTemplate()`

* Update all uses of `JavaTemplate#builder()` and `J#withTemplate()`

* Polish

* ChangeText not extendable

* Recipe#getDescription not optional

* Polish AddDependency

* Remove LargeSourceSet#flatMap and LargeSourceSet#concat

* Remove LargeSourceSet#getInitialState

* Updated `JavaTemplate` uses to context-free where possible

* Fix bug in `RecipeSchedulerCheckingExpectedCycles`

`LargeSourceSet` parameter of `afterCycle()` could be same instance from previous cycle.

* `ScanningRecipe#getVisitor()` must implement `isAcceptable()`

Also fix failing `ExtractInterfaceTest`.

* Post-merge conflict resolution

* Fix last failing test

The estimated effort was taken from the root recipe rather than from the "leaf recipe".

* Fix bug when replacing context-free method body

* For now exclude lambdas and method references from context-free support

* Further simplify LargeSourceSet, remove ForkJoinScheduler

* Rename RecipeScheduler#mapForRecipeRecursively

* LargeSourceSet edit/generate

* Add simple test for context-free template caching

* RewriteTest customizable LargeSourceSet builder

* Fix migration recipes

* Change ScanningRecipe's acc message id to use hashCode instead of UUID to solve some potential issues of multiple ScanningRecipes in a getRecipleList

* NullSkippingSet -> NullSkippingMap

* Revert "Change ScanningRecipe's acc message id to use hashCode instead of UUID to solve some potential issues of multiple ScanningRecipes in a getRecipleList"

This reverts commit bff5444.

* visitParameterizedType needs to call withType

* getRecipeList should not return new recipe instances

* Javadoc on getRecipeList

* Add validation that Recipe#getRecipeList always returns the same references

* Propagate shared root cursor via `visit(Tree, ExecutionContext, Cursor)`

To let the recipe's visitor use the shared root cursor the `RecipeScheduler` now invokes `visit(Tree, ExecutionContext, Cursor)` rather than first explicitly setting the `TreeVisitor#cursor` field and then calling `visit(Tree, ExecutionContext)`. This is necessary for the wrapping visitors created by `Preconditions` and `ScanningRecipe`, as the shared root otherwise won't get propagated to the actual visitor.

* Fix compiler problem with Lombok `@Getter(lazy = true)`

* Set root cursor before calling `TreeVisitor#isAcceptable()`

This is required by the implementation of `ScanningRecipe#isAcceptable()` since it retrieves the accumulator from the root cursor

* make RemoveGradleDependency recipe public

* Add `@JsonIgnore` to `FindDeprecatedUses#recipeList`

Somehow the `@Getter(lazy=true)` coupled with the initializer method appears to change the serialization.

* Fix bug in `SemanticallyEqual` when comparing modifiers

`J.Modifier` instances cannot be compared using `Object#equals(Object)` since this will only compare the `id` field. Instead, the `J.Modifier#type` should be compared.

* Fix failing HCL tests after root cursor sharing refactoring

* Fix failing YAML tests after root cursor sharing refactoring

* Avoid J.ArrayType double visit on type

* Don't add types in use to `JavaSourceSet`

`Assertions#addTypesToSourceSet()` now no longer iterates over the `TypesInUse` of each compilation unit to add the `JavaType`s to the `JavaSourceSet` classpath.

* Restore ShortenFullyQualifiedTypeReferences from rewrite-static-analysis

* Eliminate double visit on AnnotatedType type

* chore: deprecate PartProvider, and change to use internal helper function in MigrateRecipeToRewrite8

* Fix error in `README.md`

* getRecipeList called once per cycle

* getInitialValue() -> getInitialValue(ExecutionContext)

* Remove recipeList validation from RewriteTest

* Remove deprecated Gradle recipes

* Get tests passing

* Update benchmark code

* Fixes compilation issues with gradle

* Revert "Fixes compilation issues with gradle"

This reverts commit 486c30e.

* Flush RecipeRunStats

* Autodetecte

* Fix SourceFileResults data table (#3280)

---------

Co-authored-by: Sam Snyder <[email protected]>
Co-authored-by: Knut Wannheden <[email protected]>
Co-authored-by: Kun Li <[email protected]>
Co-authored-by: Tim te Beek <[email protected]>
Co-authored-by: Kun Li <[email protected]>
Co-authored-by: rpau <[email protected]>

README.md

@@ -1,5 +1,5 @@
 <h1 align="center">OpenRewrite</h1>
-<p align="center"><i></i><b>🎉 Automate software refactoring 🎉</b></i></p>
+<p align="center"><i><b>🎉 Automate software refactoring 🎉</b></i></p>
 
 ---
 

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/ChangeMethodTargetToStaticBenchmark.java

@@ -21,6 +21,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.java.ChangeMethodTargetToStatic;
 
 import java.util.concurrent.TimeUnit;
@@ -44,6 +45,6 @@ public static void main(String[] args) throws RunnerException {
     @Benchmark
     public void changeMethodTargetToStatic(JavaCompilationUnitState state) {
         new ChangeMethodTargetToStatic("org.junit.Assert assertThat(..)", "org.hamcrest.MatcherAssert", null, null)
-                .run(state.getSourceFiles());
+                .run(state.getSourceSet(), new InMemoryExecutionContext());
     }
 }

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/ChangeTypeBenchmark.java

@@ -21,6 +21,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.java.ChangeType;
 
 import java.util.concurrent.TimeUnit;
@@ -44,6 +45,6 @@ public static void main(String[] args) throws RunnerException {
     @Benchmark
     public void changeType(JavaCompilationUnitState state) {
         new ChangeType("java.util.List", "java.util.Collection", null)
-                .run(state.getSourceFiles());
+                .run(state.getSourceSet(), new InMemoryExecutionContext());
     }
 }

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/FindTypesBenchmark.java

@@ -21,6 +21,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.java.search.FindTypes;
 
 import java.util.concurrent.TimeUnit;
@@ -43,6 +44,7 @@ public static void main(String[] args) throws RunnerException {
 
     @Benchmark
     public void findTypes(JavaCompilationUnitState state) {
-        new FindTypes("java.util.List", false).run(state.getSourceFiles());
+        new FindTypes("java.util.List", false)
+                .run(state.getSourceSet(), new InMemoryExecutionContext());
     }
 }

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/JavaCompilationUnitState.java

@@ -18,18 +18,21 @@
 import org.openjdk.jmh.annotations.*;
 import org.openjdk.jmh.infra.Blackhole;
 import org.openrewrite.InMemoryExecutionContext;
+import org.openrewrite.LargeSourceSet;
+import org.openrewrite.SourceFile;
+import org.openrewrite.internal.InMemoryLargeSourceSet;
 import org.openrewrite.java.JavaParser;
-import org.openrewrite.java.tree.J;
 
 import java.net.URISyntaxException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.Arrays;
 import java.util.List;
+import java.util.stream.Collectors;
 
 @State(Scope.Benchmark)
 public class JavaCompilationUnitState {
-    List<J.CompilationUnit> sourceFiles;
+    List<SourceFile> sourceFiles;
 
     @Setup(Level.Trial)
     public void setup() throws URISyntaxException {
@@ -71,15 +74,20 @@ public void setup() throws URISyntaxException {
                         "org.eclipse.jgit")
 //                .logCompilationWarningsAndErrors(true)
                 .build()
-                .parse(inputs, null, new InMemoryExecutionContext(Throwable::printStackTrace));
+                .parse(inputs, null, new InMemoryExecutionContext(Throwable::printStackTrace))
+                .collect(Collectors.toList());
     }
 
     @TearDown(Level.Trial)
     public void tearDown(Blackhole hole) {
         hole.consume(sourceFiles.size());
     }
 
-    public List<J.CompilationUnit> getSourceFiles() {
+    public LargeSourceSet getSourceSet() {
+        return new InMemoryLargeSourceSet(sourceFiles);
+    }
+
+    public List<SourceFile> getSourceFiles() {
         return sourceFiles;
     }
 }

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/OrderImportsBenchmark.java

@@ -21,6 +21,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.java.OrderImports;
 
 import java.util.concurrent.TimeUnit;
@@ -35,7 +36,8 @@ public class OrderImportsBenchmark {
 
     @Benchmark
     public void orderImports(JavaCompilationUnitState state) {
-        new OrderImports(false).run(state.getSourceFiles());
+        new OrderImports(false)
+                .run(state.getSourceSet(), new InMemoryExecutionContext());
     }
 
     public static void main(String[] args) throws RunnerException {

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/ParserInputBenchmark.java

@@ -24,7 +24,9 @@
 import org.openrewrite.Parser;
 import org.openrewrite.java.JavaParser;
 
-import java.io.*;
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
 import java.nio.file.Files;
 import java.util.concurrent.TimeUnit;
 

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/RemoveImportBenchmark.java

@@ -22,6 +22,7 @@
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
 import org.openrewrite.ExecutionContext;
+import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.Recipe;
 import org.openrewrite.TreeVisitor;
 import org.openrewrite.java.RemoveImport;
@@ -44,11 +45,16 @@ public String getDisplayName() {
                 return "Remove Function";
             }
 
+            @Override
+            public String getDescription() {
+                return "Remove function.";
+            }
+
             @Override
             public TreeVisitor<?, ExecutionContext> getVisitor() {
-                return new RemoveImport("java.util.function.Function");
+                return new RemoveImport<>("java.util.function.Function");
             }
-        }.run(state.getSourceFiles());
+        }.run(state.getSourceSet(), new InMemoryExecutionContext());
     }
 
     public static void main(String[] args) throws RunnerException {

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/TreeVisitorAdapterBenchmark.java

@@ -20,6 +20,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openrewrite.SourceFile;
 import org.openrewrite.Tree;
 import org.openrewrite.TreeVisitor;
 import org.openrewrite.internal.TreeVisitorAdapter;
@@ -38,7 +39,7 @@ public class TreeVisitorAdapterBenchmark {
 
     @Benchmark
     public void adaptToJava(JavaCompilationUnitState cus) {
-        for (J.CompilationUnit cu : cus.getSourceFiles()) {
+        for (SourceFile cu : cus.getSourceFiles()) {
             //noinspection unchecked
             TreeVisitorAdapter.adapt(new TreeVisitor<Tree, Integer>() {
                 @Override
@@ -51,7 +52,7 @@ public Tree preVisit(Tree tree, Integer p) {
 
     @Benchmark
     public void noAdaptation(JavaCompilationUnitState cus) {
-        for (J.CompilationUnit cu : cus.getSourceFiles()) {
+        for (SourceFile cu : cus.getSourceFiles()) {
             new JavaVisitor<>().visitNonNull(cu, 0);
         }
     }

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/UseStaticImportBenchmark.java

@@ -21,6 +21,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.java.UseStaticImport;
 
 import java.util.concurrent.TimeUnit;
@@ -35,7 +36,8 @@ public class UseStaticImportBenchmark {
 
     @Benchmark
     public void useStaticImport(JavaCompilationUnitState state) {
-        new UseStaticImport("java.util.List remove(..)").run(state.getSourceFiles());
+        new UseStaticImport("java.util.List remove(..)")
+                .run(state.getSourceSet(), new InMemoryExecutionContext());
     }
 
     public static void main(String[] args) throws RunnerException {

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/WeighingJavaAstsBenchmark.java

@@ -22,7 +22,7 @@
 import org.openjdk.jmh.runner.RunnerException;
 import org.openjdk.jmh.runner.options.Options;
 import org.openjdk.jmh.runner.options.OptionsBuilder;
-import org.openrewrite.java.tree.J;
+import org.openrewrite.SourceFile;
 
 import java.util.Collections;
 import java.util.IdentityHashMap;
@@ -41,7 +41,7 @@ public class WeighingJavaAstsBenchmark {
     public void bloomFilter(JavaCompilationUnitState state, Blackhole blackhole) {
         Bloom bloom = Bloom.construct(new long[0], 1);
         long weight = 0;
-        for (J.CompilationUnit sourceFile : state.getSourceFiles()) {
+        for (SourceFile sourceFile : state.getSourceFiles()) {
             weight += sourceFile.getWeight(t -> {
                 int id = System.identityHashCode(t);
                 if (bloom.mayContain(id)) {
@@ -58,7 +58,7 @@ public void bloomFilter(JavaCompilationUnitState state, Blackhole blackhole) {
     public void identitySet(JavaCompilationUnitState state, Blackhole blackhole) {
         Set<Object> uniqueTypes = Collections.newSetFromMap(new IdentityHashMap<>());
         long weight = 0;
-        for (J.CompilationUnit sourceFile : state.getSourceFiles()) {
+        for (SourceFile sourceFile : state.getSourceFiles()) {
             weight += sourceFile.getWeight(uniqueTypes::add);
         }
         blackhole.consume(weight);

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/java/package-info.java

@@ -13,9 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-@PolyglotNamespace("OpenRewrite")
 @NonNullApi
 package org.openrewrite.benchmarks.java;
 
-import org.openrewrite.PolyglotNamespace;
 import org.openrewrite.internal.lang.NonNullApi;

rewrite-benchmarks/src/jmh/java/org/openrewrite/benchmarks/maven/MavenParserBenchmark.java

@@ -16,6 +16,7 @@
 package org.openrewrite.benchmarks.maven;
 
 import org.openjdk.jmh.annotations.*;
+import org.openjdk.jmh.infra.Blackhole;
 import org.openjdk.jmh.profile.GCProfiler;
 import org.openjdk.jmh.runner.Runner;
 import org.openjdk.jmh.runner.RunnerException;
@@ -31,7 +32,9 @@
 import org.openrewrite.xml.tree.Xml;
 
 import java.nio.file.Paths;
+import java.util.Optional;
 import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 
 @Fork(1)
 @Measurement(iterations = 2)
@@ -54,11 +57,12 @@ public static void main(String[] args) throws RunnerException {
     }
 
     @Benchmark
-    public void parse() {
+    public void parse(Blackhole blackhole) {
         MavenExecutionContextView ctx = MavenExecutionContextView.view(new InMemoryExecutionContext());
         ctx.setPomCache(pomCache);
 
-        Xml.Document maven = MavenParser.builder().build().parse(ctx, "" +
+        Optional<Xml.Document> maven = MavenParser.builder().build().parse(ctx,
+                "" +
                 "<project>" +
                 "  <parent>" +
                 "    <groupId>org.springframework.boot</groupId>" +
@@ -80,9 +84,8 @@ public void parse() {
                 "    </dependency>" +
                 "  </dependencies>" +
                 "</project>"
-        ).get(0);
+        ).findFirst();
 
-//        System.out.println(maven.getMarkers().findFirst(MavenResolutionResult.class).get().getDependencies()
-//                .get(org.openrewrite.maven.tree.Scope.Compile).size());
+        blackhole.consume(maven);
     }
 }

rewrite-java/src/main/java/org/openrewrite/java/RewriteSkip.java → rewrite-core/src/main/java/org/openrewrite/Changeset.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2022 the original author or authors.
+ * Copyright 2023 the original author or authors.
  * <p>
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,15 +13,16 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.openrewrite.java;
+package org.openrewrite;
 
-import org.openrewrite.Recipe;
+import java.util.List;
 
-import java.lang.annotation.*;
+public interface Changeset {
+    int size();
 
-@Target(ElementType.PACKAGE)
-@Retention(RetentionPolicy.CLASS)
-public @interface RewriteSkip {
-    String[] recipes() default {};
-    Class<? extends Recipe>[] recipeClasses() default {};
+    List<Result> getPage(int start, int count);
+
+    default List<Result> getAllResults() {
+        return getPage(0, size());
+    }
 }