P2Tests fail since separation of batch compiler plugin from jdt core #419

iloveeclipse · 2022-12-02T12:44:47Z

See for example https://download.eclipse.org/eclipse/downloads/drops4/I20221201-1800/testresults/html/org.eclipse.pde.build.tests_ep427I-unit-cen64-gtk3-java11_linux.gtk.x86_64_11.html

org.eclipse.jdt.core.JDTCompilerAdapter is now in separated bundle (after eclipse-jdt/eclipse.jdt.core#181), and some platform code somewhere used jdt.core bundle as ant dependency.
Now the bundle is there, but outside of OSGI world it simply can't find batch compiler classes anymore, and ant rightly complains that the Class org.eclipse.jdt.core.JDTCompilerAdapter could not be loaded because of an invalid dependency.

If I only knew where org.eclipse.jdt.core library was set to be in the classpath of ant!
It should be now replaced through org.eclipse.jdt.core.compiler.batch and the problem will be solved.

Stack:

The following error occurred while executing this line: /home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/build.xml:108: The following error occurred while executing this line: 
/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/templates/headless-build/customTargets.xml:13: 
The following error occurred while executing this line: 
/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/templates/headless-build/allElements.xml:17: The following error occurred while executing this line: 
/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/genericTargets.xml:119: 
The following error occurred while executing this line: /home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/genericTargets.xml:124: The following error occurred while executing this line: 
/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/features/F/build.xml:43: 
The following error occurred while executing this line: /home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/features/F/build.xml:14: 
The following error occurred while executing this line: 
/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/plugins/c/build.xml:124: 
The following error occurred while executing this line: 
/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/plugins/c/build.xml:86: 
Class org.eclipse.jdt.core.JDTCompilerAdapter could not be loaded because of an invalid dependency./home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/build.xml:39: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/build.xml:108: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/templates/headless-build/customTargets.xml:13: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/templates/headless-build/allElements.xml:17: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/genericTargets.xml:119: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.pde.build_3.11.400.v20220923-1253/scripts/genericTargets.xml:124: The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/features/F/build.xml:43: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/features/F/build.xml:14: The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/plugins/c/build.xml:124: 
The following error occurred while executing this line:/home/jenkins/agent/workspace/AutomatedTests/ep427I-unit-cen64-gtk3-java11/workarea/I20221201-1800/eclipse-testing/test-eclipse/eclipse/pdebuild_folder/pde.build/263272/plugins/c/build.xml:86: 
Class org.eclipse.jdt.core.JDTCompilerAdapter could not be loaded because of an invalid dependency.
at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:582)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:440)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:106)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:299)
at jdk.internal.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
at org.apache.tools.ant.Task.perform(Task.java:350)at org.apache.tools.ant.Target.execute(Target.java:449)
at org.apache.tools.ant.Target.performTasks(Target.java:470)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1401)
at org.apache.tools.ant.Project.executeTarget(Project.java:1374)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.eclipse.ant.internal.core.ant.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:34)
at org.apache.tools.ant.Project.executeTargets(Project.java:1264)
at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:716)
at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:573)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)at org.eclipse.ant.core.AntRunner.run(AntRunner.java:506)
at org.eclipse.pde.build.tests.PDETestCase.runAntScript(PDETestCase.java:158)
at org.eclipse.pde.build.tests.PDETestCase.runAntScript(PDETestCase.java:150)
at org.eclipse.pde.build.tests.PDETestCase.runBuild(PDETestCase.java:128)
at org.eclipse.pde.build.internal.tests.p2.P2Tests.testBug263272(P2Tests.java:677)

Most important part of the stack

Caused by: java.lang.NoClassDefFoundError: org/apache/tools/ant/taskdefs/compilers/DefaultCompilerAdapter
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.defineClass(ModuleClassLoader.java:283)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.defineClass(ClasspathManager.java:716)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findClassImpl(ClasspathManager.java:639)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClassImpl(ClasspathManager.java:607)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClassImpl(ClasspathManager.java:587)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:566)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:335)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:397)
at org.eclipse.osgi.internal.loader.sources.SingleSourcePackage.loadClass(SingleSourcePackage.java:41)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:496)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:622)
at org.eclipse.ant.core.AntCorePreferences$WrappedClassLoader.findClass(AntCorePreferences.java:94)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.eclipse.ant.internal.core.AntClassLoader.loadClassPlugins(AntClassLoader.java:69)
at org.eclipse.ant.internal.core.AntClassLoader.findClass(AntClassLoader.java:47)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1421)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1108)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.tools.ant.util.ClasspathUtils.newInstance(ClasspathUtils.java:253)
... 217 more
Caused by: java.lang.ClassNotFoundException: org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter cannot be found by org.eclipse.jdt.core.compiler.batch_3.33.0.v20221205-1735
at org.eclipse.osgi.internal.loader.BundleLoader.generateException(BundleLoader.java:541)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:536)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 246 more

The text was updated successfully, but these errors were encountered:

iloveeclipse · 2022-12-02T12:49:35Z

@vik-chand : do you know where in the platform code (PDE?/Ant?) we specify or resolve ant classpath and add org.eclipse.jdt.core library?

iloveeclipse · 2022-12-02T12:58:14Z

OK, that was not that easy but I believe I've found it: org.eclipse.jdt.core bundle embedded jdtCompilerAdapter.jar that was extracted by PDE from the jar and placed to ant runtime, see org.eclipse.ant.internal.ui.datatransfer.BuildFileCreator...

Now, the jar simply misses classes moved to batch compiler bundle.

iloveeclipse · 2022-12-02T13:44:08Z

State before eclipse-jdt/eclipse.jdt.core#182:

Current state:

…t#181) Add moved classes located now in org.eclipse.jdt.core.compiler.batch to the jdtCompilerAdapter.jar used by pde.build & Ant (see org.eclipse.ant.internal.ui.datatransfer.BuildFileCreator). See eclipse-pde/eclipse.pde#419 eclipse-jdt#181

Add moved classes located now in org.eclipse.jdt.core.compiler.batch to the jdtCompilerAdapter.jar used by pde.build & Ant (see org.eclipse.ant.internal.ui.datatransfer.BuildFileCreator). See eclipse-pde/eclipse.pde#419 #181

…ipse-jdt#181) Add moved classes located now in org.eclipse.jdt.core.compiler.batch to the jdtCompilerAdapter.jar used by pde.build & Ant (see org.eclipse.ant.internal.ui.datatransfer.BuildFileCreator). See eclipse-pde/eclipse.pde#419 eclipse-jdt#181

Add moved classes located now in org.eclipse.jdt.core.compiler.batch to the jdtCompilerAdapter.jar used by pde.build & Ant (see org.eclipse.ant.internal.ui.datatransfer.BuildFileCreator). See eclipse-pde/eclipse.pde#419 #181

iloveeclipse · 2022-12-02T20:13:45Z

I hope with eclipse-jdt/eclipse.jdt.core#579 the problem should be fixed.
Will see in next SDK build...

iloveeclipse · 2022-12-05T15:54:26Z

I hope with eclipse-jdt/eclipse.jdt.core#579 the problem should be fixed.

Nope, same error: https://download.eclipse.org/eclipse/downloads/drops4/I20221205-0710/testresults/html/org.eclipse.pde.build.tests_ep427I-unit-macM1-java17_macosx.cocoa.aarch64_17.html

Hmm. I can't see what should be different now, because I don't know how ant/PDE uses ecj.

merks · 2022-12-05T16:04:07Z

It's little odd. It says org.eclipse.jdt.core.JDTCompilerAdapter could not be loaded because of an invalid dependency and also says org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter cannot be found by org.eclipse.jdt.core.compiler.batch_3.33.0.v20221202-1820 but I wouldn't expect the latter to be trying to find (load the class for) the former...

iloveeclipse · 2022-12-05T16:52:19Z

and also says org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter cannot be found by org.eclipse.jdt.core.compiler.batch_3.33.0.v20221202-1820

Arrgh, Ed, I was blind, I didn't expected that there is more than "just" missing JDTCompilerAdapter!

The code in org.eclipse.ant.internal.core.AntClassLoader.loadClassPlugins(String) looks over contributed org.eclipse.ant.core.AntCorePreferences.WrappedClassLoader instances.
I assume these don't know that batch compiler should be re-exported by jdt.core or something like this.
So trivial patch would be to contribute something to ant.core from batch compiler. Non trivial patch would fix classloading :-)

iloveeclipse · 2022-12-06T10:15:11Z

So trivial patch would be to contribute something to ant.core from batch compiler. Non trivial patch would fix classloading :-)

Neither one is trivial. There must be yet another hack or hardcoded assumption somewhere.

Most important part of the stack

Caused by: java.lang.NoClassDefFoundError: org/apache/tools/ant/taskdefs/compilers/DefaultCompilerAdapter
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.defineClass(ModuleClassLoader.java:283)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.defineClass(ClasspathManager.java:716)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findClassImpl(ClasspathManager.java:639)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClassImpl(ClasspathManager.java:607)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClassImpl(ClasspathManager.java:587)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:566)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:335)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:397)
at org.eclipse.osgi.internal.loader.sources.SingleSourcePackage.loadClass(SingleSourcePackage.java:41)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:496)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:622)
at org.eclipse.ant.core.AntCorePreferences$WrappedClassLoader.findClass(AntCorePreferences.java:94)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.eclipse.ant.internal.core.AntClassLoader.loadClassPlugins(AntClassLoader.java:69)
at org.eclipse.ant.internal.core.AntClassLoader.findClass(AntClassLoader.java:47)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1421)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1108)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.tools.ant.util.ClasspathUtils.newInstance(ClasspathUtils.java:253)
... 217 more
Caused by: java.lang.ClassNotFoundException: org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter cannot be found by org.eclipse.jdt.core.compiler.batch_3.33.0.v20221205-1735
at org.eclipse.osgi.internal.loader.BundleLoader.generateException(BundleLoader.java:541)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:536)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 246 more

If I contribute batch compiler as "extraClasspathEntry", I still see same error.

   <extension
         point="org.eclipse.ant.core.extraClasspathEntries">
      <extraClasspathEntry
            eclipseRuntime="false"
            headless="true"
            library="/">
      </extraClasspathEntry>
   </extension>

If I add ant library as optional dependency to batch compiler, I can load DefaultCompilerAdapter from JDTCompilerAdapter but in next step ant throws exception that the super interfaces are different - because they were loaded from different class loaders.

So JDT or PDE or ANT did before something to load batch compiler from same ant classloader that loaded DefaultCompilerAdapter. I don't see how they did that yet. :-(

laeubi · 2022-12-06T10:48:19Z

What sounds strange is that "org.eclipse.jdt.core.compiler.batch" is mentioned in the exception but you said it does not depends on ant (or does it?). I think I have seen some ant jar mangling stuff.

Have you tried using

DynamicImport-Package: org.apache.tools.ant.*, ... probably more ant stuff that is used by any ant.jar ...

SO probably you could remove the ant.jar from the classpath, then see what packages are required and add all of those as optional/dynamic imports...

iloveeclipse · 2022-12-06T10:57:46Z

The problem is not missing ant but wrong classloader.
The hack was to NOT load class by jdt core

laeubi · 2022-12-06T11:04:55Z

The problem is not missing ant but wrong classloader. The hack was to NOT load class by jdt core

So it does work now? Because what I wanted to say is that if you only import one package but others are also required they might be loaded from somwhere else and producing issues.

iloveeclipse · 2022-12-06T13:25:50Z

So it does work now?

Yes. I'm just busy with other work. I have to write some documentation wrap up before committing how this magic works so next time someone wants to debug a problem it would be easier to find who does what and why and why not in usual way.

laeubi · 2022-12-06T13:40:02Z

Great! That is really good to hear.

See README.md why the JDTCompilerAdapter classloading is not trivial. TL;DR: OSGI classloader from org.eclipse.jdt.core bundle should not find JDTCompilerAdapter class (!) so the ant started from Eclipse can find it inside jdtCompilerAdapter.jar contributed by ... org.eclipse.jdt.core bundle. Fixes eclipse-pde/eclipse.pde#419

iloveeclipse · 2022-12-06T15:13:51Z

See https://github.com/eclipse-jdt/eclipse.jdt.core/blob/d294d9918c9d3118b974f37c12f0cf9abe5aa447/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/core/README.md for my findings.

See README.md why the JDTCompilerAdapter classloading is not trivial. TL;DR: OSGI classloader from org.eclipse.jdt.core bundle should not find JDTCompilerAdapter class (!) so the ant started from Eclipse can find it inside jdtCompilerAdapter.jar contributed by ... org.eclipse.jdt.core bundle. Fixes eclipse-pde/eclipse.pde#419

iloveeclipse · 2022-12-07T07:32:58Z

Fixed via eclipse-jdt/eclipse.jdt.core#592, verified in https://download.eclipse.org/eclipse/downloads/drops4/I20221206-1800/