Move equo caches into build / build tool directory #1687

jochenberger · 2023-04-26T08:57:13Z

The move to equo for the Eclipse-based formatters causes issues in out CI build. See #1524 (comment)
The problem is, that the plugin tries to write the equo cache into a directory that's not writable/resolvable.
I suggest moving the cache into the ${user.home}/.m2/GRADLE_USER_HOME directories or into the project build directory.
See also #1669 (comment)

The text was updated successfully, but these errors were encountered:

jochenberger · 2023-04-26T09:40:30Z

The problem in my case is that System.getProperty("user.home") returns "?" in https://github.com/equodev/equo-ide/blob/main/solstice/src/main/java/dev/equo/solstice/p2/CacheLocations.java#L111.

nedtwigg · 2023-05-24T02:55:46Z

Fixed in plugin-gradle 6.19.0 and plugin-maven 2.37.0.

jochenberger · 2023-05-24T06:02:51Z

Unfortunately, this doesn't seem to have the desired effect. Now I'm getting

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':spotlessCheck'.
> Could not create task ':spotlessGroovyGradleCheck'.
   > Could not create task ':spotlessGroovyGradle'.
      > java.io.IOException: Failed to load eclipse groovy formatter: java.lang.IllegalArgumentException: Need to create directory first! /gradle-home/daemon/8.1.1/?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/queries

jochenberger · 2023-05-24T06:23:55Z

What if the Gradle/Maven plugin set CacheLocations.override_p2data appropriately instead of trying to figure out the current environment in solstice?

nedtwigg · 2023-05-24T06:24:08Z

Interesting! What system is this running on? I'm confused by the persistent ? thing.

Can you add --stacktrace? I believe the only place that this can be happening is downstream of here, where we clearly create the directory first.

https://github.com/equodev/equo-ide/blob/35986ecdc5e6dd6dd5a293c348bfe0330e49ab96/solstice/src/main/java/dev/equo/solstice/p2/QueryCacheOnDisk.java#L29-L35

Also, it is strange that System.getProperty("user.home") is returning /gradle-home/daemon/8.1.1/?/. Very strange... I tried to test for this exact case by creating the .m2 directory, seeing if there is an exception, and using that as a cue to read GRADLE_USER_HOME. I'm kinda surprised that we could create /gradle-home/daemon/8.1.1/?/.m2 ...

https://github.com/equodev/equo-ide/blob/35986ecdc5e6dd6dd5a293c348bfe0330e49ab96/solstice/src/main/java/dev/equo/solstice/p2/CacheLocations.java#L80-L92

jochenberger · 2023-05-24T06:33:15Z

What system is this running on?

It's running in an eclipse-temurin:17-jdk Docker container.

Can you add --stacktrace?

Sure:

...
Caused by: java.io.IOException: Failed to load eclipse groovy formatter: java.lang.IllegalArgumentException: Need to create directory first! /gradle-home/daemon/8.1.1/?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/queries
	at com.diffplug.spotless.extra.EquoBasedStepBuilder.get(EquoBasedStepBuilder.java:109)
	at com.diffplug.spotless.FormatterStepImpl.calculateState(FormatterStepImpl.java:58)
	at com.diffplug.spotless.LazyForwardingEquality.state(LazyForwardingEquality.java:56)
	... 224 more
Caused by: java.lang.IllegalArgumentException: Need to create directory first! /gradle-home/daemon/8.1.1/?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/queries
	at dev.equo.solstice.p2.FileMisc.writeToken(FileMisc.java:77)
	at dev.equo.solstice.p2.QueryCacheOnDisk.<init>(QueryCacheOnDisk.java:34)
	at dev.equo.solstice.p2.P2Model.query(P2Model.java:118)
	at com.diffplug.spotless.extra.EquoBasedStepBuilder.get(EquoBasedStepBuilder.java:107)
	... 226 more

Also, it is strange that System.getProperty("user.home") is returning /gradle-home/daemon/8.1.1/?/.

It isn't. It's returning ? (probably because PAM is not available, see https://stackoverflow.com/questions/1503284/java-system-getpropertyuser-home-returns).
Apparently, /gradle-home/daemon/8.1.1/?/ is what we get when trying to resolve it to an absolute directory, which we do to create the error message (https://github.com/equodev/equo-ide/blob/solstice/1.3.1/solstice/src/main/java/dev/equo/ide/FileMisc.java#L77). I think we're shadowing the real issue here.

jochenberger · 2023-05-24T07:16:43Z

I tried to test for this exact case by creating the .m2 directory, seeing if there is an exception, and using that as a cue to read GRADLE_USER_HOME

Shouldn't we rather do it the other way around? I.e. check for GRADLE_USER_HOME first? The variable should serve as a clear indication that we're running in a Gradle context and even point us to the correct location. I'm not sure whether there's something similar for Maven though. ~/.m2 just seems to be the default.

jochenberger · 2023-05-24T11:07:15Z

Maybe GRADLE_USER_HOME isn't passed down to Solstice at all? 🤔
I think we should try to set CacheLocations.override_p2data from the plugin code.

jochenberger · 2023-05-24T11:49:56Z

Apparently, /gradle-home/daemon/8.1.1/?/ is what we get when trying to resolve it to an absolute directory

Yes, I think that's the problem. ? is resolved to an absolute directory depending on the working directory. And when initializing CacheLocations, it is the Gradle project's working directory. That one is writable, so we create $PROJECT_DIR/?/.m2/repository....
Later, when the EquoBasedStepBuilder runs, for some reason, the working directory is /gradle-home/daemon/8.1.1/, where the directory is obviously missing.
So, if new File(System.getProperty("user.home")) is not absolute, we should probably not use it.

I think that both suggestions above should fix that problem and I prefer the latter one.

jochenberger · 2023-05-24T12:08:14Z

I tried adding

dev.equo.solstice.p2.CacheLocations.override_p2data = gradle.gradleUserHomeDir.toPath().resolve("caches/p2-data").toFile();`

to by build. That fixes the exception, however, now I get

...
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: ?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/nested-jars/org.codehaus.groovy_4.0.11.v202303311542-e2303.jar__eclipse-trace.jar__Z5ch70p+Jsgu4XHQc0MVqw--.jar (No such file or directory)
	at dev.equo.solstice.Unchecked.wrap(Unchecked.java:47)
	at dev.equo.solstice.NestedJars.extractNestedJar(NestedJars.java:323)
	at dev.equo.solstice.NestedJars.extractAllNestedJars(NestedJars.java:291)
	at dev.equo.solstice.NestedJars$OnClassPath.confirmAllNestedJarsArePresentOnClasspath(NestedJars.java:194)
	at com.diffplug.spotless.extra.glue.groovy.GrEclipseFormatterStepImpl.<clinit>(GrEclipseFormatterStepImpl.java:55)
	... 142 more
Caused by: java.io.FileNotFoundException: ?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/nested-jars/org.codehaus.groovy_4.0.11.v202303311542-e2303.jar__eclipse-trace.jar__Z5ch70p+Jsgu4XHQc0MVqw--.jar (No such file or directory)
	at java.base/java.io.FileOutputStream.open0(Native Method)
	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:184)
	at dev.equo.solstice.NestedJars.extractNestedJar(NestedJars.java:317)
	... 145 more

jochenberger · 2023-05-24T12:28:23Z

Oh, I guess that's a forked process that has its own CacheLocations class and its own working directory. That makes a lot of sense.
I wonder if it inherits the GRADLE_USER_HOME env variable.

nedtwigg · 2023-05-25T06:35:47Z

I just took another hack at it

Fix cache location when we can't determine the home directory. equodev/equo-ide#132

published in solstice 1.3.2. It'll be at least a few weeks before a new Spotless comes that uses these.

jochenberger · 2023-05-25T07:16:22Z

Looks promising, thanks!

jochenberger · 2023-05-25T07:58:11Z

@nedtwigg, still not working. I added a version override to my build, but I get the same error as in #1687 (comment):

$ GRADLE_USER_HOME=/tmp ./gradlew --rerun-tasks -Duser.home=? spotlessGroovy

...
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: ?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/nested-jars/org.codehaus.groovy_4.0.11.v202303311542-e2303.jar__eclipse-trace.jar__Z5ch70p+Jsgu4XHQc0MVqw--.jar (Datei oder Verzeichnis nicht gefunden)
        at dev.equo.solstice.Unchecked.wrap(Unchecked.java:47)
        at dev.equo.solstice.NestedJars.extractNestedJar(NestedJars.java:323)
        at dev.equo.solstice.NestedJars.extractAllNestedJars(NestedJars.java:291)
        at dev.equo.solstice.NestedJars$OnClassPath.confirmAllNestedJarsArePresentOnClasspath(NestedJars.java:194)
        at com.diffplug.spotless.extra.glue.groovy.GrEclipseFormatterStepImpl.<clinit>(GrEclipseFormatterStepImpl.java:55)
        ... 134 more
Caused by: java.io.FileNotFoundException: ?/.m2/repository/dev/equo/p2-data/repository/dev/equo/p2-data/nested-jars/org.codehaus.groovy_4.0.11.v202303311542-e2303.jar__eclipse-trace.jar__Z5ch70p+Jsgu4XHQc0MVqw--.jar (Datei oder Verzeichnis nicht gefunden)
        at dev.equo.solstice.NestedJars.extractNestedJar(NestedJars.java:317)
        ... 137 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':spotlessGroovy'.
> java.lang.ExceptionInInitializerError (no error message)

Not sure what's happening. It looks as if NestedJars uses a different solstice version. So the override only works partially?

I still think it would be best if the Gradle plugin set the CacheLocations overrides since it knows where its user home is.

blacelle · 2023-06-24T16:36:27Z

In mvn world, the local reposirory can be configured through -Dmaven.local.repo=XXX (https://stackoverflow.com/a/57223696/1439964). If this is customized, I feel EquoDev would write in the default local repository instead of the custom one.

I also have related issues by deploying Spotless to AWS Lambda, as user.home resolves to something inexistent+read-only (https://stackoverflow.com/questions/61614561/aws-lambda-java-runtime-home-directory-does-not-exist).

attiand · 2023-12-19T07:21:46Z

I have the same problem. In a CI environment building in a docker container I would like to set the Maven repository directory to point inside the Jenkins workspace.

The problem is simple to reproduce. I have a simple groovy/greclipse spotless setup using version 2.41.1.

Make the ~/.m2 directory unwritable: chmod 000 ~/.m2/

;mvn spotless:apply
[ERROR] Could not create local repository at /home/maan0496/.m2/repository -> [Help 1]

I could work around the problem with (although it seems strange to a Maven user):

GRADLE_USER_HOME=/tmp/gradle mvn -Duser.home=/tmp/user-home spotless:apply

I would like spotless to honor the maven.repo.local property, so I could write:

mvn -Dmaven.repo.local=/tmp/repository spotless:apply

nedtwigg added the enhancement label May 18, 2023

nedtwigg mentioned this issue May 23, 2023

Improve cache location for p2 data equodev/equo-ide#131

Merged

nedtwigg closed this as completed May 24, 2023

nedtwigg mentioned this issue May 25, 2023

Fix cache location when we can't determine the home directory. equodev/equo-ide#132

Merged

nedtwigg reopened this May 25, 2023

blacelle mentioned this issue Jun 24, 2023

Support for concurrent operations - P2 operation already in progress equodev/equo-ide#146

Open

nedtwigg mentioned this issue Dec 4, 2023

GrEclipseFormatterStep fails when ~/.m2/ is not writable #1797

Closed

nedtwigg mentioned this issue Jan 15, 2024

Embed p2 lockfiles for Eclipse JDT #1996

Open

kevinleturc mentioned this issue Aug 28, 2024

Leverage Maven local repository for p2 cache directory in Eclipse step #2238

Merged

nedtwigg mentioned this issue Nov 5, 2024

Eclipse formatters on 2.43.0 does not use correctly maven local repository #2329

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move equo caches into build / build tool directory #1687

Move equo caches into build / build tool directory #1687

jochenberger commented Apr 26, 2023

jochenberger commented Apr 26, 2023 •

edited

Loading

nedtwigg commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

nedtwigg commented May 24, 2023

jochenberger commented May 24, 2023 •

edited

Loading

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

nedtwigg commented May 25, 2023

jochenberger commented May 25, 2023

jochenberger commented May 25, 2023 •

edited

Loading

blacelle commented Jun 24, 2023

attiand commented Dec 19, 2023

Move equo caches into build / build tool directory #1687

Move equo caches into build / build tool directory #1687

Comments

jochenberger commented Apr 26, 2023

jochenberger commented Apr 26, 2023 • edited Loading

nedtwigg commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

nedtwigg commented May 24, 2023

jochenberger commented May 24, 2023 • edited Loading

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

jochenberger commented May 24, 2023

nedtwigg commented May 25, 2023

jochenberger commented May 25, 2023

jochenberger commented May 25, 2023 • edited Loading

blacelle commented Jun 24, 2023

attiand commented Dec 19, 2023

jochenberger commented Apr 26, 2023 •

edited

Loading

jochenberger commented May 24, 2023 •

edited

Loading

jochenberger commented May 25, 2023 •

edited

Loading