Immutable workspace contents have been modified with current gradle (8.6) and this action

[mikehardy]

Hi there!

We've started getting build failures with this message since updating to gradle 8.6 while using this action:

Execution failed for task ':AnkiDroid:testPlayDebugUnitTest'.
> A build operation failed.
      Immutable workspace contents have been modified: /home/runner/.gradle/caches/transforms-4/f46cf63d0a549c8bc814e25133f449eb. These workspace directories are not supposed to be modified once they are created. Deleting the directory in question can allow the content to be recreated.
   > Immutable workspace contents have been modified: /home/runner/.gradle/caches/transforms-4/f46cf63d0a549c8bc814e25133f449eb. These workspace directories are not supposed to be modified once they are created. Deleting the directory in question can allow the content to be recreated.

Here's a sample run: https://github.com/ankidroid/Anki-Android/actions/runs/7932349193/job/21658645926?pr=15520

It reproduced originally in windows runners, and on a hunch it was windows file locking I disabled caching there and it worked

Now it is reproducing on ubuntu runners as well so I suppose it is an incompatibility?

Unfortunately it is persistent across runs - once whatever the incompatibility is triggers, the cache is saved with the problem latent in it, and any future runs that restore that gradle cache will fail with the error above.

This never happened prior to updating our gradle from 8.5 to 8.6 so perhaps that is the source of the incompatibility?

[bigdaz]

There are significant changes to the way artifact transforms are used with Gradle 8.6. The transforms-4 directory is new, and Gradle is now using artifact transforms in order to perform instrumentation of jar files.

I've never seen the message Immutable workspace contents have been modified previously. But one possibility is that the Gradle User Home cache cleanup, which modifies timestamps on directories to force Gradle to cleanup files, could be triggering this check.

Can you please disable cache cleanup, reset all cache entries and see if the problem reappears?
If that doesn't work, then it would be great to compare the contents of the directory in question when it fails and when it succeeds.

[bigdaz]

Looking at the code in question, it's not obvious how setting timestamps will be causing the problem. So it would be really useful to compare the exact contents of the directory in question (eg caches/transforms-4/f46cf63d0a549c8bc814e25133f449eb) in the case of failure, and then after deleting the Gradle User Home and running the build successfully.

[mikehardy]

Okay - thanks for looking in to this

I'm usually pretty handy with workflow yaml, I'll try to make a test jig for this

1- I'll add an upload-artifact step to save caches/transforms-4 so it is inspectable and make it so that step runs even on workflow failure

Then I'll think about it to see how I can get some combination of workflow dispatch arguments (for manual runs) that allow cache restore / cleanup / save behavior to be manipulated on successive runs so it can hopefully be triggered and cleaned and then the uploaded artifacts examined to hopefully shed light on it

[bigdaz]

@mikehardy Were you able to reproduce this issue? If not, I guess we'll just close it and assume it was anomalous.

[mikehardy]

Oh yes we had to disable caching as it doesn't just fail it poisons the cache and fails all future runs when it happens. The Gradle action is broken with Gradle 8.5 unless work has been specifically done here in this repo. I just haven't had time to pack up a repro to demonstrate

[bigdaz]

By "disable caching" do you mean --no-configuration-cache, --no-build-cache or cache-disabled: true?

Yes please if you are able to reproduce and help us identify the difference in /home/runner/.gradle/caches/transforms-4/ that triggers the exception, that would be fantastic.

We haven't had other reports of the same issue with Gradle 8.6. Without a reproducer or more evidence it's difficult to get the Build Tool team responsible to investigate further.

[david-allison]

Error logs can be found on the following actions runs & should be reproducible via GitHub Actions on the following commits:

• https://github.com/ankidroid/Anki-Android/actions/runs/8041535145 (at ankidroid/Anki-Android@0ab90ba)
• https://github.com/ankidroid/Anki-Android/actions/runs/7919143740 (at ankidroid/Anki-Android@9310987)

Caches have been disabled using cache-disabled: true

Commits disabling the cache:

• ankidroid/Anki-Android@6768519
• ankidroid/Anki-Android@5ad2157

---

GitHub Actions workflow (at ankidroid/Anki-Android@0ab90ba) with failures on macos/ubuntu. Cache disabled on Windows

https://github.com/ankidroid/Anki-Android/blob/0ab90ba8c1def105b512b507800c823d85bc9af4/.github/workflows/tests_unit.yml

[bigdaz]

Thanks @david-allison . Using a fork of the repository, I can execute the workflow multiple times with bigdaz/Anki-Android@0ab90ba and have not been able to reproduce the failure.

I'll continue to investigate, but any hints as to a reliable process to reproduce would be helpful.

[david-allison]

I'm not going to have the capacity for significant investigation.

The following lines may be relevant.

ankidroid/Anki-Android@8174898 -> ankidroid/Anki-Android@0ab90ba

The ubuntu check @8174898 passed, but saving to the cache had an error:

https://github.com/ankidroid/Anki-Android/actions/runs/8041286323/job/21960377455

2024-02-25T22:57:34.9317172Z Caching dependencies with path '/home/runner/.gradle/caches/modules-*/files-*/*/*/*/*' and cache key: dependencies-d73166a878f35edb0236287d9f5ce202
2024-02-25T22:57:35.3282511Z [command]/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/Anki-Android/Anki-Android --files-from manifest.txt --use-compress-program zstdmt
2024-02-25T22:57:37.7074249Z Failed to save cache entry with path '/home/runner/.gradle/caches/modules-*/files-*/*/*/*/*' and key: dependencies-d73166a878f35edb0236287d9f5ce202: ReserveCacheError: Unable to reserve cache with key dependencies-d73166a878f35edb0236287d9f5ce202, another job may be creating this cache. More details: Cache already exists. Scope: refs/heads/main, Key: dependencies-d73166a878f35edb0236287d9f5ce202, Version: 1c52a3a7b4256dbf4cb0b91ebdf0637b7fca6c8cf427ab687855671174f38301
2024-02-25T22:57:37.9759118Z Caching Gradle User Home with cache key: v9-gradle|Linux|unit[5988ad96f1286ffe5bfe74fc2ac093ce]-81748980632ce2fd12316a9cba83cfd57ef069b9
2024-02-25T22:57:37.9849462Z [command]/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/Anki-Android/Anki-Android --files-from manifest.txt --use-compress-program zstdmt

[bigdaz]

Thanks. I don't think that error is relevant.
I'm sequentially running actions for main commits leading up to the failure. Let's see if the issue is reproducible in that way, or if it's something less deterministic.

[lptr]

In Gradle 8.7-rc-4 we'll release this that is going to allow builds to proceed even when Gradle detects modifications:

• Improve handling of inconsistent immutable workspaces gradle#28503

[bigdaz]

I'm going to close this issue now that Gradle 8.7 has been released with a fix.
Please let us know if you continue to experience this problem with Gradle 8.7.