[Bug] Scalafmt::format fails to download scalafmt artifacts

[guizmaii]

Hi 🙂,

In Caliban (https://github.com/ghostdogpr/caliban), we generate some code and then we scalafmt it thanks to this piece of code:

  def format(strs: List[(String, String)], fmtPath: Option[String]): RIO[Blocking, List[(String, String)]] =
    effectBlocking {
      val config: Path = {
        @inline def defaultConfigPath = Paths.get(".scalafmt.conf")
        @inline def defaultConfig     = if (Files.exists(defaultConfigPath)) defaultConfigPath else Paths.get("")

        fmtPath.fold(defaultConfig)(Paths.get(_))
      }

      val scalafmt =
        Scalafmt
          .create(this.getClass.getClassLoader)
          .withRespectVersion(false)
          .withDefaultVersion("2.7.5") // For retro-compatibility

      val result = strs.map { case (name, code) => name -> scalafmt.format(config, Paths.get(s"$name.scala"), code) }
      scalafmt.clear()
      result
    }

(Comes from: https://github.com/ghostdogpr/caliban/blob/master/tools/src/main/scala/caliban/tools/Formatter.scala)

This piece of code, most of the time, fails if scalafmt hasn't been already downloaded on the computer executing it, with errors like this one:

error: .scalafmt.conf: org.scalafmt.dynamic.exceptions.ScalafmtException: failed to download v=2.7.5
Caused by: coursierapi.error.DownloadingArtifactsError: https://repo1.maven.org/maven2/org/scalameta/scalafmt-dynamic_2.13/2.7.5/scalafmt-dynamic_2.13-2.7.5.jar: checksum not found: /home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalameta/scalafmt-dynamic_2.13/2.7.5/scalafmt-dynamic_2.13-2.7.5.jar

	at coursierapi.error.DownloadingArtifactsError.of(DownloadingArtifactsError.java:23)
	at coursierapi.shaded.coursier.internal.api.ApiHelper$.doFetch(ApiHelper.scala:346)
	at coursierapi.shaded.coursier.internal.api.ApiHelper.doFetch(ApiHelper.scala)
	at coursierapi.Fetch.fetchResult(Fetch.java:244)
	at coursierapi.Fetch.fetch(Fetch.java:239)
	at org.scalafmt.dynamic.ScalafmtDynamicDownloader.$anonfun$download$1(ScalafmtDynamicDownloader.scala:36)
	at scala.util.Try$.apply(Try.scala:210)
	at org.scalafmt.dynamic.ScalafmtDynamicDownloader.download(ScalafmtDynamicDownloader.scala:22)
	at org.scalafmt.dynamic.ScalafmtDynamic.$anonfun$resolveFormatter$1(ScalafmtDynamic.scala:162)
	at scala.util.Try$.apply(Try.scala:210)
	at org.scalafmt.dynamic.utils.ReentrantCache.getOrAddToCache(ReentrantCache.scala:41)
	at org.scalafmt.dynamic.ScalafmtDynamic.resolveFormatter(ScalafmtDynamic.scala:159)
	at org.scalafmt.dynamic.ScalafmtDynamic.$anonfun$resolveConfigWithScalafmt$1(ScalafmtDynamic.scala:140)
	at scala.util.Either.flatMap(Either.scala:352)
	at org.scalafmt.dynamic.ScalafmtDynamic.resolveConfigWithScalafmt(ScalafmtDynamic.scala:135)
	at org.scalafmt.dynamic.ScalafmtDynamic.$anonfun$resolveConfig$4(ScalafmtDynamic.scala:121)
	at scala.util.Try$.apply(Try.scala:210)
	at org.scalafmt.dynamic.utils.ReentrantCache.getOrAddToCache(ReentrantCache.scala:41)
	at org.scalafmt.dynamic.ScalafmtDynamic.resolveConfig(ScalafmtDynamic.scala:120)
	at org.scalafmt.dynamic.ScalafmtDynamic.formatDetailed(ScalafmtDynamic.scala:104)
	at org.scalafmt.dynamic.ScalafmtDynamic.format(ScalafmtDynamic.scala:74)
	at caliban.tools.Formatter$.$anonfun$format$5(Formatter.scala:29)
	at scala.collection.immutable.List.map(List.scala:246)
	at caliban.tools.Formatter$.$anonfun$format$2(Formatter.scala:29)
	at zio.internal.FiberContext.evaluateNow(FiberContext.scala:490)
	at zio.internal.FiberContext.$anonfun$evaluateLater$1(FiberContext.scala:776)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
error: .scalafmt.conf: org.scalafmt.dynamic.exceptions.ScalafmtException: failed to download v=2.7.5
Caused by: coursierapi.error.DownloadingArtifactsError: https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar: checksum not found: /home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar

	at coursierapi.error.DownloadingArtifactsError.of(DownloadingArtifactsError.java:23)
	at coursierapi.shaded.coursier.internal.api.ApiHelper$.doFetch(ApiHelper.scala:346)
	at coursierapi.shaded.coursier.internal.api.ApiHelper.doFetch(ApiHelper.scala)
	at coursierapi.Fetch.fetchResult(Fetch.java:244)
	at coursierapi.Fetch.fetch(Fetch.java:239)
	at org.scalafmt.dynamic.ScalafmtDynamicDownloader.$anonfun$download$1(ScalafmtDynamicDownloader.scala:36)
	at scala.util.Try$.apply(Try.scala:210)
	at org.scalafmt.dynamic.ScalafmtDynamicDownloader.download(ScalafmtDynamicDownloader.scala:22)
	at org.scalafmt.dynamic.ScalafmtDynamic.$anonfun$resolveFormatter$1(ScalafmtDynamic.scala:162)
	at scala.util.Try$.apply(Try.scala:210)
	at org.scalafmt.dynamic.utils.ReentrantCache.getOrAddToCache(ReentrantCache.scala:41)
	at org.scalafmt.dynamic.ScalafmtDynamic.resolveFormatter(ScalafmtDynamic.scala:159)
	at org.scalafmt.dynamic.ScalafmtDynamic.$anonfun$resolveConfigWithScalafmt$1(ScalafmtDynamic.scala:140)
	at scala.util.Either.flatMap(Either.scala:352)
	at org.scalafmt.dynamic.ScalafmtDynamic.resolveConfigWithScalafmt(ScalafmtDynamic.scala:135)
	at org.scalafmt.dynamic.ScalafmtDynamic.$anonfun$resolveConfig$4(ScalafmtDynamic.scala:121)
	at scala.util.Try$.apply(Try.scala:210)
	at org.scalafmt.dynamic.utils.ReentrantCache.getOrAddToCache(ReentrantCache.scala:41)
	at org.scalafmt.dynamic.ScalafmtDynamic.resolveConfig(ScalafmtDynamic.scala:120)
	at org.scalafmt.dynamic.ScalafmtDynamic.formatDetailed(ScalafmtDynamic.scala:104)
	at org.scalafmt.dynamic.ScalafmtDynamic.format(ScalafmtDynamic.scala:74)
	at caliban.tools.Formatter$.$anonfun$format$5(Formatter.scala:29)
	at scala.collection.immutable.List.map(List.scala:246)
	at caliban.tools.Formatter$.$anonfun$format$2(Formatter.scala:29)
	at zio.internal.FiberContext.evaluateNow(FiberContext.scala:490)
	at zio.internal.FiberContext.$anonfun$evaluateLater$1(FiberContext.scala:776)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
error: .scalafmt.conf: org.scalafmt.dynamic.exceptions.ScalafmtException: failed to download v=2.7.5
Caused by: coursierapi.error.DownloadingArtifactsError: https://repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar: download error: Caught java.nio.file.NoSuchFileException: /home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar.part -> /home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar (/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar.part -> /home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar) while downloading https://repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar

In this execution, it failed to download scopt_2.13-3.7.1.jar but it's not always this lib that fails.

It's pretty much non-deterministic. Sometimes it works, sometimes it fails.
You can observe this behaviour in these Github Actions: https://github.com/guizmaii/poc_compile_time_caliban_client_generation/actions

This repo contains a project I use to develop a new sbt plugin for Caliban that generates Caliban client code at compile time from the Caliban server code: when you configure this plugin and then compile your project, the Caliban client code will be generated (and formatted with Scalafmt if necessary/configured). But because of this bug, the execution of my sbt plugin task fails. 😕

Here's an example of a run that failed because of that bug: https://github.com/guizmaii/poc_compile_time_caliban_client_generation/runs/3835687661?check_suite_focus=true

And here's one that worked: https://github.com/guizmaii/poc_compile_time_caliban_client_generation/runs/3754046726?check_suite_focus=true

[tgodzik]

Thanks for reporting! This looks like an issue with coursier though, maybe it's an older version of coursier in 2.7.5? Have you tried using the latest version of scalafmt?

What is weird is that we actually run tests that download scalafmt in Metals and I have never seen them fail. Maybe it's an issue with github runners? Did this happen only recently?

Anyway, this doesn't look like something that we can fix in scalafmt 🤔

[guizmaii]

maybe it's an older version of coursier in 2.7.5? Have you tried using the latest version of scalafmt?

Yes, I did. Initially, I was using v3.0.x in my "POC project" and was having the same issue. I moved to 2.7.5 because I thought the bug was coming from v3 but in fact, it happens in both.

Maybe it's an issue with github runners?

Caliban is not using Github Actions. It's using CircleCI and IIRC, we also had this issue in CircleCI.

To validate this point I just created 2 draft PRs in Caliban to try to reproduce the issue in CircleCI.
See:

• Try( Reproduce scalafmt bug in CircleCI with v2.7.5 ) ghostdogpr/caliban#1070
• Try( Reproduce scalafmt bug in CircleCI with v3.0.x ) ghostdogpr/caliban#1071

Corresponding CircleCI runs:

• https://app.circleci.com/pipelines/github/ghostdogpr/caliban/3007/workflows/d68dc38b-351c-4ebc-b00a-399d38f12517/jobs/13644
• https://app.circleci.com/pipelines/github/ghostdogpr/caliban/3008/workflows/8a01ada7-d5fa-4960-9553-da30e433e26a/jobs/13640

Did this happen only recently?

I started to work on this sbt plugin maybe 1 or 2 months ago.

Can you maybe point me where is the code in charge of interacting with Coursier in Scalafmt, so I can have a look?

Edit:

Interestingly, on the 2 CircleCI runs, one failed and one passed 😄
Illustrating the problem perfectly 😄

[tgodzik]

Could it bee that it's being run in parallel? I found an issue in coursier coursier/coursier#1815

[guizmaii]

Could it be that it's being run in parallel?

Maybe, but in theory, seeing the code here: #2793 (comment), it should be sequential (all the code is run in one "blocking" task), except if it's Coursier or Scalafmt that are parallelizing the calls.

But for sure, there's something around that. The non-deterministic nature of this bug clearly points towards this IMO.

[tgodzik]

Maybe it's the case that you are trying to load Scalafmt multiple times? So if you are running format method in parallel this could show up, though I don't know ZIO to see exactly what is going on.

[guizmaii]

Well, based on this potential solution to my problem: coursier/coursier#2022 (comment), I implemented an "ugly fix" in Caliban: ghostdogpr/caliban#1072 and rebased my 2 draft PRs on top of this "ugly fix" to relaunch the 2 CircleCI runs:

• https://app.circleci.com/pipelines/github/ghostdogpr/caliban/3014/workflows/df14cd61-d6d5-43d6-83d5-8d9dd2a6fbbc/jobs/13675
• https://app.circleci.com/pipelines/github/ghostdogpr/caliban/3013/workflows/6b3b5978-7fed-4de1-aece-91f3b82f559e/jobs/13672

[guizmaii]

Ok. It works and I understand why I have this issue.

I am indeed running multiple instances of Coursier in parallel because I'm generating, in this test, something like 4 different Caliban clients in paralleled so when the code is generated, the calls the ScalaFmt are in parallel too.

So I'm effectively experiencing the race condition that you mentioned (coursier/coursier#1815)

Thanks for your help 🙏

[lallea]

Fixed in #3530.