(Randomly) Unresolved scalafmt on version 2.0.0

[GMadorell]

• Version: 2.0.0-RC5
• Integration: SBT
• Configuration:

version=2.0.0-RC5
style = defaultWithAlign
maxColumn = 120
align.openParenDefnSite = false
#danglingParentheses = true
project.excludeFilters = [
  target
]
trailingCommas = never

Also, in project/plugins.sbt

addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.0.0-RC5")

Steps

Run sbt scalafmtCheck in our CI pipeline.

Problem

Hi! We have upped the version of scalafmt recently and we have since got this problem when trying to check if files are formatted.

We don't really know how to proceed. We have checked github issues and tried everything that came up from Google before trying to post this issue.

It seems that sometimes the scalafmt check fails to resolve the dependency and fails overall with the error explained below.

How could we troubleshoot this?

We get the following error:

failed to resolve Scalafmt version '2.0.0-RC5'

Expectation

Dependency is resolved correctly.

Workaround

Rerunning the build makes it work sporadically.

Notes

.

[GMadorell]

Maybe duplicated of #1399

[dhoepelman]

Running into the same problem with RC7 on CirclecCI

As a work-around, we're now using a build cache with the correct version included.

[poslegm]

@GMadorell please try to update sbt plugin to latest version:

addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.0.1")

[dhoepelman]

Problem seems to be gone in 2.0.1/2.0.0-RC8, thanks!

[GMadorell]

Update, I am now using both

addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.0.2")

and

version=2.0.0

in the scalafmt config file.

It is still failing randomly (even with concurrency disabled).

[GMadorell]

@tanishiking Do you want me to open another ticket?

[jgogstad]

I also experience this, randomly our CI fails with

[error] failed to resolve Scalafmt version '2.0.0': /home/jenkins/workspace/somebranch/.scalafmt.conf

Rerun usually works. Using sbt-scalafmt 2.0.2 and version = 2.0.0 in .scalafmt.conf

[GMadorell]

@tanishiking this is still failing by the way, do you want me to open another ticket for this?

[tanishiking]

@GMadorell @jgogstad Thank you for the information. Hmm, maybe we need some more investigation... Could you open another ticket?
And is it possible to attach a CI result that reproduces this phenomenon?

[hajnalmt]

This is still an issue. Did you open a ticket @GMadorell ? Just to track the update of this issue.

[tanishiking]

Re-opened this issue for tracking on the problem. If anyone has any clues: something like CI results or detailed environment, it would be helpful for tackling this :)

[martinffx]

After fighting with this all day, I've nailed it down to sbt-scalafmt. It seems to move to the new repo has broken something. If I use addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.0.0") or subsequent versions I get this unresolved scalafmt issue. Moving back to the last version in the old repo addSbtPlugin("com.geirsson" % "sbt-scalafmt" % "1.6.0-RC4") resolves the issue for me.

Both plugins work for me locally, even after clearing my cache...

I don't know if that helps?

[Rich-Carreiro-Imprivata]

We are also currently seeing this problem with addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.0.4"). Works fine on our local Mac development machines, but fails most the time in our linux-based CI/CD environment.

[davidnadeau]

Using bitbucket pipline as a ci env, and getting this issue with scala fmt 2.0.X.

[eatkins]

I also see this both in sbt and intellij. Weirdly, with intellij, it will often work with 2.0.1, but not 2.0.0, 2.0.3 or 2.0.4.

[eatkins]

Also, my unsubstantiated hunch is that this is a coursier bug. I think scalafmt is using a fairly old version of coursier via coursier small? Perhaps it needs an upgrade.

[poslegm]

Is there any chance that this could be related to the current coursier version?

[eatkins]

I wouldn't think so. Scalafmt uses https://github.com/olafurpg/coursier-small to do the downloading and resolution and, according to the readme, coursier-small shades the coursier namespace so it should be using the shaded version of coursier (which is quite old at this point).

[Rich-Carreiro-Imprivata]

FWIW, there is a pending pull request to use coursier instead of coursier-small:
#1443

[Rich-Carreiro-Imprivata]

One thing we have noticed is:

• We do not have this problem on our Macs.
• We do have it on our linux-based CI machines.
• Even though all our stuff is configured to use our local artifactory, at least on our Macs sbt-scalafmt is going straight to repo1.maven.org (I can verify this by looking at ~/Library/Caches/Coursier and seeing where stuff is coming from), whereas literally every other dependency of ours is coming from our artifactory (where we cache 3rd party dependencies as well as our own stuff).

So could it be possible that what's happening is CI machines getting blacklisted by repo1.maven.org? (If CI build agents start "too fresh" they won't have any locally-cached stuff and so sbt-scalafmt will go to repo1.maven.org to get it, and that'll happen again and again. Could that trigger a blacklist?)

For those familiar with coursier-small, could whatever code repo1.maven.org sends back when it's blacklisted someone result in the errors we're all seeing?

[poslegm]

FWIW, there is a pending pull request to use coursier instead of coursier-small:
#1443

I should definitely keep working on that PR. Currently coursier update in scalafmt blocked by coursier/interface#57. I hope that I can find the time to solve this issue but, anyway, any help is welcome 😄

[ghost]

Maybe the authors of the library can provide some workaround instructions for using scalafmt CLI to format main, test, and integration test, until this issue is resolved? Thanks!

[jasiak]

One thing we have noticed is:

• We do not have this problem on our Macs.
• We do have it on our linux-based CI machines.
• Even though all our stuff is configured to use our local artifactory, at least on our Macs sbt-scalafmt is going straight to repo1.maven.org (I can verify this by looking at ~/Library/Caches/Coursier and seeing where stuff is coming from), whereas literally every other dependency of ours is coming from our artifactory (where we cache 3rd party dependencies as well as our own stuff).

So could it be possible that what's happening is CI machines getting blacklisted by repo1.maven.org? (If CI build agents start "too fresh" they won't have any locally-cached stuff and so sbt-scalafmt will go to repo1.maven.org to get it, and that'll happen again and again. Could that trigger a blacklist?)

For those familiar with coursier-small, could whatever code repo1.maven.org sends back when it's blacklisted someone result in the errors we're all seeing?

I've faced the same problems with scalafmt. Runs fine on local dev machine (also mac) but fails on Jenkins CI all the time with same error "failed to resolve Scalafmt version". I've tried with a few different versions (incl. 2.0.0, 2.0.1) but it didn't help and issue still occurs.

[Rich-Carreiro-Imprivata]

Another piece of evidence in favor of my blacklisting theory -- we just ssh'd into one of our CI build agents and tried to do
curl -i -v -o /dev/null https://repo1.maven.org/maven2/org/scalameta/scalafmt-core_2.12/2.0.0/scalafmt-core_2.12-2.0.0.jar
and got a 403, indicating the CI agents have been blacklisted. When I do this from my local Mac I get a 200 and the file downloads.

And when we look at the full response the CI agents are getting to that command see the blacklist warning text (You've been blocked from this operation due to violation of the Central terms of service. Please see https://central.sonatype.org/blacklisted to remediate this issue.).

In short, I think the problem is because the coursier embedded in sbt-scalafmt is ignoring the repos that have been configured for sbt via .sbtopts and the like and is just going straight to Maven itself.

[eatkins]

That could certainly be a contributing factor, but I run into this on my local machine quite often. I can’t prove it isn’t, but I doubt my home ip is being rate limited by maven.

[adamw]

But then, if it was because of blacklisting, wouldn't downloads of other artifacts fail as well? Without scalafmt, or with an older one, the build runs just fine - downloads all dependencies, among other things.

[Rich-Carreiro-Imprivata]

Well at least for us, all our dependencies are cached in artifactory, so other than scalafmt we are never going to maven directly (and what we see in the coursier cache bears that out) -- the request is to artifactory and it either satisfies the request if it has the artifact or else downloads and caches the artifact). And I'm pretty sure our artifactory is not in the same IP block that the CI build agents are. So a blacklist of the build agents would not affect our artifactory.

[poslegm]

Please try it with sbt-scalafmt v2.0.6. There was coursier update in scalafmt-dynamic.

[ghost]

Excited! Quick question, will COURSIER_REPOSITORIES env var allow us to overrride coursier-interfaces behavior despite the repositories being set for scalafmt fetch?

[poslegm]

@kanishka-azimi I haven't tried it, but expected behavior is "yes, it should override repositories"

[t-botz]

Same problem.
On my CI environment there is no internet access and we have artifactory mirrors internally.
We use -Dsbt.repository.config=<repositoryConf> to override the repositories which works with everything but scalafmt. Considering sbt now work with coursier and still honor our repo configuration, it doesn't seem that scalafmt has a good reason to work any differently.

Setting COURSIER_REPOSITORIES definitely works as a work around.

[tanishiking]

@thibaultdelor which version of scalafmt are you using?
scalafmt 2.1.1, sbt-scalafmt 2.0.7 honor the COURSIER_REPOSITORIES as you said.
for sbt resolver setting, maybe sbt-scalafmt can evaluate the sbt resolvers setting #1521

[tanishiking]

@GMadorell doea this stil reproduce in sbt-scalafmt 2.0.7 ?

[Rich-Carreiro-Imprivata]

I can confirm on 2.0.7/2.1.1 that COURSIER_PROPERTIES appears to be honored.

However, I would argue sbt-scalafmt is still broken if it doesn't honor the -Dsbt.repository.config setting because setting that property should be sufficient to configure all repos used by all stages of the build -- we shouldn't have to set COURSIER_PROPERTIES (and keep it in sync with -Dsbt.respository.config settings, or abandon -Dsbt.respository.config) just for sbt-scalafmt. Especially since the now coursier-based sbt does honor the property.

And I can confirm that as of 2.0.7 (and scalafmt 2.1.1) it does not honor the property.

[ghost]

Alternatively, sbt might consider deprecating sbt.repository.config and standardize on COURSIER_REPOSITORIES. Until a decision emerges there, I would consider this issue complete for now, as there is a decent workaround possible

[t-botz]

@kanishka-azimi I am more than happy with how sbt is working at the moment regarding overriding repos.
Deprecating sbt.repository.config, which I think is a bad idea, won't happen in a day anyway.

At the moment, scalafmt-sbt is inconsistent with sbt behaviour, it is a bug, I don't think that we should argue about that.

Given the work around and the number of people impacted, I can understand that the priority of this issue might be low.

[t-botz]

@tanishiking I am on 2.0.7/2.1.1.

[tanishiking]

Can we talk about -Dsbt.repository.config at #1521 ?
I wanna set the topic of this thread unresolved scalafmt that happens randomly like #1415 (comment)
we can support the option from sbt-scalafmt but we have some TODOs to realize the feature.

[tanishiking]

Let me close this issue for inactivity.
If anybody still has a problem with downloading scalafmt-core failing ** randomly** in a latest version of sbt-scalafmt, feel free to open a new issue :)
If you have a problem with custom repositories while downloading scalafmt-core, please see #1521
Thank you everybody for the information.