Incorrect report with older library versions when newer versions are available in repository #103

gogreennow · 2016-05-30T21:59:55Z

Hello!
Below is the snippet of report generated using the plugin, where the latest third party library versions are incorrectly reported.

The following dependencies exceed the version found at the release revision level:

junit:junit [4.11 <- 4.5]
org.apache.ws.commons.axiom:axiom-api [1.2.13 <- 1.2.9]
org.apache.ws.commons.axiom:axiom-impl [1.2.13 <- 1.2.9]
xerces:xercesImpl [2.9.1 <- 2.9.1-appian-1.0]

The following dependencies have later release versions:

antlr:antlr [2.7.6 -> 20030911]
asm:asm-analysis [3.1 -> 20041228.180559]
asm:asm-util [3.1 -> 20041228.180559]

ben-manes · 2016-05-30T22:14:07Z

Can you provide a trimmed down build script? This could be because a repository has bad metadata (e.g. gradle's plugin portal doesn't update the maven-metadata.xml files after a release)

ben-manes · 2016-05-30T22:29:53Z

This could also be a bug in Gradle or the tool that performed the dependency's release. We delegate to Gradle's dependency resolution rather than reinvent it.

For example if we look at the metadata for asm-analysis:

<metadata>
  <groupId>asm</groupId>
  <artifactId>asm-analysis</artifactId>
  <version>1.5.1</version>
  <versioning>...</versioning>

A properly formed metadata should look something like,

<metadata>
  <groupId>com.github.ben-manes.caffeine</groupId>
  <artifactId>caffeine</artifactId>
  <versioning>
    <latest>2.3.0</latest>
    <release>2.3.0</release>
    <versioning>...</versioning>

Because asm-analysis doesn't specify the latest release both Gradle and Maven Central take the largest value in the version list. The easiest solution would be to use a resolutionStrategy to reject this bad version (20041228.180559).

The exceeded look fine on Maven Central. If you are using a virtual repository (Sonatype, Artifactory) that proxies then perhaps its returning the metadata from one with stale information. You might want to order the repository list if that's the case.

gogreennow · 2016-05-31T01:55:06Z

Thanks for your quick response.

Yes, we use Sonatype Nexus. The latest version of junit:junit is 4.12, while the report indicates we are ahead of the latest assuming 4.5 is the latest
Similarly the latest version of axiom-api and axiom-impl is 1.2.19, while the report indicates 1.2.9 is the latest. What could be contributing to incorrect latest version getting picked?

Relevant snippet from the build script shown below -
buildscript {
repositories {
jcenter()
}
dependencies {
classpath 'com.github.ben-manes:gradle-versions-plugin:0.12.0'
...
}
}

// Dependency Management
allprojects {
ext {
axiom_version = '1.2.13'
junitVersion = '4.11'
}
}
...
dep("org.apache.ws.commons.axiom:axiom-api:$axiom_version") {
exclude group: 'commons-logging', module: 'commons-logging'
}
dep("org.apache.ws.commons.axiom:axiom-impl:$axiom_version") {
exclude group: 'commons-logging', module: 'commons-logging'
}
dep "junit:junit:${junitVersion}"
...
...
// Runtime dependency for tests
testCompile /runtime/ "asm:asm-analysis:3.1"
testCompile /runtime/ "asm:asm-util:3.1"

ben-manes · 2016-05-31T02:40:47Z

Sorry, I meant a build script that demonstrated the problem that I could run to debug with.

I think the problem is due to your Sonatype Nexus repository configuration. If you are using it as your project's repository then it might be returning back the maven-metadata.xml from the first repository that the dependency was found in. If it searched Maven Central first then it might find the latest. The way to test is to run your isolated build script that resolves against Maven Central and Nexus to see if it is the problem.

Another possibility is something like #97. In that case a BOM was used to lock the dependency resolution to a specific version. This caused the dependency search to not discover the newer versions because the resolution rule disallowed it.

If I run your snippet standalone then the report shows the correct information. This indicates that there is a configuration issue in your build or Nexus repository.

apply plugin: 'com.github.ben-manes.versions'
apply plugin: 'java'

buildscript {
  repositories {
    jcenter()
  }
  dependencies {
    classpath 'com.github.ben-manes:gradle-versions-plugin:0.12.0'
  }
}

repositories {
  jcenter()
}

ext {
  axiom_version = '1.2.13'
  junitVersion = '4.11'
}

dependencies {
  testCompile "asm:asm-analysis:3.1"
  testCompile "asm:asm-util:3.1"
  testCompile "junit:junit:${junitVersion}"
  compile "org.apache.ws.commons.axiom:axiom-api:$axiom_version"
  compile "org.apache.ws.commons.axiom:axiom-impl:$axiom_version"
}

The following dependencies are using the latest milestone version:
 - com.github.ben-manes:gradle-versions-plugin:0.12.0

The following dependencies have later milestone versions:
 - asm:asm-analysis [3.1 -> 3.3.1]
 - asm:asm-util [3.1 -> 3.3.1]
 - junit:junit [4.11 -> 4.12]
 - org.apache.ws.commons.axiom:axiom-api [1.2.13 -> 1.2.19]
 - org.apache.ws.commons.axiom:axiom-impl [1.2.13 -> 1.2.19]

silpamittapalli · 2016-06-19T01:08:24Z

Appreciate your prompt reply, @ben-manes.

Per your recommendation, tried out few things.

Changed my build script to search dependencies in Maven Central first, before looking at the internal Nexus repos and local Maven home. Confirmed that the search order is correct using info flag (shown below)

Searched in the following locations:
https://jcenter.bintray.com/commons-resources/commons-resources/maven-metadata.xml
https://jcenter.bintray.com/commons-resources/commons-resources/
https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/maven-metadata.xml
https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.pom
https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.jar
https://internal-repo2/repo/commons-resources/commons-resources/maven-metadata.xml
https://internal-repo2/repo/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.pom
https://internal-repo2/repo/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.jar
file:/local-m2/repository/commons-resources/commons-resources/maven-metadata.xml
file:/local-m2/repository/commons-resources/commons-resources/

Removed the configuration in one of the test-gradle scripts that was forcing to use JUnit version 4.5 using resolution strategy since those old tests work only with that specific version of JUnit. After removing that resolutionStrategy, I no longer see 4.5 being picked as the latest version of JUnit in the report. But now 4.11 is always reported as the latest version of JUnit even though both Maven Central and our internal nexus repos have version 4.12 listed first in the maven-metadata.xml. Tried searching in both JCenter as well as Maven Central first, but still seeing 4.11

Comparing dependency (current: junit:junit:4.11, latest: 4.11)
...

http://repo1.maven.org/maven2/junit/junit/maven-metadata.xml
junit junit 4.12 4.12 3.7 3.8 3.8.1 3.8.2 4.0 4.1 4.2 4.3 4.3.1 4.4 4.5 4.6 4.7 4.8 4.8.1 4.8.2 4.9 4.10 4.11-beta-1 4.11 4.12-beta-1 4.12-beta-2 4.12-beta-3 4.12 20141204181041

https://internal-repo1/nexus/content/groups/public/junit/junit/maven-metadata.xml
junit junit 4.12 4.12 3.7 3.8 3.8.1 3.8.2 4.0 4.1 4.2 4.3 4.3.1 4.4 4.5 4.6 4.7 4.8 4.8.1 4.8.2 4.9 4.10 4.11-beta-1 4.11 4.12-beta-1 4.12-beta-2 4.12-beta-3 4.12 20141204181041

We are using a custom version 0.1-dev-xxx-1.0 of commons-resources but that is not identified by the plugin and the report doesn't show any version, while the meta data xml provided by internal Nexus repo has the version listed, while the library jar does exist in Maven Central.

https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/maven-metadata.xml
commons-resources commons-resources 0.1-dev 0.1-dev 0.1-dev

http://repo1.maven.org/maven2/commons-resources/commons-resources/maven-metadata.xml
commons-resources commons-resources 0.1-dev 0.1-dev

Failed to determine the latest version for the following dependencies (use --info for details):

commons-resources:commons-resources

The exception that is the cause of unresolved state: Could not find any matches for commons-services:commons-services:+ as no versions of commons-services:commons-services are available.
Searched in the following locations:
https://repo1.maven.org/maven2/commons-services/commons-services/maven-metadata.xml
https://repo1.maven.org/maven2/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.pom
https://repo1.maven.org/maven2/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.jar
...
...
Can you please provide your thoughts?

ben-manes · 2016-06-19T06:59:34Z

It sounds like you probably still have a resolution strategy defined somewhere (maybe a plugin) restricting the versions. I think the info level prints out when a candidate is rejected when Gradle tries to resolve dependencies. You might have a plugin like dependency-management-plugin adding restrictions.

I'd first try adding an afterEvaluate that prints the selection, substitution, and forced version rules. If you have any listings it might give you a lead as to who added it.

silpamittapalli · 2016-06-19T22:30:12Z

You're correct. Having a resolution strategy that forces a specific version of library is causing it. It would be nice to have the plugin generate correct report even though the build script has such resolution strategies. Would it be possible to update the plugin to handle this?

silpamittapalli · 2016-06-19T22:41:58Z

Also, displaying the age of the library version referenced in build script would be quite helpful.

For example, our build script references jcs:jcs:1.3, while Maven Central metadata.xml doesn't have 1.3 listed even though https://repo1.maven.org/maven2/jcs/jcs/1.3 is available. In that case, the plugin chose 20030822.182132 as the latest version which is not right. It is indeed 6 months behind 1.3 .

Comparing dependency (current: jcs:jcs:1.3, latest: 20030822.182132)
...
The following dependencies have later release versions:

jcs:jcs [1.3 -> 20030822.182132]

Is there anything that can be done to correct this? I guess we can't have a resolution strategy to ignore such versions because in some instances, there could be versions in that format.

silpamittapalli · 2016-06-19T22:50:34Z

The plugin fails to resolve versions for some of the libraries though the Maven Central has 0.1-dev version listed in the meta-data.xml and our build script references a custom version of it 0.1-dev-xxx-1.0

Any thoughts on why the plugin fails to resolve the versions for such libraries?

The exception that is the cause of unresolved state: Could not find any matches for commons-resources:commons-resources:+ as no versions of commons-resources:commons-resources are available.
Searched in the following locations:
https://repo1.maven.org/maven2/commons-resources/commons-resources/maven-metadata.xml
https://repo1.maven.org/maven2/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.pom
https://repo1.maven.org/maven2/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.jar
https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/maven-metadata.xml
https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.pom
https://internal-repo1/nexus/content/groups/public/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.jar
https://internal-repo2/commons-resources/commons-resources/maven-metadata.xml
https://internal-repo2/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.pom
https://internal-repo2/commons-resources/commons-resources/0.1-dev/commons-resources-0.1-dev.jar

commons-services:commons-services
The exception that is the cause of unresolved state: Could not find any matches for commons-services:commons-services:+ as no versions of commons-services:commons-services are available.
Searched in the following locations:
https://repo1.maven.org/maven2/commons-services/commons-services/maven-metadata.xml
https://repo1.maven.org/maven2/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.pom
https://repo1.maven.org/maven2/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.jar
https://internal-repo1/nexus/content/groups/public/commons-services/commons-services/maven-metadata.xml
https://internal-repo1/nexus/content/groups/public/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.pom
https://internal-repo1/nexus/content/groups/public/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.jar
https://internal-repo2/commons-services/commons-services/maven-metadata.xml
https://internal-repo2/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.pom
https://internal-repo2/commons-services/commons-services/1.0-dev/commons-services-1.0-dev.jar

ben-manes · 2016-06-19T23:05:15Z

Its messy because how much flexibility we have depends on Gradle's APIs. I didn't want to reinvent the dependency management (parsing poms, querying repositories, etc) and then struggle to stay forward compatible (Gradle added resolution strategies, S3 repositories, etc). Delegating to their APIs makes this plugin fairly trivial, but also limits adding workarounds.

The JCS example is a broken release, as I'd expect Maven's version plugin to report the wrong result due to maven-metadata.xml not being updated. You could add a resolution strategy scoped to this plugin, but I don't think it would fix this particular case.

Gradle doesn't let us remove resolution strategies from a configuration. We clone the original, rewrite the dependencies, add our own resolution strategy, and resolve it to capture the versions. Potentially we could use a detached configuration instead of copying, like we did originally. That lets us not copy over the configuration attributes, like your resolution strategies, which may or may not be desirable for a given user. However we switched away because RepositoryHandler was buggy when adding repositories (at the time to an aggregate), so a clone avoided that issue. Potentially a plugin flag could be set to decide between cloning or using a detached to honor / ignore your configuration.

Any thoughts on why the plugin fails to resolve the versions for such libraries?

I suspect Gradle is struggling to understand the maven-metadata.xml. If you look at the commons-services' xml and at Caffeine's you'll see the former lacks the latest and release tags. It may be that Gradle needs to be more flexible in what it accepts, as its failing with the proper declaration of "commons-services:commons-services:+". There's nothing we can do at the plugin and you'd have to file a bug on the Gradles' forum.

dorogush · 2016-08-12T10:53:10Z

Confirm this issue.
I planned to use

resolutionStrategy {
  failOnVersionConflict()
  force ...
}

in order to explicitly resolve all conflicts. Outdated dependencies are not detected.

ben-manes · 2016-10-25T22:40:05Z

Closing since I don't think there is anything for me to do here. But please re-open if I can help.

nzakharchenko · 2017-07-18T17:16:13Z

Hey @ben-manes , I have exactly the same issue with force dependencies. I'm using it to resolve all conflicts and I don't see any reason why does it implies on work of plugin. Is there any possibility to change that behavior? Because this plugin is really nice, but becomes unusable for me - almost all my dependencies are forced :(

ben-manes · 2017-07-18T17:27:22Z

Unfortunately, no.

The only way to safely apply the user's configuration (e.g. repositories, dependencies) is to use Configuration#copyRecursive which includes the resolutionStrategy. The previous logic using a detached configuration and RepositoryHandler uncovered bugs in Gradle causing duplicates or missing repositories. The configuration intersection of what users want, other plugin interactions, Gradle's expectations and bugs, etc. becomes too difficult to manage smoothly.

Instead you could conditionally force only if this task isn't on the task graph. This would be something like if (!gradle.taskGraph.hasTask(tasks.dependencyUpdates)) { force }. Then when this plugin is run the dependencies are not forced, but when not run they are.

nzakharchenko · 2017-07-18T18:08:50Z

Thank you @ben-manes for fast response! I will try out your suggestion.

PLNech mentioned this issue Oct 25, 2016

Reported last version of dependencies is outdated #122

Closed

ben-manes closed this as completed Oct 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect report with older library versions when newer versions are available in repository #103

Incorrect report with older library versions when newer versions are available in repository #103

gogreennow commented May 30, 2016

ben-manes commented May 30, 2016

ben-manes commented May 30, 2016 •

edited

Loading

gogreennow commented May 31, 2016

ben-manes commented May 31, 2016 •

edited

Loading

silpamittapalli commented Jun 19, 2016 •

edited

Loading

ben-manes commented Jun 19, 2016

silpamittapalli commented Jun 19, 2016

silpamittapalli commented Jun 19, 2016

silpamittapalli commented Jun 19, 2016

ben-manes commented Jun 19, 2016

dorogush commented Aug 12, 2016

ben-manes commented Oct 25, 2016

nzakharchenko commented Jul 18, 2017

ben-manes commented Jul 18, 2017

nzakharchenko commented Jul 18, 2017

Incorrect report with older library versions when newer versions are available in repository #103

Incorrect report with older library versions when newer versions are available in repository #103

Comments

gogreennow commented May 30, 2016

ben-manes commented May 30, 2016

ben-manes commented May 30, 2016 • edited Loading

gogreennow commented May 31, 2016

ben-manes commented May 31, 2016 • edited Loading

silpamittapalli commented Jun 19, 2016 • edited Loading

ben-manes commented Jun 19, 2016

silpamittapalli commented Jun 19, 2016

silpamittapalli commented Jun 19, 2016

silpamittapalli commented Jun 19, 2016

ben-manes commented Jun 19, 2016

dorogush commented Aug 12, 2016

ben-manes commented Oct 25, 2016

nzakharchenko commented Jul 18, 2017

ben-manes commented Jul 18, 2017

nzakharchenko commented Jul 18, 2017

ben-manes commented May 30, 2016 •

edited

Loading

ben-manes commented May 31, 2016 •

edited

Loading

silpamittapalli commented Jun 19, 2016 •

edited

Loading