Gradle build: cleanup of dependency resolution and consolidation of dependency versions

[dweiss]

This patch attempts to clean up and consolidate all kinds of aspects related to dependencies used throughout the build and "versions", understood not only as dependency versions but also minimum java version, google java format version, etc.

Notable changes.

• palantir-consistent-versions plugin is dropped entirely and replaced with explicit versions using a single gradle's version catalog (top-level versions.toml file). This file contains all dependencies for plugins, build script dependencies and module dependencies, along with their version numbers. It also contains the minimum java version, google formatter version and min gradle version to run the build. The same catalog is reused for included builds (composite builds) as well.
• buildSrc is replaced with build-tools/build-infra composite build project. This is the first step on the long road to move some of the scripts into a convention/ build helper plugin, making them hopefully faster to load and execute.
• tidy and forbidden apis are now applied to composite builds as well.

While this isn't strictly necessary, the concept of a "lock file" for dependencies used in the main and test modules is useful - it allows some manual control when transitive dependency versions or dependencies change (otherwise they'd go unnoticed). I used my own small plugin to collect these dependencies and update/ verify their consistency (on tidy and check tasks). This plugin does not alter dependency resolution (like palantir's did). It merely sums up what is used, in which version and where. See versions.lock to see the output.

I also corrected a few deprecations and minor glitches in gradle files.

This patch does not upgrade any dependencies - it's left as a follow up.

[dweiss]

I'm going to merge this in. Please report any problems with the build if you encounter them.

[dsmiley]

Overall; thanks for doing this @dweiss !

What script generates versions.lock? The "because" sections look mysterious.

[dweiss]

A plugin does this. It is similar in nature to palantir's but "passive" - it only collects dependencies from selected configurations and makes sure they're consistent, it won't lock their versions or try to force them. It's more of a safety/ early warning mechanism - you'll have to use gradle's own mechanisms for version alignment, if something is not right.

The documentation is non-existent but you can grep through the code, if you're interested.

https://github.com/carrotsearch/gradle-dependencychecks-plugin

[dweiss]

Oh - the "because" contain a hash key and all configurations/projects which refer to a dependency. These hashes are used next to dependencies to shorten up the list. For example these are pulled in by the same set of configurations:

      "net.sourceforge.nekohtml:nekohtml:1.9.17" : "6f16ff86,refs=2",
      "org.apache.commons:commons-compress:1.19" : "6f16ff86,refs=2",

from:

"because" : {
    ...
    "6f16ff86" : [
      {
        "configuration" : "testCompileClasspath",
        "projectPath" : ":lucene:benchmark"
      },
      {
        "configuration" : "testRuntimeClasspath",
        "projectPath" : ":lucene:benchmark"
      }
    ],

the plugin needs a set of projects and configurations to take into account - this is set up in dependencies.gradle.