allow excluding transitive jars from jvm_artifact targets

[tdyas]

Pants v1 supported excluding certain jars from the transitive dependency graph of a jar_libary target. The jvm_artifact target type should also support exclusions. If done in a similar manner to v1, it would be via an exclusions field on jvm_artifact.

It also supported an intransitive=True flag, which excluded all transitive dependencies.

[Eric-Arellano]

Can you use !! ignores in the dependencies field instead?

The dependencies toolbox at https://www.pantsbuild.org/docs/targets

[tdyas]

Can you use !! ignores in the dependencies field instead?

The dependencies toolbox at https://www.pantsbuild.org/docs/targets

I'm not sure that would cover this case. Given jvm_artifact does not support dependencies, the user would have to write a jvm_artifact target and then reference it with !! on every target that pulled in the original dependency where we need that exclusion. The issue here is to filter that jar(s) from the classpath entirely with respect to the transitive dependencies of a specific dependency. The excluded jar at a different version might appear (and be wanted) to come in from another part of the transitive dependency graph. Easier to specify the exclusion on the jvm_artifact that represents the part of the dependency graph where we want the filtering.

[chrisjrn]

Taking this on -- it looks like cs fetch has an excludes option which lets you specify partially qualified coordinates (i.e. group:artifact without version). Let's see what it would take to make that work.

[tdyas]

I see at least two similar but slightly different use cases for exclusions:

1. Exclude a jar from a resolve entirely.
2. Exclude a jar that is a transitive dependency of a jvm_artifact but allow it to come in via another jvm_artifact (presumably at a different version). That is, influence the selection of jars by selectively pruning parts of the jar dependency graph. This use case may be moot with lockfiles though since the user could maybe pin the transitive jar via an explicit jvm_artifact?

(1) points at exclusions being a per-resolve item (like Scala version), while (2) points at it being a per-jvm_artifact field. But depends on how the use case in (2) should be accomplished.

[tdyas]

I had the use case in (2) in the past in Pants v1 to resolve conflicting versions of netty by pruning netty from one of the jar_library v1 targets. v1 didn't have lockfiles though, so for v2, how we would handle pinning netty in such a case probably will point at how to model this.

[chrisjrn]

@tdyas I'm going to aim for solving (2) initially, since it's the easier option, which should be easy enough, given that we fetch artifacts on an individual basis. We may need to think a bit harder about how to model (1).

I haven't tested coursier's behaviour if you specify a specific version to fetch, but it'd be interesting to see what happens if a higher version shows up as a transitive dependency. Let me go try that one out.

[tdyas]

We fetch individually but we resolve lockfiles collectively by passing in all of the coordinates:

pants/src/python/pants/jvm/resolve/coursier_fetch.py

Line 371 in ad85e9b

*coursier_resolve_info.coord_arg_strings,

Thinking "aloud", this brings up some points about when exclusion can be applied (with some regard to whether the user writes it in a jvm_artifact field or in a per-resolve exclusions option):

1. If the exclusion is applied during lockfile resolution, it would be removed from resolve entirely unless Coursier had a way to prune for a specific coordinate. And given how we pass coordinate strings during lockfile resolution, not sure if that is the case.
2. If the exclusion is applied when constructing a classpath, then it could influence what subset of jars to take from the resolve. Not sure if this corresponds to a real use case though. Thinking to the netty use case, if I used the jvm_artifact with the exclusion as a dependency and not the other jvm_artifact with a netty dep, then netty would be missing when instead I wanted to influence what netty version to include.

[chrisjrn]

Per below, it looks like Coursier's default way to handle a complex resolve is to use the newest version of the artifact. This means we'll need to force our version of the artifact when we do the resolve (re-specifying the versions from our jvm_artifact targets as a conflict resolution preference should be sufficient here, tbqh).

We will still need to specify excludes at fetch time to ensure that the transitive dependency isn't fetched (this is actually even more important than before, since we won't have a SHA digest for the excluded dependency.

chrisjrn@chrisjrns-MacBook-Pro example-jvm % less coursier_report.json | jq .
{
  "conflict_resolution": {
    "com.fasterxml.jackson.core:jackson-core:2.11.4": "com.fasterxml.jackson.core:jackson-core:2.12.2"
  },
  "dependencies": [
    {
      "coord": "com.fasterxml.jackson.core:jackson-annotations:2.12.2",
      "file": "/Users/chrisjrn/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.2/jackson-annotations-2.12.2.jar",
      "directDependencies": [],
      "dependencies": []
    },
    {
      "coord": "com.fasterxml.jackson.core:jackson-core:2.12.2",
      "file": "/Users/chrisjrn/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.2/jackson-core-2.12.2.jar",
      "directDependencies": [],
      "dependencies": []
    },
    {
      "coord": "com.fasterxml.jackson.core:jackson-databind:2.12.2",
      "file": "/Users/chrisjrn/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.2/jackson-databind-2.12.2.jar",
      "directDependencies": [
        "com.fasterxml.jackson.core:jackson-annotations:2.12.2",
        "com.fasterxml.jackson.core:jackson-core:2.12.2"
      ],
      "dependencies": [
        "com.fasterxml.jackson.core:jackson-annotations:2.12.2",
        "com.fasterxml.jackson.core:jackson-core:2.12.2"
      ]
    }
  ],
  "version": "0.1.0"
}

[tdyas]

Might be relevant: I had to disable "strict" resolution of versions due to a bug described in #13496. Not directly on-point with exclusions but we may want to revisit strict dependency resolution again.

[chrisjrn]

So for starters, which version of jackson-core should we be returning by default here?

rule_runner.request(
        CoursierResolvedLockfile,
        [
            ArtifactRequirements.from_coordinates(
                [
                    Coordinate(group="com.fasterxml.jackson.core", artifact="jackson-databind", version="2.12.1"),
                    Coordinate(group="com.fasterxml.jackson.core", artifact="jackson-core", version="2.11.1"),
                ]
            ),
        ],
    )

[stuhood]

The flag used for a local exclude in v1 was:

pants/src/python/pants/backend/jvm/tasks/coursier_resolve.py

Lines 509 to 516 in 674df55

	# Local exclusions
	local_exclude_args = []
	for jar in jars:
	for ex in jar.excludes:
	# `--` means exclude. See --local-exclude-file in `coursier fetch --help`
	# If ex.name does not exist, that means the whole org needs to be excluded.
	ex_arg = f"{jar.org}:{jar.name}--{ex.org}:{ex.name or '*'}"
	local_exclude_args.append(ex_arg)