missing transitive dependencies are not detected anymore

[dadoonet]

When you add or modify a dependency in a plugin, for example, changing azure SDK from 0.7.0 to 0.9.0, you need with gradle to explicitly declare all transitive dependencies.

For example in #15232, upgrading from com.microsoft.azure:azure-core:0.7.0 to com.microsoft.azure:azure-core:0.9.0 pulls another transitive dependency as we can see when we run gradle dependencies:

_transitive_com.microsoft.azure:azure-core:0.9.0
\--- com.microsoft.azure:azure-core:0.9.0
     +--- org.apache.httpcomponents:httpclient:4.3.6
     |    +--- org.apache.httpcomponents:httpcore:4.3.3
     |    +--- commons-logging:commons-logging:1.1.3
     |    \--- commons-codec:commons-codec:1.6 -> 1.10
     +--- commons-codec:commons-codec:1.10
     +--- commons-lang:commons-lang:2.6
     +--- javax.mail:mail:1.4.5
     |    \--- javax.activation:activation:1.1
     +--- javax.inject:javax.inject:1
     +--- com.sun.jersey:jersey-client:1.13
     |    \--- com.sun.jersey:jersey-core:1.13
     \--- com.sun.jersey:jersey-json:1.13
          +--- org.codehaus.jettison:jettison:1.1
          |    \--- stax:stax-api:1.0.1
          +--- com.sun.xml.bind:jaxb-impl:2.2.3-1
          |    \--- javax.xml.bind:jaxb-api:2.2.2
          |         +--- javax.xml.stream:stax-api:1.0-2
          |         \--- javax.activation:activation:1.1
          +--- org.codehaus.jackson:jackson-core-asl:1.9.2
          +--- org.codehaus.jackson:jackson-mapper-asl:1.9.2
          |    \--- org.codehaus.jackson:jackson-core-asl:1.9.2
          +--- org.codehaus.jackson:jackson-jaxrs:1.9.2
          |    +--- org.codehaus.jackson:jackson-core-asl:1.9.2
          |    \--- org.codehaus.jackson:jackson-mapper-asl:1.9.2 (*)
          +--- org.codehaus.jackson:jackson-xc:1.9.2
          |    +--- org.codehaus.jackson:jackson-core-asl:1.9.2
          |    \--- org.codehaus.jackson:jackson-mapper-asl:1.9.2 (*)
          \--- com.sun.jersey:jersey-core:1.13

The new dependency is commons-lang:commons-lang:2.6.

As we did not include it previously in gradle config file, it's never packaged within the final ZIP file and never tested by the license_checker.

When back porting to 2.x, 2.x is using transitive maven dependencies so it's automatically added to the build and to the final ZIP file. That means also that the license_checker catch this:

main:
[license check] Running license check
     [exec] LICENSE DIR: /Users/dpilato/Documents/Elasticsearch/dev/es-2.x/elasticsearch/plugins/cloud-azure/licenses/
     [exec] SOURCE: /Users/dpilato/Documents/Elasticsearch/dev/es-2.x/elasticsearch/plugins/cloud-azure/target/releases/cloud-azure-2.2.0-SNAPSHOT.zip
     [exec] IGNORE: cloud-azure-2.2.0-SNAPSHOT
     [exec] commons-codec-1.10.jar.sha1: SHA is missing
     [exec] commons-io-2.4.jar.sha1: SHA is missing
     [exec] commons-lang-2.6.jar.sha1: SHA is missing
     [exec] Extra SHA files present for: commons-codec-1.6.jar.sha1
     [exec] Extra LICENCE file present: commons-codec
     [exec] Extra NOTICE file present: commons-codec

We should try to:

• have a gradle task which checks transitive dependencies vs the ones explicitly added in gradle file
  Having an output like the following would be awesome to copy and paste the result in gradle file :):

Missing deps:
  compile 'commons-lang:commons-lang:2.6'

• may be WARN when we build

[clintongormley]

In master we don't use transitive dependencies.

[dadoonet]

Reopening the issue as per the discussion we had on FixIt Friday session about #21373.

The goal is to have a gradle task which now detects:

• when a transitive dependency differs from the version one of our dependencies depends on
• when a transitive dependency is missing from the list of dependencies we are manually declaring

This task could be called as part of the check task and can fail the build.

Bonus points:

• be able to provide a list of jars/expected version/actual version which we can ignore (and not fail the build). Note that expected version and actual version are important here as we want to ask ourselves anytime we update a dependency in our project.
• provide a list of what has to be changed in the gradle build file

[dadoonet]

This PR #27361 is an example of why having something which detects missing/outdated deps is important IMO.

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

I think the third party audit that among other things looks for missing classes will detect missing transitive dependencies. Build scans are also enabled on all CI builds or available on the command line with --scan and will show a nice view of dependencies and how the resolution worked and why. Feel free to re-open if you think this is not fully covered.