Dependency found that is not listed in sbt

[francisdb]

Somehow this action came up wit a dependency on protobuf-java but I'm unable to get sbt to report me that dependency

show Runtime/dependencyClasspathAsJars
show Compile/dependencyClasspathAsJars
show Test/dependencyClasspathAsJars

Any ideas where these could be coming from?

[jpsacha]

https://github.com/scalafx is getting dependency and security alert for jackson-databind but the project does not depend on it

[francisdb]

For my case I suspect some optional runtime / provided maven dependency but would be good if sbt could point me to it.

[MartinHH]

https://github.com/scalafx is getting dependency and security alert for jackson-databind but the project does not depend on it

I noticed that in one of my projects as well. I guess the reason is that Scala 3 itself depends on it - at least this search shows a dependency on jackson-databind 2.12.1 which is the version I am getting an alert for: https://github.com/lampepfl/dotty/search?q=jackson

[MartinHH]

I tested sbt-dependency-submission with a totally empty sbt project and got dependencies on both protobuf-java (3.7.0) and jackson-databind (2.12.1) (for Scala 3.1.3 and sbt 1.7.1) (along with some others, but those were the most interesting one because they both yielded dependabot security alerts).

Downgrading the Scala version to 2.13.8 lead to a drastic reduction of dependencies (only five were left, none of them yielding alerts).

[francisdb]

I can confirm that I also tested on a Scala 3 project

[ybasket]

I believe it's due to scaladoc on Scala 3, it brings Jackson & Flexmark. This plugin uses allDependencies from sbt which lists scaladoc (and hence its transitive dependencies). A possible (quick-)fix could be allow configuring the plugin to use libraryDependencies or to exclude certain libs from allDependencies (before the transitive deps are calculated).

[adpi2]

Thanks @ybasket. Indeed the Jackson & Flexmark dependencies are used by scaladoc on Scala 3.

There is a scope field in the dependencies of the Github Dependency Submission API, which can be runtime or development. It is set to development for the compiler, scaladoc and all their dependencies. Unfortunately this information is not yet shown to the user. That's probably because the API is still in beta and the Dependency view has not yet been re-designed.

Github recently announced a sope filter in the dependabot alerts, but for reasons I don't know, it does not seem to work with the Github dependency submissions.

Should we exclude those dependencies from the report? I would say no because the vulnerabilities in the dependencies of the build tool could be exploited. I think we should always take as much dependencies as possible even if it is not always easy to understand where they are coming from. Hopefully the Github team will make it easier some day.

About the vulnerabilities, they should be fixed in scaladox itself. In the meantime this is how you can fix them in your build:

1. If you don't use scaladoc, you can remove it from the list of dependencies:

lazy val foo = project.in(file(""))
  .settings(
    allDependencies -= "org.scala-lang" %% "scaladoc" % scalaVersion.value % "scala-doc-tool->default(compile)",
    Compile / doc / sources := Seq.empty // disable the `doc` task
  )

2. If you use scaladoc, you can update the dependencies

lazy val foo = project.in(file(""))
  .settings(
    allDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.13.3" % Configurations.ScalaDocTool
  )

[ybasket]

Should we exclude those dependencies from the report?

@adpi2 Would it be hard to make that decision configurable? I agree with your reasoning in that they should be included by default, but for the given GitHub circumstances and my situation at $WORK, having all those extra dependencies stops me from using the sbt-dependency-submission. The build is quite well-protected, so we care much more about the runtime dependencies.

PS: Thank your for the scaladoc removal snippets, that may help working around in the meantime.

[adpi2]

@ybasket If you have time to contribute on this, I would be happy to help.

[ybasket]

@ybasket If you have time to contribute on this, I would be happy to help.

I can't make any promises, but if I find time, I'll try to implement this.

[adpi2]

To get started, you can add a field in this file and consume here to filter some dependencies out. The difficulty would be to define what to filter out. Do you want to filter out all developement dependencies? Even the test dependencies? Maybe a flexible solution would be to take a set of configs to ignore (scala-tool, scala-doc-tool) as input.

[ybasket]

@adpi2 I tried implementing this and got stuck on how to pass the information along inside the sbt state (something I have no experience with). More concretely, it seems as the state isn't yet populated when the manifest task is run, hence the exclusions can't be extracted. Do you happen to have a hint on how to work around that? I opened a draft PR to share the current version I have, will probably take a bit until I find time again to work on it: #51

[adpi2]

Thanks @ybasket for looking into this. I left some comments in #51. I hope it helps.

[francisdb]

If you don't use scaladoc, you can remove it from the list of dependencies:

This removes the jackson-databind but the protobuf-java remains. Any idea how that one can be excluded? Can we request a dependecy tree for this allDependencies?

[adpi2]

I don't think it is possible to get a dependency tree for allDependencies easily. But I managed to add the dependencyTree task to the ScalaTool configuration with:

import sbt.librarymanagement.Configurations
import sbt.plugins.DependencyTreeSettings

inConfig(Configurations.ScalaTool)(DependencyTreeSettings.baseBasicReportingSettings)

sbt:example> ScalaTool / dependencyTree
[info] example:example_3:0.1.0-SNAPSHOT
[info]   +-org.scala-lang:scala3-compiler_3:3.0.0
[info]     +-org.jline:jline-reader:3.19.0
[info]     | +-org.jline:jline-terminal:3.19.0
[info]     | 
[info]     +-org.jline:jline-terminal-jna:3.19.0
[info]     | +-net.java.dev.jna:jna:5.3.1
[info]     | +-org.jline:jline-terminal:3.19.0
[info]     | 
[info]     +-org.jline:jline-terminal:3.19.0
[info]     +-org.scala-lang.modules:scala-asm:9.1.0-scala-1
[info]     +-org.scala-lang:scala3-interfaces:3.0.0
[info]     +-org.scala-lang:scala3-library_3:3.0.0 [S]
[info]     +-org.scala-lang:tasty-core_3:3.0.0
[info]     | +-org.scala-lang:scala3-library_3:3.0.0 [S]
[info]     | 
[info]     +-org.scala-sbt:compiler-interface:1.3.5
[info]       +-com.google.protobuf:protobuf-java:3.7.0
[info]       +-org.scala-sbt:util-interface:1.3.0

So it seems that protobuf-java is a transitive dependency of scala3-compiler via sbt's compiler-interface.

[francisdb]

I see this dependency is no longer part of the compiler-interface starting at 1.4.0 but I wonder if scala3 can update to that.

https://github.com/lampepfl/dotty/blob/36f0e09c6926f8824c65f0c7a2cd730f4d71e176/project/Dependencies.scala#L28-L29

[adpi2]

The Scala 3 compiler still depends on interface 1.3 to be compatible with Bloop < 1.5. See scala/scala3#10816.

But Bloop 1.5 now depends on Zinc 1.6 so we should be able to move this forward.

[rossabaker]

Would we expect modules-ignore: scaladoc_3 to work? I tried it in this demo repo, but I still see scaladoc_3 (and all the vulnerabilities that come with it) in my dependency report.

[adpi2]

Would we expect modules-ignore: scaladoc_3 to work?

No it should not, and it should warn that scaladoc_3 is not a valid module to ignore, which it does not. modules-ignore can only be used to prevent a project, on a Scala version, from submitting its dependencies, it does not exclude any dependency. In your example, the only "module" is modules-ignore-issue_3.

Adding a dependency-ignore field would be possible. However, to solve the specific issue of scaladoc bringing a lot of vulnerable dependencies, I am rather thinking of using a config-ignore setting, to ignore the entire scala-doc-tool config.

[adpi2]

Fixed by #73

It is now possible to add configs-ignore: scala-doc-tool to exclude all the ScalaDocTool dependencies.