Reachability: custom jars (#1382)

fossas · Mar 5, 2024 · b5f97eb · b5f97eb
1 parent 6775077
commit b5f97eb
Show file tree

Hide file tree

Showing 20 changed files with 359 additions and 84 deletions.
diff --git a/Changelog.md b/Changelog.md
@@ -1,51 +1,54 @@
 # FOSSA CLI Changelog
 
+## v3.9.8
+- Reachability: Users may now provide custom locations for the JAR files emitted by projects and used for reachability analysis ([#1382](https://github.com/fossas/fossa-cli/pull/1382)).
+
 ## v3.9.7
 - Add preflight permission checks to validate token type, subscription type, project permissions, and release group permissions [#1383](https://github.com/fossas/fossa-cli/pull/1383)
 
 ## v3.9.6
 - Add debug logs for build warnings in `analyze` commands [#1386](https://github.com/fossas/fossa-cli/pull/1386)
 
 ## v3.9.5
-- Maven: Fix hanging maven analysis [#1381](https://github.com/fossas/fossa-cli/pull/1381)
+- Maven: Fix hanging maven analysis ([#1381](https://github.com/fossas/fossa-cli/pull/1381)).
 
 ## v3.9.4
-- Reachability: Includes reachability analysis in scan summary [#1379](https://github.com/fossas/fossa-cli/pull/1379)
+- Reachability: Includes reachability analysis in scan summary ([#1379](https://github.com/fossas/fossa-cli/pull/1379)).
 
 ## v3.9.3
-- Update error structure [#1364](https://github.com/fossas/fossa-cli/pull/1364)
+- Update error structure ([#1364](https://github.com/fossas/fossa-cli/pull/1364)).
 
 ## v3.9.2
-- Maven: Adds reachability analysis [#1372](https://github.com/fossas/fossa-cli/pull/1377)
-- Gradle: Adds reachability analysis [#1377](https://github.com/fossas/fossa-cli/pull/1377)
+- Maven: Adds reachability analysis ([#1372](https://github.com/fossas/fossa-cli/pull/1377)).
+- Gradle: Adds reachability analysis ([#1377](https://github.com/fossas/fossa-cli/pull/1377)).
 
 ## v3.9.1
-- `--detect-dynamic`: Safely ignores scenarios in ldd output parsing where we run into not found error. ([#1376](https://github.com/fossas/fossa-cli/pull/1376)) 
+- `--detect-dynamic`: Safely ignores scenarios in ldd output parsing where we run into not found error ([#1376](https://github.com/fossas/fossa-cli/pull/1376)).
 
 ## v3.9.0
-- Emits a warning instead of an error when no analysis targets are found ([#1375](https://github.com/fossas/fossa-cli/pull/1375))
+- Emits a warning instead of an error when no analysis targets are found ([#1375](https://github.com/fossas/fossa-cli/pull/1375)).
 
 ## 3.8.37
 
-- Container Scans: Bugfix for some registry scans that fail with an STM error. ([#1370](https://github.com/fossas/fossa-cli/pull/1370))
+- Container Scans: Bugfix for some registry scans that fail with an STM error. ([#1370](https://github.com/fossas/fossa-cli/pull/1370)).
 
 ## v3.8.36
-- `fossa feedback`: Allow users to provide feedback on their cli experience ([#1368](https://github.com/fossas/fossa-cli/pull/1368))
-- Add preflight checks to validate API key, connection to FOSSA app, and ability to write to temp directory in relevant commands
+- `fossa feedback`: Allow users to provide feedback on their cli experience ([#1368](https://github.com/fossas/fossa-cli/pull/1368)).
+- Add preflight checks to validate API key, connection to FOSSA app, and ability to write to temp directory in relevant commands.
 
 
 ## v3.8.35
 - Running `fossa analyze --detect-vendored` no longer fails if there are no detected vendored dependencies ([#1373](https://github.com/fossas/fossa-cli/pull/1373)).
 
 ## v3.8.34
-- Add color and update formatting in cli help commands ([#1367](https://github.com/fossas/fossa-cli/pull/1367))
+- Add color and update formatting in cli help commands ([#1367](https://github.com/fossas/fossa-cli/pull/1367)).
 
 ## v3.8.33
-- Removes warnings and tracebacks to stderr [#1358](https://github.com/fossas/fossa-cli/pull/1358)
+- Removes warnings and tracebacks to stderr ([#1358](https://github.com/fossas/fossa-cli/pull/1358)).
 
 ## v3.8.32
 
-- Options: Add a `--static-only-analysis` option. ([#1362](https://github.com/fossas/fossa-cli/pull/1362))
+- Options: Add a `--static-only-analysis` option ([#1362](https://github.com/fossas/fossa-cli/pull/1362)).
 
 ## v3.8.31
 
@@ -55,33 +58,33 @@
 
 ## v3.8.30
 
-- Fix an issue with long-option syntax for older versions of `sbt` ([#1356](https://github.com/fossas/fossa-cli/pull/1356))
-- Debug: add more logging for debugging missing dependencies. ([#1360](https://github.com/fossas/fossa-cli/pull/1360))
+- Fix an issue with long-option syntax for older versions of `sbt` ([#1356](https://github.com/fossas/fossa-cli/pull/1356)).
+- Debug: add more logging for debugging missing dependencies ([#1360](https://github.com/fossas/fossa-cli/pull/1360)).
 
 ## v3.8.29
-- Prevents showing SCM warnings in fossa analyze, test, and report [#1354](https://github.com/fossas/fossa-cli/pull/1354)
-- Pathfinder: Pathfinder has been deprecated and removed. ([#1350](https://github.com/fossas/fossa-cli/pull/1350))
+- Prevents showing SCM warnings in fossa analyze, test, and report ([#1354](https://github.com/fossas/fossa-cli/pull/1354)).
+- Pathfinder: Pathfinder has been deprecated and removed ([#1350](https://github.com/fossas/fossa-cli/pull/1350)).
 
 ## v3.8.28
-- VSI: no longer reports paths inside of extracted archives with the `!_fossa.virtual_!` literal [#1345](https://github.com/fossas/fossa-cli/pull/1345)
+- VSI: no longer reports paths inside of extracted archives with the `!_fossa.virtual_!` literal ([#1345](https://github.com/fossas/fossa-cli/pull/1345)).
 
 ## v3.8.27
-- Maven: Fix a bug that broke maven analysis if the build directory was in a non-standard location ([#1343](https://github.com/fossas/fossa-cli/pull/1343))
+- Maven: Fix a bug that broke maven analysis if the build directory was in a non-standard location ([#1343](https://github.com/fossas/fossa-cli/pull/1343)).
 
 ## v3.8.26
-- Maven: add support for maven submodule filtering [#1339](https://github.com/fossas/fossa-cli/pull/1339)
+- Maven: add support for maven submodule filtering ([#1339](https://github.com/fossas/fossa-cli/pull/1339)).
 
 ## v3.8.25
-- Maven: add support for maven scope filtering ([#1331](https://github.com/fossas/fossa-cli/pull/1331))
-- `fossa init`: adds new `fossa init` command which creates `.fossa.yml.example`, and `fossa-deps.yml.example` file. ([#1323](https://github.com/fossas/fossa-cli/pull/1323))
+- Maven: add support for maven scope filtering ([#1331](https://github.com/fossas/fossa-cli/pull/1331)).
+- `fossa init`: adds new `fossa init` command which creates `.fossa.yml.example`, and `fossa-deps.yml.example` file. ([#1323](https://github.com/fossas/fossa-cli/pull/1323)).
 
 ## v3.8.24
 
-- Python: use `pip` to determine transitive dependencies for setuptool projects that contain a req*.txt or setup.py file. ([#1334](https://github.com/fossas/fossa-cli/pull/1334))
-- Container Scanning: warn and exclude rpm packages that are missing attributes. ([#1335](https://github.com/fossas/fossa-cli/pull/1335))
+- Python: use `pip` to determine transitive dependencies for setuptool projects that contain a req*.txt or setup.py file. ([#1334](https://github.com/fossas/fossa-cli/pull/1334)).
+- Container Scanning: warn and exclude rpm packages that are missing attributes ([#1335](https://github.com/fossas/fossa-cli/pull/1335)).
 
 ## v3.8.23
-- Custom License Scans: Support full-file uploads for custom license scans ([#1333](https://github.com/fossas/fossa-cli/pull/1333))
+- Custom License Scans: Support full-file uploads for custom license scans ([#1333](https://github.com/fossas/fossa-cli/pull/1333)).
 
 ## v3.8.22
 - path: adds path dependency scanning functionality. ([#1327](https://github.com/fossas/fossa-cli/pull/1327))

diff --git a/docs/features/vuln_reachability.md b/docs/features/vuln_reachability.md
@@ -2,16 +2,16 @@
 
 ### What is Reachability?
 
-Reachability Analysis is a security offering designed to enhance FOSSA's security analysis by providing context on vulnerable packages. It alleviates the constraints of traditional CVE assessments through the static analysis of application and dependency code, confirming the presence of vulnerable call paths. 
+Reachability Analysis is a security offering designed to enhance FOSSA's security analysis by providing context on vulnerable packages. It alleviates the constraints of traditional CVE assessments through the static analysis of application and dependency code, confirming the presence of vulnerable call paths.
 
 ### Limitations
 
-- Reachability currently supports all Maven and Gradle projects dynamically analyzed by FOSSA CLI. 
+- Reachability currently supports all Maven and Gradle projects dynamically analyzed by FOSSA CLI.
 - The target jar of the project must exist, prior to the analysis. If the jar artifact is not present, or FOSSA CLI fails to
 associate this jar with project, FOSSA CLI will not perform reachability analysis.
 - Reachability requires that `java` is present in PATH, and `java` version must be greater than `1.8` (jdk8+).
 
-For example, 
+For example,
 - if you are using maven, you should run `mvn package` to ensure jar artifact exists, prior to running `fossa analyze`
 - if you are using gradle, you should run `gradlew build` to ensure jar artifact exists, prior to running `fossa analyze`
 
@@ -23,6 +23,50 @@ For Maven projects, FOSSA CLI performs an analysis to infer dependencies. If FOS
 
 For Gradle projects, FOSSA CLI invokes `./gradlew -I jsonpaths.gradle jsonPaths`. Where [jsonpaths.gradle](./../../scripts/jarpaths.gradle) is gradle script, which uses `java` plugin, and `jar` task associated with gradle to infer path of the built jar file. If neither of those are present, FOSSA CLI won't be able to identify jar artifacts for analysis.
 
+### Custom JAR locations
+
+For both Gradle and Maven, it is possible to choose different locations for output JAR files, or to move them before FOSSA CLI runs.
+For these cases, you can configure the locations directly.
+
+First, run `fossa list-targets` inside your project. For example:
+```sh
+❯ fossa list-targets
+Found project: gradle@./
+Found target: gradle@./::app
+```
+
+This output specifies that a Gradle project was found at the root of the current directory (at `./`).
+Note for which project(s) you wish to provide the paths to the built JAR files, we'll need these later.
+
+Now, create a `.fossa.yml` file if one does not already exist ([more information here](../references/files/fossa-yml.md)).
+Add the following keys to it:
+```yml
+reachability:
+  jvmOutputs:
+```
+
+Inside the `jvmOutputs` object, create a map that describes, for each project path (noted down earlier) the paths to the JAR files that the project output.
+
+For example, let's say that the Gradle project at the root of my scan directory (at `./`) produces a JAR file named `app.jar`, and that I configured Gradle such that when it builds the project that `app.jar` file is also written to the root of my scan directory (so its path is `./app.jar`, relative to the scan root).
+My config file would look like this:
+
+```yml
+reachability:
+  jvmOutputs:
+    './':
+    - './app.jar'
+```
+
+You can also provide the absolute paths to make this more clear, if desired. For example, if my project was at `~/projects/example-project` and I wanted to use absolute paths in the config file, the file would look like this:
+```yml
+reachability:
+  jvmOutputs:
+    '/Users/me/projects/example-project':
+    - '/Users/me/projects/example-project/app.jar'
+```
+
+With this configuration, when FOSSA CLI analyzes a Maven or Gradle project at the specified path (in this case `/Users/me/projects/example-project`) instead of attempting to query the build tool for the location of the built JAR files it will instead use the values provided for that path (in this case `/Users/me/projects/example-project/app.jar`).
+
 ### How do I debug reachability from `fossa-cli`?
 
 ```bash
@@ -75,12 +119,12 @@ cat fossa.debug.json | jq '.bundleReachabilityEndpoint'
 }
 ```
 
-FOSSA CLI uses [jar-callgraph-1.0.0.jar](../../scripts/jar-callgraph-1.0.0.jar) to infer call path edges. 
+FOSSA CLI uses [jar-callgraph-1.0.0.jar](../../scripts/jar-callgraph-1.0.0.jar) to infer call path edges.
 FOSSA CLI uses `java -jar jar-callgraph-1.0.0.jar ./path/to/your/build.jar` command to record edges from
 the your target jar. If you are running into issues with reachability, please confirm that you can execute
 `java -jar jar-callgraph-1.0.0.jar ./path/to/your/build.jar` on your environment.
 
-<!-- 
+<!--
 ## How do I debug reachability from endpoint?
 
 ```bash
@@ -109,7 +153,7 @@ Likewise, you can also inspect analysis done in datadog, by looking at logs asso
 performs reachability analysis as part of provided build (all variants of provided builds).
 -->
 
-## F.A.Q. 
+## F.A.Q.
 
 1. What data from my codebase is uploaded to endpoint?
 
@@ -174,4 +218,3 @@ Reachability analysis
 | skipped (partial graph)          | Project has partial dependency graph (e.g. missing transitive dependencies)  |
 | skipped (not supported)          | Project is not supported for reachability analysis                           |
 | skipped (no dependency analysis) | Project's dependencies were not analyzed, so reachability cannot be computed |
-
diff --git a/docs/references/files/fossa-yml.v3.schema.json b/docs/references/files/fossa-yml.v3.schema.json
@@ -397,6 +397,23 @@
                 "$ref": "#/$defs/grepDefinition"
             }
         },
+        "reachability": {
+          "type": "object",
+          "description": "Controls the Reachability computation functionality",
+          "properties": {
+            "jvmOutputs": {
+              "type": "object",
+              "description": "Manually specify the list of JAR files output by each discovered Maven or Gradle project",
+              "additionalProperties": {
+                "type": "array",
+                "items": {
+                  "type": "string",
+                  "description": "The path to a JAR file output by the project"
+                }
+              }
+            }
+          }
+        },
         "ignoreOrgWideCustomLicenseScanConfigs": {
             "type": "boolean",
             "default": false,
@@ -477,4 +494,4 @@
     "required": [
         "version"
     ]
-}
+}
diff --git a/integration-test/Analysis/FixtureUtils.hs b/integration-test/Analysis/FixtureUtils.hs
@@ -10,7 +10,7 @@ module Analysis.FixtureUtils (
   TestC,
   performDiscoveryAndAnalyses,
   getArtifact,
-  testRunnerWithLogger,
+  testRunner,
   withResult,
 ) where
 
@@ -122,8 +122,8 @@ type TestC m =
     $ StackC
     $ IgnoreTelemetryC m
 
-testRunnerWithLogger :: TestC IO a -> FixtureEnvironment -> IO (Result a)
-testRunnerWithLogger f env =
+testRunner :: TestC IO a -> FixtureEnvironment -> IO (Result a)
+testRunner f env =
   f
     & runExecIOWithinEnv env
     & runReadFSIO
@@ -160,10 +160,10 @@ performDiscoveryAndAnalyses targetDir AnalysisTestFixture{..} = do
   _ <- sendIO $ runCmd environment buildCmd
 
   -- Perform discovery
-  discoveryResult <- sendIO $ testRunnerWithLogger (discover targetDir) environment
+  discoveryResult <- sendIO $ testRunner (discover targetDir) environment
   withResult discoveryResult $ \_ dps ->
     for dps $ \dp -> do
-      analysisResult <- sendIO $ testRunnerWithLogger (ignoreDebug $ analyzeProject (projectBuildTargets dp) (projectData dp)) environment
+      analysisResult <- sendIO $ testRunner (ignoreDebug $ analyzeProject (projectBuildTargets dp) (projectData dp)) environment
       withResult analysisResult $ \_ dr -> pure (dp, dr)
   where
     runCmd :: FixtureEnvironment -> Maybe (Command) -> IO ()

diff --git a/integration-test/Reachability/UploadSpec.hs b/integration-test/Reachability/UploadSpec.hs
@@ -3,7 +3,7 @@
 
 module Reachability.UploadSpec (spec) where
 
-import Analysis.FixtureUtils (FixtureEnvironment (..), TestC, testRunnerWithLogger, withResult)
+import Analysis.FixtureUtils (FixtureEnvironment (..), TestC, testRunner, withResult)
 import App.Fossa.Analyze.Project (ProjectResult (..))
 import App.Fossa.Analyze.Types (
   DiscoveredProjectIdentifier (..),
@@ -15,15 +15,18 @@ import App.Fossa.Reachability.Types (
   CallGraphAnalysis (..),
   ContentRef (..),
   ParsedJar (..),
+  ReachabilityConfig (..),
   SourceUnitReachability (..),
  )
 import App.Fossa.Reachability.Upload (
   analyzeForReachability,
   callGraphOf,
   onlyFoundUnits,
  )
+import Control.Carrier.Reader (ReaderC, runReader)
 import Data.ByteString.Lazy qualified as LB
 import Data.Foldable (for_)
+import Data.Map qualified as Map
 import Data.String.Conversion (toText)
 import Data.Text (Text)
 import Data.Text.Encoding qualified as TL
@@ -42,6 +45,7 @@ import Path (
 import Path.IO qualified as PIO
 import Test.Hspec (Spec, describe, it, runIO, shouldBe)
 import Text.RawString.QQ (r)
+import Type.Operator (type ($))
 import Types (
   DiscoveredProjectType (MavenProjectType),
   GraphBreadth (..),
@@ -60,7 +64,10 @@ java21 :: FixtureEnvironment
 java21 = NixEnv ["jdk21"]
 
 run :: FixtureEnvironment -> TestC IO a -> IO (Result a)
-run env act = testRunnerWithLogger act env
+run env act = testRunner act env
+
+runConf :: FixtureEnvironment -> ReachabilityConfig -> (ReaderC ReachabilityConfig $ TestC IO) a -> IO (Result a)
+runConf env conf act = testRunner (runReader conf act) env
 
 spec :: Spec
 spec = describe "Reachability" $ do
@@ -86,7 +93,7 @@ spec = describe "Reachability" $ do
     it "should retrieve call graph" $ do
       let (dpi, dps) = mavenCompleteScan projDir
       let expected = SourceUnitReachabilityFound dpi (Success [] (mavenCompleteScanUnit projDir jarFile))
-      resp <- run java8 $ callGraphOf dps
+      resp <- runConf java8 mempty $ callGraphOf dps
       withResult resp $ \_ res -> res `shouldBe` expected
 
   describe "analyzeForReachability" $ do
@@ -96,12 +103,32 @@ spec = describe "Reachability" $ do
     it "should return analyzed reachability unit" $ do
       let (_, dps) = mavenCompleteScan projDir
       let expected = [mavenCompleteScanUnit projDir jarFile]
-      analyzed <- run java8 $ analyzeForReachability [dps]
+      analyzed <- runConf java8 mempty $ analyzeForReachability [dps]
+      withResult analyzed $ \_ analyzed' -> (onlyFoundUnits analyzed') `shouldBe` expected
+
+  describe "user provided jar" $ do
+    projDir <- (</> sampleMavenProjectMissingOutputJarDir) <$> runIO PIO.getCurrentDir
+    jarFile <- (</> sampleMavenProjectJar) <$> runIO PIO.getCurrentDir
+
+    it "should fail to analyze reachability when jar is missing" $ do
+      let (_, dps) = mavenCompleteScan projDir
+      let expected = [mavenEmptyScanUnit projDir]
+      analyzed <- runConf java8 mempty $ analyzeForReachability [dps]
+      withResult analyzed $ \_ analyzed' -> (onlyFoundUnits analyzed') `shouldBe` expected
+
+    it "should succeed analyzing the project when a missing jar is manually specified" $ do
+      let conf = ReachabilityConfig $ Map.fromList [(projDir, [jarFile])]
+      let (_, dps) = mavenCompleteScan projDir
+      let expected = [mavenCompleteScanUnit projDir jarFile]
+      analyzed <- runConf java8 conf $ analyzeForReachability [dps]
       withResult analyzed $ \_ analyzed' -> (onlyFoundUnits analyzed') `shouldBe` expected
 
 sampleMavenProjectDir :: Path Rel Dir
 sampleMavenProjectDir = $(mkRelDir "test/Reachability/testdata/maven-default/")
 
+sampleMavenProjectMissingOutputJarDir :: Path Rel Dir
+sampleMavenProjectMissingOutputJarDir = $(mkRelDir "test/Reachability/testdata/maven-default-missing-jar/")
+
 sampleMavenProjectJar :: Path Rel File
 sampleMavenProjectJar = $(mkRelFile "test/Reachability/testdata/maven-default/target/project-1.0.0.jar")
 
@@ -117,6 +144,9 @@ mavenCompleteScanUnit projDir jarFile =
         (ContentRaw sampleJarParsedContent')
     ]
 
+mavenEmptyScanUnit :: Path Abs Dir -> SourceUnitReachability
+mavenEmptyScanUnit projDir = mkReachabilityUnit projDir []
+
 mkDiscoveredProjectScan :: DiscoveredProjectType -> Path Abs Dir -> GraphBreadth -> (DiscoveredProjectIdentifier, DiscoveredProjectScan)
 mkDiscoveredProjectScan projectType dir breadth =
   ( dpi

diff --git a/src/App/Fossa/Analyze.hs b/src/App/Fossa/Analyze.hs
@@ -400,7 +400,7 @@ analyze cfg = Diag.context "fossa-analyze" $ do
     (False, _) -> traverse (withPathDependencyNudge includeAll) filteredProjects
   logDebug $ "Filtered projects with path dependencies: " <> pretty (show filteredProjects')
 
-  reachabilityUnitsResult <- analyzeForReachability projectScans
+  reachabilityUnitsResult <- Diag.context "reachability analysis" . runReader (Config.reachabilityConfig cfg) $ analyzeForReachability projectScans
   let reachabilityUnits = onlyFoundUnits reachabilityUnitsResult
 
   let analysisResult = AnalysisScanResult projectScans vsiResults binarySearchResults manualSrcUnits dynamicLinkedResults maybeLernieResults reachabilityUnitsResult