NuGet component with space breaks analyzer #3688

peterloron · 2024-05-09T22:56:03Z

Current Behavior

A CDXJSON1.4 SBOM from an internal docker image was uploaded to DT 4.11. The SBOM contains a component which is a NuGet package. When analysis is attempted, an Illegal Character exception is thrown. I suspect that the space (%20) character is not being handled properly. Here is a sample section of the SBOM:

    {
      "bom-ref": "pkg:nuget/Simple%[email protected]?package-id=f17979bac9f7e205",
      "type": "library",
      "name": "Simple Launcher",
      "version": "1.1.0.14",
      "cpe": "cpe:2.3:a:Simple_Launcher:Simple_Launcher:1.1.0.14:*:*:*:*:*:*:*",
      "purl": "pkg:nuget/Simple%[email protected]",
      "properties": [
        {
          "name": "syft:package:foundBy",
          "value": "dotnet-portable-executable-cataloger"
        },
        { "name": "syft:package:language", "value": "dotnet" },
        { "name": "syft:package:type", "value": "dotnet" },
        {
          "name": "syft:package:metadataType",
          "value": "dotnet-portable-executable-entry"
        },
        {
          "name": "syft:location:0:layerID",
          "value": "sha256:9ce7e12c92bbf09aaaa5aaa621cff892fc3ce1c1fdc2eb31ce562790df8be75c"
        },
        {
          "name": "syft:location:0:path",
          "value": "/usr/local/lib/python3.8/dist-packages/distlib/t64.exe"
        }
      ]
    },

During the analysis, I saw the following error in the log:

2024-05-09 13:06:07,188 ERROR [NugetMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 45: https://api.nuget.org/v3-flatcontainer/simple launcher/index.json
	at java.base/java.net.URI$Parser.fail(Unknown Source)
	at java.base/java.net.URI$Parser.checkChars(Unknown Source)
	at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
	at java.base/java.net.URI$Parser.parse(Unknown Source)
	at java.base/java.net.URI.<init>(Unknown Source)
	at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
	at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:105)
	at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:108)
	at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:179)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:123)
	at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
	at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
	at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:128)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:93)
	at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2024-05-09 13:06:07,190 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:nuget/Simple%[email protected] : java.lang.NullPointerException: Cannot invoke "org.apache.http.client.methods.CloseableHttpResponse.getStatusLine()" because "response" is null
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException: Cannot invoke "org.apache.http.client.methods.CloseableHttpResponse.getStatusLine()" because "response" is null
	at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:124)
	at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:179)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:123)
	at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
	at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
	at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:128)
	at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:93)
	at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.http.client.methods.CloseableHttpResponse.getStatusLine()" because "response" is null
	at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:109)
	... 12 common frames omitted
2024-05-09 13:06:08,418 INFO [PolicyEngine] Policy analysis complete

Steps to Reproduce

Upload SBOM with the CPE/PURL shown above.

Expected Behavior

Dependencytrack properly handles the analysis.

Dependency-Track Version

4.11.0

Dependency-Track Distribution

Container Image

Database Server

PostgreSQL

Database Server Version

No response

Browser

Apple Safari

Checklist

I have read and understand the contributing guidelines
I have checked the existing issues for whether this defect was already reported

The text was updated successfully, but these errors were encountered:

nscuro · 2024-05-10T09:48:02Z

Similar issue was fixed for NPM in v4.11: #3456

laurentiu-ghergu · 2024-06-03T14:29:02Z

I am interested in working on this one. Please assign me if possible. Thanks.

lameirat · 2024-07-23T14:07:14Z

Hi, thank you for your contribute. There is any new about this issue?

z1atk0 · 2024-08-13T11:07:22Z

Same problem here with Maven (https://repo1.maven.org/maven2/), NuGet (https://api.nuget.org/) and Python (https://pypi.org/). URLs passed to MetaAnalyzers do not get URL-encoded, so any URL with a space character (" ") causes a Java error, and the project's risk analysis is aborted.

Maven example:

2024-08-13 08:58:19,086 ERROR [MavenMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 57: https://repo1.maven.org/maven2/org/ops4j/pax/url/mvn/mvn; singleton:=true/maven-metadata.xml
        at java.base/java.net.URI$Parser.fail(Unknown Source)
        at java.base/java.net.URI$Parser.checkChars(Unknown Source)
        at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
        at java.base/java.net.URI$Parser.parse(Unknown Source)
        at java.base/java.net.URI.<init>(Unknown Source)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
        at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
        at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:81)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:58:19,086 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:maven/org.ops4j.pax.url.mvn/mvn%3B%20singleton%3A%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
        at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:110)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null

NuGet example:

2024-08-13 08:33:49,448 ERROR [NugetMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 43: https://api.nuget.org/v3-flatcontainer/mono common language infrastructure/index.json
        at java.base/java.net.URI$Parser.fail(Unknown Source)
        at java.base/java.net.URI$Parser.checkChars(Unknown Source)
        at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
        at java.base/java.net.URI$Parser.parse(Unknown Source)
        at java.base/java.net.URI.<init>(Unknown Source)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
        at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:108)
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:33:49,448 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:nuget/Mono%20Common%20Language%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:124)
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null

Python/PyPi example:

2024-08-13 08:59:54,474 ERROR [PypiMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 27: https://pypi.org/pypi/magic file extensions/json
        at java.base/java.net.URI$Parser.fail(Unknown Source)
        at java.base/java.net.URI$Parser.checkChars(Unknown Source)
        at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
        at java.base/java.net.URI$Parser.parse(Unknown Source)
        at java.base/java.net.URI.<init>(Unknown Source)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
        at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
        at org.dependencytrack.tasks.repositories.PypiMetaAnalyzer.analyze(PypiMetaAnalyzer.java:75)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:59:54,474 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:pypi/magic%20file%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
        at org.dependencytrack.tasks.repositories.PypiMetaAnalyzer.analyze(PypiMetaAnalyzer.java:107)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null

Note that in all CacheStampedeBlocker warnings following the MetaAnalyzer error, the reported key is actually properly URL-encoded. I'm not a programmer, but wouldn't it make sense to simply URL-encode any URL before passing it on to its respective MetaAnalyzer? 🤔 🤷‍♂️

We are currently using syft-1.8.0 and DependencyTrack-4.9.1, so if this problem is already fixed in a later version please let me know (but then this issue would already be closed, I guess 😇).

Thanks for listening! 🙂
Thomas

nscuro · 2024-08-13T11:19:21Z

Note that in all CacheStampedeBlocker warnings following the MetaAnalyzer error, the reported key is actually properly URL-encoded.

The PURL specification requires URL encoding of special characters. However, when accessing individual parts of the PURL (say the package's name), you URL-decode to get the original value.

Analyzers will decode the PURL in order to assemble repository URLs, whereas warning / error logs will simply use the entire PURL, that's the reason why you're seeing this difference.

I'm not a programmer, but wouldn't it make sense to simply URL-encode any URL before passing it on to its respective MetaAnalyzer? 🤔 🤷‍♂️

Yes. I guess the original implementation assumed that PURLs would contain valid namespaces and names according to the respective ecosystem's conventions, but in the examples you shared that is clearly not the case.

To my knowledge, neither Maven, nor Python, nor NuGet packages are allowed to contain spaces.

We labeled this issue as good first issue since it's easy to resolve. Seems like @laurentiu-ghergu did not end up working on it. If no one picks it up prior to the 4.12 release, I'll do it.

PURLs containing special characters such as spaces could render the constructed repository URL invalid. Fixes DependencyTrack#3688 Signed-off-by: nscuro <[email protected]>

github-actions · 2024-10-01T10:04:57Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

peterloron added defect Something isn't working in triage labels May 9, 2024

nscuro added p2 Non-critical bugs, and features that help organizations to identify and reduce risk good first issue Good for newcomers size/S Small effort and removed in triage labels May 10, 2024

nscuro assigned laurentiu-ghergu Jun 3, 2024

nscuro unassigned laurentiu-ghergu Aug 13, 2024

nscuro added this to the 4.12 milestone Aug 13, 2024

nscuro mentioned this issue Aug 31, 2024

Ensure URL-encoding of repository URL path segments #4107

Merged

2 tasks

nscuro closed this as completed in #4107 Aug 31, 2024

nscuro closed this as completed in 31cc33e Aug 31, 2024

github-actions bot locked as resolved and limited conversation to collaborators Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NuGet component with space breaks analyzer #3688

NuGet component with space breaks analyzer #3688

peterloron commented May 9, 2024

nscuro commented May 10, 2024

laurentiu-ghergu commented Jun 3, 2024

lameirat commented Jul 23, 2024

z1atk0 commented Aug 13, 2024

nscuro commented Aug 13, 2024

github-actions bot commented Oct 1, 2024

NuGet component with space breaks analyzer #3688

NuGet component with space breaks analyzer #3688

Comments

peterloron commented May 9, 2024

Current Behavior

Steps to Reproduce

Expected Behavior

Dependency-Track Version

Dependency-Track Distribution

Database Server

Database Server Version

Browser

Checklist

nscuro commented May 10, 2024

laurentiu-ghergu commented Jun 3, 2024

lameirat commented Jul 23, 2024

z1atk0 commented Aug 13, 2024

nscuro commented Aug 13, 2024

github-actions bot commented Oct 1, 2024