Composer Meta Analyzer Exceptions

[valentijnscholten]

Current Behavior:

Log is full of Exceptions of the Composer Meta Analyzer:

2022-11-03 12:37:32,722 ERROR [LoggableUncaughtExceptionHandler] An unknown error occurred in an asynchronous event or notification thread
kong.unirest.json.JSONException: JSONObject["magento/adobe-ims"] not found.
        at kong.unirest.json.JSONObject.getProperty(JSONObject.java:963)
        at kong.unirest.json.JSONObject.getJSONObject(JSONObject.java:304)
        at org.dependencytrack.tasks.repositories.ComposerMetaAnalyzer.analyze(ComposerMetaAnalyzer.java:101)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:95)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:60)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:101)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

Steps to Reproduce:

Expected Behavior:

No exceptions or another log level for non critical errors

Environment:

• Dependency-Track Version: 4.6.1
• Distribution: Docker
• BOM Format & Version: n/a
• Database Server: PostgreSQL
• Browser: n/a

Additional Details:

Source code: https://github.com/DependencyTrack/dependency-track/blob/master/src/main/java/org/dependencytrack/tasks/repositories/ComposerMetaAnalyzer.java#L101

[valentijnscholten]

I see about 40 occurrences of this error on a daily basis. All related to magento modules. I did check a couple of the responses. I attached the responses (with a .txt extension as github won't allow me to upload .json files).

So far I see two cases:

• I get back a list of different modules which seem to be replacing the module that we requested (our module occurs in the replaces object)
• the module seems have been moved to a different namespace. In some cases this also looks like a replacement and in those cases we find our module in the replaces object again

Not sure if there is something meaningful we can do here.

In the first case the module seems to have a completely different name and is replacing 10s of other modules. Sometimes the components are named like remove-magento-core which seems to suggest they are just being used to remove stuff by (ab)using the replaces object.

If only the namespace is changed, we could use the metadata from the new component with the changed namespace. But I'm not familiar enough with composer and magento to know if that is correct/reliable. It might lead to lots of confusion and incorrect conclusions. Safest option is to ignore the results (and not throw an exception).

The problem with ignoring the result is that the result won't get cached. Would it make sense to have an extra field on MetaModel to store the fact that the component as replaced, not-found, end-of-life, ... ?

This could also be of help for other usecases. For example there are components end of life, and do not have a newer version available. So everything looks OK in DT. This new flag might be helpful for when in the future we can get that flag from somewhere else, or when users want to flag a component manually to make it visible in DT and/or result in policy violations.

wdyt?

adobe-ims.json.txt
adobe-stock-integration.json.txt
composter-root-update-plugin.json.txt
module-aws-s3.json.txt

[jkowalleck]

Composer packages get outdated. Composer package magento/adobe-ims no longer exists. Therefore, it is impossible to install it at all. If you are able to install it, that is because you still have the old package in your caches somehow.
So to make a current use case, let's say you actually are running DependencyTrack analyzes over an old SBOM that you generated at a time when magento/adobe-ims still existed.

---

The following information is based on latest DependencyTrack:

The issue you are facing regarding ComposerMetaAnalyzer is caused by the usage of packagist's package-metadata API -- for which DependencyTrack actually uses the outdated/deprecated API endpoint. See https://repo.packagist.org/p/magento/adobe-ims.json.
For this response, the query is for magento/adobe-ims but this is not included in the response. And here comes the crash.

A possible code-fix could be: check if the expected key exists in the response, and just do nothing if it is missing.

[valentijnscholten]

Yes, that is what I mentioned above. But there are downsides to just ignoring so I suggested maybe a flag is needed to mark packages for which DT cannot find any metadata. But we could raise that as a separate / bigger issue and go for the quickfix here.

[jkowalleck]

I'd suggest switching to the packagist latest API (endpoint p2).
This one does not answer with a result that does not contain the expected response, but actually responds wit ha HTTP status code 404 -- see https://repo.packagist.org/p2/magento/adobe-ims.json

Then we would have a proper response we could tailor the result on.

[valentijnscholten]

Yes, but still the question remains if we should flag the packages in case of a 404 or other problems retrieving metadata (also for other package managers/repos).

[jkowalleck]

re: #2134 (comment)

Don't like the idea of flagging them.
I usually have a lot of private or unpublished packages in my SBOM that are just not available on https://packagist.org. I do not want so see hundreds of red exclamation marks in DT, telling me that the package/version I am using could not be validated/found on some remote source.
I find the current behavior sufficient.

But that's just my 2 cents.

[valentijnscholten]

The flag I mentioned above is mainly an extra field on MetaModel to capture the fact that the metadata for the component could not be retrieved. This would allow us to cache that result and prevent any unnecessary requests for metadata that doesn't.
Showing something in the UI might be useful, maybe only for non-internal components, maybe only when users/admins opt.
But maybe the current UI is good enough as users can opt in to the latest version column which will be empty if the metadata cannot be retrieved.

[valentijnscholten]

Hmm looks like there is already a lastOccurrence timestamp that ensures caching. Last time I checked results weren't cached if there was an empty response, but maybe that was with the Vulnerability Analysis Cache.

[syalioune]

Since #2129, RepositoryMetaComponent.lastCheck and ComponentAnalysisCache.lastOccurence are indeed used for caching. As long as the analyzer returns a MetaModel object (complete or not), it will be cached.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.