Gemfile parsing for Bundler (Ruby) doesn't correctly take into account platforms (ruby, java etc.) #8695
Comments
|
Thanks for the sophisticated report. In essence, this is a specific case of #4013. |
|
I guess part of the problem is this line: |
|
I haven't traced the whole call chain and I'm by no means a Kotlin expert, but based on the stack trace, the problem is in this function: It seems that independent of that Ruby script that uses Bundler's own facilities to resolve dependencies, the RubyGems API is still queried directly, using ORT's HTTP client (at least that's how I'm interpreting This is also evidenced by the fact that ORT seems to be trying to resolve a dependency for the If ORT were to solely rely on Bundler's mechanisms instead of querying the API manually, I think it might generally be fine, but you might want to either remove the |
|
Not sure if it helps, but I toyed around with it a bit more and the minimal source 'https://rubygems.org'
gem 'rails'
gem 'rails-i18n'
And When I remove one or the other, the problem doesn't appear. But when I have both in my The same is true when I replace either of them with But the problem doesn't appear when I put in 2 packages that both depend on source 'https://rubygems.org'
gem 'irb'
gem 'minitest' |
|
Further digging: According to the source code of rubygems.org, it is expected behavior that when not explicitly specifying platform, it picks whatever was last created. For psych, this seems to be the Java version: Whereas e.g. for nokogiri (another multi-platform gem), it's Ruby: This still doesn't explain why sometimes it seems to be fine and sometimes it isn't (see my previous comment), but at least indicates that if ORT wants to have deterministic behavior, the platform needs to always be explicitly specified. (And which platform it is can be derived from |
|
Any news on this one, @sschuberth ? Can I help with further info? Also, if the intention was to enforce |
I believe your info is more than sufficient, thank you @clemens. It's just that to be best of my knowledge no one from the community is planning to work on this (for free) in the near future. If you're interested in a commercial ORT support subscription that could accelerate this (also see #8381), feel free to contact any of the mentioned parties directly. |
|
That's understandable. :) Would you accept an MR that appends |
Sounds reasonable. But for me it's the easiest to tell from the diff of a MR / PR, as that would need to include any necessary changes to the expected test results. |
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
When fetching version data for individual Ruby gems, we don't want to rely on creation order of the gems on RubyGems.org. Instead, we use the standard `ruby` platform as a sensible default (as we do elsewhere). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
Fetching version data for individual Ruby gems shouldn't rely on creation order of the gems on RubyGems.org. Instead, use the standard `ruby` platform as a sensible default (as it already happens in Bundler's resolving logic elsewhere in the code). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
Fetching version data for individual Ruby gems shouldn't rely on creation order of the gems on RubyGems.org. Instead, use the standard `ruby` platform as a sensible default (as it already happens in Bundler's resolving logic elsewhere in the code). Partially addresses oss-review-toolkit#8695. Signed-off-by: Clemens Kofler <[email protected]>
Fetching version data for individual Ruby gems shouldn't rely on creation order of the gems on RubyGems.org. Instead, use the standard `ruby` platform as a sensible default (as it already happens in Bundler's resolving logic elsewhere in the code). Partially addresses #8695. Signed-off-by: Clemens Kofler <[email protected]>
Describe the bug
Disclaimer: Strictly speaking, this is probably a mixture of a bug and a feature request. But since it does cause unexpected error messages in the output, I'll qualify it as a bug in this ticket.
The gist is that in a Ruby project, ORT doesn't seem to take into account Bundler's concept of "platforms" (ruby, java etc.) and subsequently might look up wrong transitive dependencies and/or complain about missing dependencies.
To Reproduce
Steps to reproduce the behavior:
psych, which depends onstringioon therubyplatform, butjar-dependencieson thejavaplatformbundle installNote that running
bundle lock --add-platform javaalso doesn't make a difference – while it includes the dependencies correctly inGemfile.lock, the error persistsExpected behavior
Gemfile.lock). In the example, I'd expect to not see an error that tells me thatjar-dependenciescouldn't be found.Gemfile.lockincludes onlypsych (5.1.2), ORT should correctly handle it with itsrubyplatform dependencies; if it includespsych (5.1.2-java), ORT should correctly handle it with itsjavaplatform dependencies; and if it includes multiple platforms, ORT should handle both versionsA common use case for having multiple platforms is when engineers are using Macs for development (=>
arm64-darwinorx86_64-darwinplatform, depending on whether it's an older or newer Mac) but Linux for production (=> e.g.x86_64-linux). A gem likenokogiriwould then be listed with multiple versions:Notice again the differences between the platforms, e.g. whether or not
mini_portile2is a dependency and the fact that there is a specific version ofraccfor thejavaplatform but not e.g. Macs.Console / log output
Snippet from GitLab CI output:
When doing
curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml'(thus effectively reproducing what ORT does inort/plugins/package-managers/bundler/src/main/kotlin/Bundler.kt
Line 360 in 740b7d6
javaversion of it:As specified in http://guides.rubygems.org/rubygems-org-api-v2/, a
platformparameter can be given (curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml?platform=ruby'), which then returns the correct version:Take note of the different
dependencies–jar-dependenciesfor thejavaversion,stringiofor therubyversion.Environment
Output of the
ort requirementscommand:And specify (relevant parts of) your ORT configuration (
config.yml):Nothing special, just using the default config that https://github.com/oss-review-toolkit/ort-ci-gitlab/blob/main/templates/ort-scan.yml provides/generates.
Additional context
n/a
The text was updated successfully, but these errors were encountered: