Bundler not working as expected for cross-platform gems #3371

xam7247 · 2017-08-31T09:15:01Z

After going through rubygems/bundler#5863, rubygems/bundler#4517, rubygems/bundler#646, rubygems/bundler#2040, rubygems/bundler#2283, rubygems/bundler-features#4, and many more issues on the same note, I still face this issue.

Development machine: Windows
CI machine: Linux (No internet connection, no other Linux machine available to generate a *nix based Gemfile / caching the gems)
Ruby: v2.3.3
Bundler: v1.15.1

Gemfile:

source 'https://rubygems.org'

gem 'rspec', '>= 3.4.0'
gem 'colorize'
gem 'mongo'
gem 'json'
gem 'capybara'
gem "json-schema"
gem 'rubyXL'
gem "roo", "~> 2.7.0"
gem "rest-client", "2.0.2"
gem "ffi"
gem "unf_ext"
gem "nokogiri"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.5.2)
      public_suffix (>= 2.0.2, < 4.0)
    bson (4.2.2)
    capybara (2.15.1)
      addressable
      mini_mime (>= 0.1.3)
      nokogiri (>= 1.3.3)
      rack (>= 1.0.0)
      rack-test (>= 0.5.4)
      xpath (~> 2.0)
    colorize (0.8.1)
    diff-lcs (1.3)
    domain_name (0.5.20170404)
      unf (>= 0.0.5, < 1.0.0)
    ffi (1.9.18)
    ffi (1.9.18-x64-mingw32)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    json (2.1.0)
    json-schema (2.8.0)
      addressable (>= 2.4)
    mime-types (3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2016.0521)
    mini_mime (0.1.4)
    mini_portile2 (2.2.0)
    mongo (2.4.3)
      bson (>= 4.2.1, < 5.0.0)
    netrc (0.11.0)
    nokogiri (1.8.0)
      mini_portile2 (~> 2.2.0)
    nokogiri (1.8.0-x64-mingw32)
      mini_portile2 (~> 2.2.0)
    public_suffix (3.0.0)
    rack (2.0.3)
    rack-test (0.7.0)
      rack (>= 1.0, < 3)
    rest-client (2.0.2)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    rest-client (2.0.2-x64-mingw32)
      ffi (~> 1.9)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    roo (2.7.1)
      nokogiri (~> 1)
      rubyzip (~> 1.1, < 2.0.0)
    rspec (3.6.0)
      rspec-core (~> 3.6.0)
      rspec-expectations (~> 3.6.0)
      rspec-mocks (~> 3.6.0)
    rspec-core (3.6.0)
      rspec-support (~> 3.6.0)
    rspec-expectations (3.6.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.6.0)
    rspec-mocks (3.6.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.6.0)
    rspec-support (3.6.0)
    rubyXL (3.3.26)
      nokogiri (>= 1.4.4)
      rubyzip (>= 1.1.6)
    rubyzip (1.2.1)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.4)
    unf_ext (0.0.7.4-x64-mingw32)
    xpath (2.1.0)
      nokogiri (~> 1.3)

PLATFORMS
  ruby
  x64-mingw32

DEPENDENCIES
  capybara
  colorize
  ffi
  json
  json-schema
  mongo
  nokogiri
  rest-client (= 2.0.2)
  roo (~> 2.7.0)
  rspec (>= 3.4.0)
  rubyXL
  unf_ext

BUNDLED WITH
   1.15.1

As per rubygems/bundler#5863, I did:
bundle config specific_platform true
bundle lock --add-platform ruby
bundle package --all-platforms

Doing this, Bundler did download the gems for other platform ('ruby' in this case), along with gem files for current platform ('x64-mingw32').

Note: I had to add the dependent gems 'ffi', 'unf_ext' and 'nokogiri' since bundler was not downloading other platform gems unless these were explicitly mentioned in Gemfile.

Problem:
After this, when you do bundle install --local (on the windows machine), it installs gems from vendor/cache, but throws an error randomly for one of the cross-platform gems.
e.g., "Some gems seem to be missing from your vendor/cache directory. Could not find ffi-1.9.18 in any of the sources"

Post this, bundle exec ... still works.

So:

The error is misleading since the gem file is in fact available in vendor/cache.
And we need to to explain to each team mate to ignore the error due to a known caveat in Bundler.
And we need to be super careful when adding new gems, to ensure the "missing gems" is not a false alarm.

Any solutions would be much appreciated.

The text was updated successfully, but these errors were encountered:

colby-swandale · 2017-08-31T12:27:40Z

Can you please share the output of bundle env

colby-swandale · 2017-09-08T10:43:18Z

ping @xam7247

xam7247 · 2017-09-11T07:44:37Z

@colby-swandale following is the output of bundle env (I have obscured username and workspace location)

Environment

Bundler   1.15.1
Rubygems  2.3.0
Ruby      2.3.3p222 (2016-11-21 revision 56859) [x64-mingw32]
Git       2.9.0.windows.1
Platform  x64-mingw32

Bundler settings

specific_platform
  Set for the current user (C:/Users/<username>/.bundle/config): "true"
cache_all_platforms
  Set for your local app (D:/<workspace>/.bundle/config): "true"

Gemfile

source 'https://rubygems.org'

# Core gems
gem 'rspec', '>= 3.4.0'
gem 'colorize'
gem 'mongo'
gem 'json'
gem 'capybara'
gem "json-schema"
gem 'rubyXL'
gem "roo", "~> 2.7.0"

# Specify gems that have different version
# on windows and linux, so that bundler
# caches other platform gems too.
# See README for detailed explaination

gem "ffi"
gem "unf_ext"
gem "rest-client", path: "rest-client-2.0.2"
gem "nokogiri"

Gemfile.lock

PATH
  remote: rest-client-2.0.2
  specs:
    rest-client (2.0.2-x64-mingw32)
      ffi (~> 1.9)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.5.2)
      public_suffix (>= 2.0.2, < 4.0)
    bson (4.2.2)
    capybara (2.15.1)
      addressable
      mini_mime (>= 0.1.3)
      nokogiri (>= 1.3.3)
      rack (>= 1.0.0)
      rack-test (>= 0.5.4)
      xpath (~> 2.0)
    colorize (0.8.1)
    diff-lcs (1.3)
    domain_name (0.5.20170404)
      unf (>= 0.0.5, < 1.0.0)
    ffi (1.9.18)
    ffi (1.9.18-x64-mingw32)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    json (2.1.0)
    json-schema (2.8.0)
      addressable (>= 2.4)
    mime-types (3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2016.0521)
    mini_mime (0.1.4)
    mini_portile2 (2.2.0)
    mongo (2.4.3)
      bson (>= 4.2.1, < 5.0.0)
    netrc (0.11.0)
    nokogiri (1.8.0)
      mini_portile2 (~> 2.2.0)
    nokogiri (1.8.0-x64-mingw32)
      mini_portile2 (~> 2.2.0)
    public_suffix (3.0.0)
    rack (2.0.3)
    rack-test (0.7.0)
      rack (>= 1.0, < 3)
    roo (2.7.1)
      nokogiri (~> 1)
      rubyzip (~> 1.1, < 2.0.0)
    rspec (3.6.0)
      rspec-core (~> 3.6.0)
      rspec-expectations (~> 3.6.0)
      rspec-mocks (~> 3.6.0)
    rspec-core (3.6.0)
      rspec-support (~> 3.6.0)
    rspec-expectations (3.6.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.6.0)
    rspec-mocks (3.6.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.6.0)
    rspec-support (3.6.0)
    rubyXL (3.3.26)
      nokogiri (>= 1.4.4)
      rubyzip (>= 1.1.6)
    rubyzip (1.2.1)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.4)
    unf_ext (0.0.7.4-x64-mingw32)
    xpath (2.1.0)
      nokogiri (~> 1.3)

PLATFORMS
  ruby
  x64-mingw32

DEPENDENCIES
  capybara
  colorize
  ffi
  json
  json-schema
  mongo
  nokogiri
  rest-client!
  roo (~> 2.7.0)
  rspec (>= 3.4.0)
  rubyXL
  unf_ext

BUNDLED WITH
   1.15.1

xam7247 · 2018-01-17T07:36:51Z

Any updates on this please ?

colby-swandale · 2018-01-17T10:37:17Z

There is no update for this issue, no one on the core team runs Windows so this issue is hard and very timely for any of us to look into to try to reproduce it.

abelsromero · 2018-02-10T11:17:14Z

Is there really no solution for this issue?

Xetoris · 2018-03-14T16:43:07Z

I'm not sure if it helps anyone, but I experience this same issue in 1.16.*.

Specifically the ffi gem, mentioned in the original post.

I've found that reverting back to 1.15.4 seems to remove the issue? I haven't spent time trying to figure out why yet.

JJJJust · 2018-08-07T20:57:03Z

To stop the cache call on an install, pass --no-cache. Bundler tries to load/update the cache if it exists anytime that bundle install is ran, unless, among other conditions, --no-cache is passed.

I don't know enough about the code to figure out what the issue is, but going down the stack with a gemfile.lock that has pg 1.0.0 and pg 1.0.0-x64-mingw32 in it and is setup for ruby, x64-mingw32, and x84_64-linux...

The error is raised by cli\install.rb as it rescues either GemNotFound or VersionConflict (I'm betting it's the first in this case) around line 88.

cli\install.rb has the line:

Bundler.load.cache if Bundler.app_cache.exist? && !options["no-cache"] && !Bundler.frozen?

That eventually leads you to runtime.rb#cache.

   129:         FileUtils.mkdir_p(p)
   130:       end unless File.exist?(cache_path)
   131:
   132:       Bundler.ui.info "Updating files in #{Bundler.settings.app_cache_path}"
   133:       byebug
=> 134:       specs_to_cache = Bundler.settings[:cache_all_platforms] ? @definition.resolve.materialized_for_all_platforms : specs
   135:       specs_to_cache.each do |spec|
   136:         next if spec.name == "bundler"
   137:         next if spec.source.is_a?(Source::Gemspec)
   138:         spec.source.send(:fetch_gem, spec) if Bundler.settings[:cache_all_platforms] && spec.source.respond_to?(:fetch_gem, true)

Line 134 is causing the exception as it chokes on a gem that isn't for the platform the command is being run on.

D:/Development/Projects/bundler/bundler-1.16.3/lib/bundler/spec_set.rb
   103:     def materialized_for_all_platforms
   104:       names = @specs.map(&:name).uniq
   105:       @specs.map do |s|
   106:         next s unless s.is_a?(LazySpecification)
   107:         s.source.dependency_names = names if s.source.respond_to?(:dependency_names=)
=> 108:         spec = s.__materialize__
   109:         raise GemNotFound, "Could not find #{s.full_name} in any of the sources" unless spec
   110:         spec
   111:       end
   112:     end

spec = s.__materialize__ is coming back nil and s is #<Bundler::LazySpecification:0x00000000040d3168 @name="pg", @version=#<Gem::Version "1.0.0">, @dependencies=[], @platform="ruby", @source=#<Bundler::Source::Rubygems:0x34049260 rubygems repository https://rubygems.org/ or installed locally>, @specification=nil, @__identifier=#<struct Bundler::LazySpecification::Identifier name="pg", version=#<Gem::Version "1.0.0">, source=#<Bundler::Source::Rubygems:0x34049260 rubygems repository https://rubygems.org/ or installed locally>, platform="ruby", dependencies=[]>, @__to_s="pg (1.0.0)"> -- the ruby platform pg gem.

[71, 80] in D:/Development/Projects/bundler/bundler-1.16.3/lib/bundler/lazy_specification.rb
   71:     def __materialize__
   72:       search_object = Bundler.feature_flag.specific_platform? || Bundler.settings[:force_ruby_platform] ? self : Dependency.new(name, version)
   73:       @specification = if source.is_a?(Source::Gemspec) && source.gemspec.name == name
   74:         source.gemspec.tap {|s| s.source = source }
   75:       else
   76:         search = source.specs.search(search_object).last
   77:         if search && Gem::Platform.new(search.platform) != Gem::Platform.new(platform) && !search.runtime_dependencies.-(dependencies.reject {|d| d.type == :development }).empty?
   78:           Bundler.ui.warn "Unable to use the platform-specific (#{search.platform}) version of #{name} (#{version}) " \
   79:             "because it has different dependencies from the #{platform} version. " \
   80:             "To use the platform-specific version of the gem, run `bundle config specific_platform true` and install again."
   81:           search = source.specs.search(self).last
   82:         end
   83:         search.dependencies = dependencies if search.is_a?(RemoteSpecification) || search.is_a?(EndpointSpecification)
   84:         search
   85:       end
   86:     end
   87:

search at 76 is nil... and I'm too far out of my depth to keep going.

https://nvd.nist.gov/vuln/detail/CVE-2018-1000201 Configured bundler to update cached gems for all platforms. Helpful links on that: * https://bundler.io/v1.16/man/bundle-config.1.html * rubygems/bundler#5863 * https://github.com/bundler/bundler/issues/5989 * googleapis/google-cloud-ruby#1592

webervin · 2019-03-12T10:50:58Z

no one on the core team runs Windows so this issue is hard and very timely for any of us to look into to try to reproduce it.

Please note that issue is exactly same if 'Development machine' is mac and CI machine is linux.

In Gemfile.lock:

PLATFORMS
  ruby
  x86_64-darwin-18
  x86_64-linux

seemingly no way to make vendor/cache contain gems for linux platform if command is run on mac....

Edit: while it could be possible to ask bundler to vendor only 'ruby' platform, so that everything would be compiled from 'raw sources', that would make deploy much longer, being able to instead package multiple platform specific gems into vendor/cache would be much better.

Edit2: another workaround is to remove --deployment from bundle install, so it is allowed to use both remote and local sources, but it can introduce non-reproducibility issue across platforms.

bugg2844 · 2019-11-12T17:06:19Z

Potential workaround, in case it helps anyone. In our case, we need to cache a linux specific libv8 in vendor/cache, when building on a Mac. The following gets us around the 'could not find in any of the sources' error:

$ rm -rf vendor/cache/libv8*
$ gem uninstall libv8 --force
Successfully uninstalled libv8-3.16.14.19-x86_64-darwin-17
$ bundle package --all-platforms

Updating files in vendor/cache
Fetching libv8 3.16.14.19
Fetching libv8 3.16.14.19 (x86_64-darwin-17)
Fetching libv8 3.16.14.19 (x86_64-linux)

deivid-rodriguez · 2020-10-20T12:53:49Z

Could you try #4022 and verify whether it fixes this issue? Thanks!

deivid-rodriguez · 2020-11-05T13:30:52Z

I'll assume issue is fixed by mentioned PR.

xam7247 · 2020-11-05T19:07:09Z

Could you try #4022 and verify whether it fixes this issue? Thanks!

@deivid-rodriguez : Unfortunately I don't have access to a windows machine anymore. So until someone else lands into the same issue, we can rightly assume that #4022 might have fixed this 👍

skull-squadron · 2023-01-20T00:58:09Z

Not fixed and not working in bundler 2.4.2 on Windows, Mac, and Linux. Either bundle cache --all-platforms --no-install grabs only ruby platform gems or the current native platform. There are use-cases where vendor/cache needs to be shipped to work offline and standalone. Tried locking platforms exhaustively and permuting various bundler flags.

gemfile.lock and tree

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  arm64-darwin-22
  java
  ruby
  x64-mingw-ucrt
  x64-mingw32
  x86-linux
  x86-mingw32
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  deleteme!
  rake (~> 13.0)
  rspec (~> 3.0)
  rubocop (~> 1.21)

BUNDLED WITH
   2.4.2

vendor
└── cache
    ├── ast-2.4.2.gem
    ├── diff-lcs-1.5.0.gem
    ├── json-2.6.3-java.gem
    ├── json-2.6.3.gem
    ├── mini_portile2-2.8.1.gem
    ├── nokogiri-1.14.0.gem
    ├── parallel-1.22.1.gem
    ├── parser-3.2.0.0.gem
    ├── racc-1.6.2-java.gem
    ├── racc-1.6.2.gem
    ├── rainbow-3.1.1.gem
    ├── rake-13.0.6.gem
    ├── regexp_parser-2.6.2.gem
    ├── rexml-3.2.5.gem
    ├── rspec-3.12.0.gem
    ├── rspec-core-3.12.0.gem
    ├── rspec-expectations-3.12.2.gem
    ├── rspec-mocks-3.12.3.gem
    ├── rspec-support-3.12.0.gem
    ├── rubocop-1.43.0.gem
    ├── rubocop-ast-1.24.1.gem
    ├── ruby-progressbar-1.11.0.gem
    └── unicode-display_width-2.4.2.gem

2 directories, 23 files

deivid-rodriguez · 2023-01-20T10:38:54Z

Please report a separate issue with repro steps. Thank you.

skull-squadron · 2023-01-20T18:55:22Z

👍 I'll scare up a MCE Dockerfile when I have a chance.

…

On Fri, Jan 20, 2023 at 4:39 AM David Rodríguez ***@***.***> wrote: Please report a separate issue with repro steps. Thank you. — Reply to this email directly, view it on GitHub <#3371 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABWYMAIXZ4LA7HPKQTHBMDWTJTMTANCNFSM4LIKRV7A> . You are receiving this because you commented.Message ID: ***@***.***>

skull-squadron · 2023-01-20T18:58:33Z

Is multiplatform FOSS testing and on-demnd testing still a PITA? At work, we have to run Ruby-based software in extremely heterogeneous use-cases.

hsbt transferred this issue from rubygems/bundler Mar 14, 2020

bronzdoc added the status: triage label Mar 14, 2020

colby-swandale added the Bundler label Mar 14, 2020

bronzdoc added the platform: windows label Mar 22, 2020

deivid-rodriguez added status: feedback required and removed status: triage labels Oct 20, 2020

deivid-rodriguez closed this as completed Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundler not working as expected for cross-platform gems #3371

Bundler not working as expected for cross-platform gems #3371

xam7247 commented Aug 31, 2017

colby-swandale commented Aug 31, 2017

colby-swandale commented Sep 8, 2017

xam7247 commented Sep 11, 2017 •

edited

Loading

xam7247 commented Jan 17, 2018

colby-swandale commented Jan 17, 2018 •

edited

Loading

abelsromero commented Feb 10, 2018

Xetoris commented Mar 14, 2018 •

edited

Loading

JJJJust commented Aug 7, 2018 •

edited

Loading

webervin commented Mar 12, 2019 •

edited

Loading

bugg2844 commented Nov 12, 2019

deivid-rodriguez commented Oct 20, 2020

deivid-rodriguez commented Nov 5, 2020

xam7247 commented Nov 5, 2020

skull-squadron commented Jan 20, 2023 •

edited

Loading

deivid-rodriguez commented Jan 20, 2023

skull-squadron commented Jan 20, 2023 via email

skull-squadron commented Jan 20, 2023 via email

Bundler not working as expected for cross-platform gems #3371

Bundler not working as expected for cross-platform gems #3371

Comments

xam7247 commented Aug 31, 2017

colby-swandale commented Aug 31, 2017

colby-swandale commented Sep 8, 2017

xam7247 commented Sep 11, 2017 • edited Loading

Environment

Bundler settings

Gemfile

Gemfile

Gemfile.lock

xam7247 commented Jan 17, 2018

colby-swandale commented Jan 17, 2018 • edited Loading

abelsromero commented Feb 10, 2018

Xetoris commented Mar 14, 2018 • edited Loading

JJJJust commented Aug 7, 2018 • edited Loading

webervin commented Mar 12, 2019 • edited Loading

bugg2844 commented Nov 12, 2019

deivid-rodriguez commented Oct 20, 2020

deivid-rodriguez commented Nov 5, 2020

xam7247 commented Nov 5, 2020

skull-squadron commented Jan 20, 2023 • edited Loading

deivid-rodriguez commented Jan 20, 2023

skull-squadron commented Jan 20, 2023 via email

skull-squadron commented Jan 20, 2023 via email

xam7247 commented Sep 11, 2017 •

edited

Loading

colby-swandale commented Jan 17, 2018 •

edited

Loading

Xetoris commented Mar 14, 2018 •

edited

Loading

JJJJust commented Aug 7, 2018 •

edited

Loading

webervin commented Mar 12, 2019 •

edited

Loading

skull-squadron commented Jan 20, 2023 •

edited

Loading