-
-
Notifications
You must be signed in to change notification settings - Fork 2k
Java Heap Space With Bundle Update #4572
Comments
I'm only seeing rubygems in that backtrace and not bundler, unless I'm missing something? |
Try |
I came across similar situation during --- ERROR REPORT TEMPLATE -------------------------------------------------------
Error details
Environment Bundler 1.12.5
path It seems that multi-thread bundling is the point of this issue (no error occurred with single thread mode |
Incurred same issue with jruby-1.7.25, WORKED with @dlovellrw comment to use |
This is not a Java problem, this is a bundler problem. The heap cap is a safety measure to keep a JVM-based app from overtaking all of a system's memory for a runaway process. On older JVMs, that default maximum was very low (64MB) and so JRuby's launcher bumps it up to 500MB. It can be set higher with -J-Xmx, which the memory error mentions. JRuby 9.1's bash launcher no longer sets any limit, since modern versions of the JVM select much higher default memory caps (e.g. OpenJDK/Hotspot will set the max to 1/4 of system memory size). But the real question is why Bundler is consuming 500MB of heap space. Can someone contribute a Gemfile that runs into this problem on JRuby 9.1 or higher? Or if these are private projects, reproduce it with -J-XX:+HeapDumpOnOutOfMemoryError (a JVM flag) and post the resulting file (very large, compresses reasonably well) somewhere I can get it. It will have all live data from your heap, so be sure there's nothing sensitive. |
We have a 202MB gem. Yeah. That's our problem. |
Thanks for the reports and solutions, everyone! This looks like an expected result of asking Bundler to install .gem files that are hundreds of megabytes, so I'm going to close this issue. This ticket also revealed a problem that is a bug in Bundler, which is that we don't explain what's happening or what you can do when an OOM error occurs. I've opened a ticket to track work on fixing that at #4673. |
Do gems have to be in memory to install? That seems like a good candidate for actually fixing this issue. |
Expected? Why? Why shouldn't I be able to install a 1GB gem with less than 1GB of available memory? That seems like a bug. |
@headius yes, gems have to be read into memory to extract the .gemspec from the tarball. 202MB gems are so rare that no one has ever bothered to optimize the gemspec extraction code. Feel free to send a PR to RubyGems if you'd like to improve it, though! |
Why? Aren't they just a tar file that can be unpacked to temp space?
Ahh, this is a RubyGems issue then. Well I'm happy that it's not just "welcome to Java" :-) |
@indirect Actually, why did you close this? Only one person (@dlovellrw) that had the problem said they had a 202MB gem. @sushma-unii, @jkr2255, and @diomenas Do you have large gems you are installing, or are these fairly normal sets of gems? I'd like to see that this is fixed rather than just assuming you all have unreasonably large gems. |
@headius okay, in order:
I have no idea why, but that's what the RubyGems method that we call to get gemspecs out of gem files does. My guess as to why is "it was very easy and it worked at the time", but I didn't write it ¯_(ツ)_/¯
Because "I allowed Bundler to have less memory than it needed to install" is not a bug in Bundler. I opened a separate ticket noting that we need to explain the problem and suggest workarounds when this exception pops up, and that ticket is still open. |
But this might be a bug causing it to use too much memory (or an improvement we can make to use less). We have not done any investigation at all. |
I am absolutely sure that it is possible to use less memory than we do now (not reading gems into memory to extract tarballs, for example), but I don't think this ticket is a good place to track general memory usage improvements. This ticket is about OOM exceptions when I am happy to accept new issues (or better yet, PRs) that report on ways to reduce the amount of memory that Bundler needs, but this ticket is not the right place for that. |
@headius Gems I tried to install were: # from gemspec
spec.add_dependency 'function_module', '>= 0.1.1'
spec.add_development_dependency 'bundler', '~> 1.11'
spec.add_development_dependency 'rake', '~> 10.0'
spec.add_development_dependency 'rake-compiler'
spec.add_development_dependency 'rspec', '~> 3.0'
|
@jkr2255 Thank you for reporting back! There definitely seems to be something wrong somewhere if a Gemfile that small is causing OOM. What version of JRuby and Bundler was that? |
@headius As I mentioned above:
|
@jkr2255 Oh right, now I see it. I also see jruby-1.7.25 in your comment above. |
@jkr2255 I was unable to reproduce with JRuby 1.7.25 on either Java 7 or Java 8. What version of Java were you running?
If you can try a similar command line (mostly the heap dump part) we can have a look at your heap and figure out why it's taking so much memory. |
I can reproduce this easily with jruby-9.1.2.0, installed with rbenv on OSX 10.11. I can get it to happen sometimes with 'bundle install', but it is always reproducible for me with 'bundle outdated'.
|
Note that the line it dies on in my version of bundler is this:
|
I am trying to run Bundle update for a Logstash bundle that I downloaded and I'm running into the following error:
--- ERROR REPORT TEMPLATE -------------------------------------------------------
What did you do?
I ran the command
/usr/local/rvm/gems/jruby-9.0.5.0/bin/bundle update
What did you expect to happen?
I expected Bundler to...
What happened instead?
Instead, what actually happened was...
Error details
Environment
Bundler 1.12.3
Rubygems 2.4.8
Ruby 2.2.3p0 (2016-01-26 revision 51636) [java]
GEM_HOME /usr/local/rvm/gems/jruby-9.0.5.0
GEM_PATH /usr/local/rvm/gems/jruby-9.0.5.0:/usr/local/rvm/gems/jruby-9.0.5.0@global
RVM 1.27.0 (latest)
Git 1.8.3.1
rubygems-bundler (1.4.4)
orig_path
Set via BUNDLE_ORIG_PATH: "/usr/local/rvm/gems/jruby-9.0.5.0/bin:/usr/local/rvm/gems/jruby-9.0.5.0@global/bin:/usr/local/rvm/rubies/jruby-9.0.5.0/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin"
orig_gem_path
Set via BUNDLE_ORIG_GEM_PATH: "/usr/local/rvm/gems/jruby-9.0.5.0:/usr/local/rvm/gems/jruby-9.0.5.0@global"
--- TEMPLATE END ----------------------------------------------------------------
The text was updated successfully, but these errors were encountered: