Skip to content
This repository has been archived by the owner on Apr 14, 2021. It is now read-only.

Java Heap Space With Bundle Update #4572

Closed
billyjbryant opened this issue May 12, 2016 · 22 comments
Closed

Java Heap Space With Bundle Update #4572

billyjbryant opened this issue May 12, 2016 · 22 comments

Comments

@billyjbryant
Copy link

I am trying to run Bundle update for a Logstash bundle that I downloaded and I'm running into the following error:

--- ERROR REPORT TEMPLATE -------------------------------------------------------

  • What did you do?

    I ran the command /usr/local/rvm/gems/jruby-9.0.5.0/bin/bundle update

  • What did you expect to happen?

    I expected Bundler to...

  • What happened instead?

    Instead, what actually happened was...

Error details

Java::JavaLang::OutOfMemoryError: Java heap space
  org.jruby.util.ByteList.ensure(ByteList.java:341)
  org.jruby.util.io.EncodingUtils.strBufCat(EncodingUtils.java:1506)
  org.jruby.util.io.EncodingUtils.strBufCat(EncodingUtils.java:1491)
  org.jruby.util.io.EncodingUtils.encCrStrBufCat(EncodingUtils.java:1599)
  org.jruby.util.io.EncodingUtils.encCrStrBufCat(EncodingUtils.java:1522)
  org.jruby.RubyString.cat19(RubyString.java:1296)
  org.jruby.RubyString.cat19(RubyString.java:1285)
  org.jruby.RubyString.append19(RubyString.java:2068)
  org.jruby.RubyString.concat19(RubyString.java:2096)
  org.jruby.RubyString$INVOKER$i$1$0$concat19.call(RubyString$INVOKER$i$1$0$concat19.gen)
  org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
  usr.local.rvm.rubies.jruby_minus_9_dot_0_dot_5_dot_0.lib.ruby.stdlib.rubygems.request.invokeOther74:\=\^\^(/usr/local/rvm/rubies/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/request.rb)
  usr.local.rvm.rubies.jruby_minus_9_dot_0_dot_5_dot_0.lib.ruby.stdlib.rubygems.request.RUBY$block$perform_request$4(/usr/local/rvm/rubies/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/request.rb:161)
  java.lang.invoke.LambdaForm$DMH/1155686302.invokeStatic_L8_L(LambdaForm$DMH)
  java.lang.invoke.LambdaForm$MH/1816757085.invokeExact_MT(LambdaForm$MH)
  org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:140)
  org.jruby.runtime.BlockBody.yield(BlockBody.java:94)
  org.jruby.runtime.Block.yield(Block.java:166)
  org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:476)
  usr.local.rvm.rubies.jruby_minus_9_dot_0_dot_5_dot_0.lib.ruby.stdlib.net.protocol.RUBY$method$call_block$0(/usr/local/rvm/rubies/jruby-9.0.5.0/lib/ruby/stdlib/net/protocol.rb:411)
  java.lang.invoke.LambdaForm$DMH/446517309.invokeStatic_L7_L(LambdaForm$DMH)
  java.lang.invoke.LambdaForm$MH/1897871865.invokeExact_MT(LambdaForm$MH)
  org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:131)
  org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:194)
  org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)
  org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
  usr.local.rvm.rubies.jruby_minus_9_dot_0_dot_5_dot_0.lib.ruby.stdlib.net.protocol.invokeOther5:call_block(/usr/local/rvm/rubies/jruby-9.0.5.0/lib/ruby/stdlib/net/protocol.rb)
  usr.local.rvm.rubies.jruby_minus_9_dot_0_dot_5_dot_0.lib.ruby.stdlib.net.protocol.RUBY$method$\=\^\^$0(/usr/local/rvm/rubies/jruby-9.0.5.0/lib/ruby/stdlib/net/protocol.rb:403)
  java.lang.invoke.LambdaForm$DMH/1793784656.invokeStatic_L7_L(LambdaForm$DMH)
  java.lang.invoke.LambdaForm$MH/1897871865.invokeExact_MT(LambdaForm$MH)
  org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:131)
  org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:194)

Environment

Bundler 1.12.3
Rubygems 2.4.8
Ruby 2.2.3p0 (2016-01-26 revision 51636) [java]
GEM_HOME /usr/local/rvm/gems/jruby-9.0.5.0
GEM_PATH /usr/local/rvm/gems/jruby-9.0.5.0:/usr/local/rvm/gems/jruby-9.0.5.0@global
RVM 1.27.0 (latest)
Git 1.8.3.1
rubygems-bundler (1.4.4)

  Bundler settings

orig_path
Set via BUNDLE_ORIG_PATH: "/usr/local/rvm/gems/jruby-9.0.5.0/bin:/usr/local/rvm/gems/jruby-9.0.5.0@global/bin:/usr/local/rvm/rubies/jruby-9.0.5.0/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin"
orig_gem_path
Set via BUNDLE_ORIG_GEM_PATH: "/usr/local/rvm/gems/jruby-9.0.5.0:/usr/local/rvm/gems/jruby-9.0.5.0@global"
--- TEMPLATE END ----------------------------------------------------------------

@segiddins
Copy link
Member

I'm only seeing rubygems in that backtrace and not bundler, unless I'm missing something?

@dlovellrw
Copy link

Try JRUBY_OPTS='-J-Xmx4G' bundle install to increase the heap size available on the JVM running JRuby. Welcome to Java.

@jkr2255
Copy link

jkr2255 commented May 29, 2016

I came across similar situation during bundle install in JRuby.

--- ERROR REPORT TEMPLATE -------------------------------------------------------

  • What did you do?

    I ran the command /home/user/.rbenv/versions/jruby-1.7.25/bin/bundle install --path=vendor/bundle

  • What did you expect to happen?

    I expected Bundler to...

  • What happened instead?

    Instead, what actually happened was...

Error details

Java::JavaLang::OutOfMemoryError: Java heap space
  java.util.Hashtable.rehash(Hashtable.java:401)
  java.util.Hashtable.addEntry(Hashtable.java:425)
  java.util.Hashtable.put(Hashtable.java:476)
  java.security.Provider.implPut(Provider.java:834)
  java.security.Provider.put(Provider.java:322)
  org.bouncycastle.jce.provider.BouncyCastleProvider.addAlgorithm(Unknown Source)
  org.bouncycastle.jcajce.provider.asymmetric.IES$Mappings.configure(Unknown Source)
  org.bouncycastle.jce.provider.BouncyCastleProvider.loadAlgorithms(Unknown Source)
  org.bouncycastle.jce.provider.BouncyCastleProvider.setup(Unknown Source)
  org.bouncycastle.jce.provider.BouncyCastleProvider.access$000(Unknown Source)
  org.bouncycastle.jce.provider.BouncyCastleProvider$1.run(Unknown Source)
  java.security.AccessController.doPrivileged(Native Method)
  org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source)
  org.bouncycastle.jcajce.util.BCJcaJceHelper.getBouncyCastleProvider(Unknown Source)
  org.bouncycastle.jcajce.util.BCJcaJceHelper.<init>(Unknown Source)
  org.bouncycastle.jcajce.provider.asymmetric.x509.CertificateFactory.<init>(Unknown Source)
  sun.reflect.GeneratedConstructorAccessor79.newInstance(Unknown Source)
  sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  java.lang.Class.newInstance(Class.java:442)
  org.jruby.ext.openssl.SecurityHelper.findImplEngine(SecurityHelper.java:705)
  org.jruby.ext.openssl.SecurityHelper.getImplEngine(SecurityHelper.java:666)
  org.jruby.ext.openssl.SecurityHelper.getCertificateFactory(SecurityHelper.java:196)
  org.jruby.ext.openssl.SecurityHelper.getCertificateFactory(SecurityHelper.java:188)
  org.jruby.ext.openssl.x509store.PEMInputOutput.getX509CertificateFactory(PEMInputOutput.java:1378)
  org.jruby.ext.openssl.x509store.PEMInputOutput.readAuxCertificate(PEMInputOutput.java:1210)
  org.jruby.ext.openssl.x509store.PEMInputOutput.readPEM(PEMInputOutput.java:238)
  org.jruby.ext.openssl.x509store.Lookup.loadCertificateOrCRLFile(Lookup.java:326)
  org.jruby.ext.openssl.x509store.Lookup$ByFile.call(Lookup.java:525)
  org.jruby.ext.openssl.x509store.Lookup$ByFile.call(Lookup.java:508)
  org.jruby.ext.openssl.x509store.Lookup.control(Lookup.java:154)
  org.jruby.ext.openssl.x509store.Lookup.loadFile(Lookup.java:123)

Environment

Bundler 1.12.5
Rubygems 2.6.4
Ruby 1.9.3p551 (2016-04-13 revision 48406) [java]
GEM_HOME /outer/ruby_projects/bit_utils/vendor/bundle/jruby/1.9
GEM_PATH
Git 1.8.3.1

  Bundler settings

path
Set for your local app (/outer/ruby_projects/bit_utils/.bundle/config): "vendor/bundle"
disable_shared_gems
Set for your local app (/outer/ruby_projects/bit_utils/.bundle/config): true
jobs
Set for your local app (/outer/ruby_projects/bit_utils/.bundle/config): "4"
orig_path
Set via BUNDLE_ORIG_PATH: "/home/user/.rbenv/versions/jruby-1.7.25/bin:/home/user/.rbenv/libexec:/home/user/.rbenv/plugins/ruby-build/bin:/home/user/.rbenv/shims:/home/user/.rbenv/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/user/.local/bin:/home/user/bin"
--- TEMPLATE END ----------------------------------------------------------------

It seems that multi-thread bundling is the point of this issue (no error occurred with single thread mode -j1)

@coilysiren coilysiren added this to the 1.X -- Better Platform Support milestone Jun 8, 2016
@sushma-unii
Copy link

Incurred same issue with jruby-1.7.25, WORKED with @dlovellrw comment to use JRUBY_OPTS='-J-Xmx4G' bundle install

@headius
Copy link
Contributor

headius commented Jun 14, 2016

Welcome to Java.

This is not a Java problem, this is a bundler problem.

The heap cap is a safety measure to keep a JVM-based app from overtaking all of a system's memory for a runaway process. On older JVMs, that default maximum was very low (64MB) and so JRuby's launcher bumps it up to 500MB. It can be set higher with -J-Xmx, which the memory error mentions. JRuby 9.1's bash launcher no longer sets any limit, since modern versions of the JVM select much higher default memory caps (e.g. OpenJDK/Hotspot will set the max to 1/4 of system memory size).

But the real question is why Bundler is consuming 500MB of heap space.

Can someone contribute a Gemfile that runs into this problem on JRuby 9.1 or higher? Or if these are private projects, reproduce it with -J-XX:+HeapDumpOnOutOfMemoryError (a JVM flag) and post the resulting file (very large, compresses reasonably well) somewhere I can get it. It will have all live data from your heap, so be sure there's nothing sensitive.

@dlovellrw
Copy link

We have a 202MB gem. Yeah. That's our problem.

@indirect
Copy link
Member

indirect commented Jun 14, 2016

Thanks for the reports and solutions, everyone! This looks like an expected result of asking Bundler to install .gem files that are hundreds of megabytes, so I'm going to close this issue.

This ticket also revealed a problem that is a bug in Bundler, which is that we don't explain what's happening or what you can do when an OOM error occurs. I've opened a ticket to track work on fixing that at #4673.

@headius
Copy link
Contributor

headius commented Jun 14, 2016

We have a 202MB gem. Yeah. That's our problem.

Do gems have to be in memory to install? That seems like a good candidate for actually fixing this issue.

@headius
Copy link
Contributor

headius commented Jun 14, 2016

This looks like an expected result of asking Bundler to install .gem files that are hundreds of megabytes, so I'm going to close this issue.

Expected? Why? Why shouldn't I be able to install a 1GB gem with less than 1GB of available memory? That seems like a bug.

@indirect
Copy link
Member

@headius yes, gems have to be read into memory to extract the .gemspec from the tarball. 202MB gems are so rare that no one has ever bothered to optimize the gemspec extraction code. Feel free to send a PR to RubyGems if you'd like to improve it, though!

@headius
Copy link
Contributor

headius commented Jun 14, 2016

@headius yes, gems have to be read into memory to extract the .gemspec from the tarball.

Why? Aren't they just a tar file that can be unpacked to temp space?

Feel free to send a PR to RubyGems if you'd like to improve it, though!

Ahh, this is a RubyGems issue then. Well I'm happy that it's not just "welcome to Java" :-)

@headius
Copy link
Contributor

headius commented Jun 14, 2016

@indirect Actually, why did you close this? Only one person (@dlovellrw) that had the problem said they had a 202MB gem.

@sushma-unii, @jkr2255, and @diomenas Do you have large gems you are installing, or are these fairly normal sets of gems? I'd like to see that this is fixed rather than just assuming you all have unreasonably large gems.

@indirect
Copy link
Member

@headius okay, in order:

Why?

I have no idea why, but that's what the RubyGems method that we call to get gemspecs out of gem files does. My guess as to why is "it was very easy and it worked at the time", but I didn't write it ¯_(ツ)_/¯

Why did you close this?

Because "I allowed Bundler to have less memory than it needed to install" is not a bug in Bundler. I opened a separate ticket noting that we need to explain the problem and suggest workarounds when this exception pops up, and that ticket is still open.

@headius
Copy link
Contributor

headius commented Jun 14, 2016

Because "I allowed Bundler to have less memory than it needed to install" is not a bug in Bundler.

But this might be a bug causing it to use too much memory (or an improvement we can make to use less). We have not done any investigation at all.

@indirect
Copy link
Member

I am absolutely sure that it is possible to use less memory than we do now (not reading gems into memory to extract tarballs, for example), but I don't think this ticket is a good place to track general memory usage improvements.

This ticket is about OOM exceptions when bundle commands are run. It contains a solution/workaround for the problem as reported, and spawned another ticket for how to improve the error message to proactively help others who encounter this problem in the future.

I am happy to accept new issues (or better yet, PRs) that report on ways to reduce the amount of memory that Bundler needs, but this ticket is not the right place for that.

@jkr2255
Copy link

jkr2255 commented Jun 14, 2016

@headius Gems I tried to install were:

  # from gemspec
  spec.add_dependency 'function_module', '>= 0.1.1'
  spec.add_development_dependency 'bundler', '~> 1.11'
  spec.add_development_dependency 'rake', '~> 10.0'
  spec.add_development_dependency 'rake-compiler'
  spec.add_development_dependency 'rspec', '~> 3.0'

function_module is tiny gem I made, and the rest are popular gems for Ruby development.

@headius
Copy link
Contributor

headius commented Jun 14, 2016

function_module is tiny gem I made, and the rest are popular gems for Ruby development.

@jkr2255 Thank you for reporting back! There definitely seems to be something wrong somewhere if a Gemfile that small is causing OOM. What version of JRuby and Bundler was that?

@jkr2255
Copy link

jkr2255 commented Jun 14, 2016

@headius As I mentioned above:

Bundler 1.12.5
Rubygems 2.6.4
Ruby 1.9.3p551 (2016-04-13 revision 48406) [java]

@headius
Copy link
Contributor

headius commented Jun 14, 2016

@jkr2255 Oh right, now I see it. I also see jruby-1.7.25 in your comment above.

@headius
Copy link
Contributor

headius commented Jun 15, 2016

@jkr2255 I was unable to reproduce with JRuby 1.7.25 on either Java 7 or Java 8. What version of Java were you running?

$ jruby --dev -J-Xmx150M -J-XX:+HeapDumpOnOutOfMemoryError -S bundle install --force
Ignoring executable-hooks-1.3.2 because its extensions are not built.  Try: gem pristine executable-hooks --version 1.3.2
Ignoring gem-wrappers-1.2.7 because its extensions are not built.  Try: gem pristine gem-wrappers --version 1.2.7
Ignoring jruby-launcher-1.1.1-java because its extensions are not built.  Try: gem pristine jruby-launcher --version 1.1.1
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing rake 10.5.0
Installing diff-lcs 1.2.5
Installing function_module 0.1.1
Installing rspec-support 3.4.1
Using bundler 1.12.5
Installing rake-compiler 0.9.9
Installing rspec-core 3.4.4
Installing rspec-expectations 3.4.0
Installing rspec-mocks 3.4.1
Installing rspec 3.4.0
Bundle complete! 5 Gemfile dependencies, 10 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

If you can try a similar command line (mostly the heap dump part) we can have a look at your heap and figure out why it's taking so much memory.

@ari
Copy link

ari commented Jul 11, 2016

I can reproduce this easily with jruby-9.1.2.0, installed with rbenv on OSX 10.11. I can get it to happen sometimes with 'bundle install', but it is always reproducible for me with 'bundle outdated'.

Error details

    Java::JavaLang::OutOfMemoryError: Java heap space
      Users.ari.$_dot_rbenv.versions.jruby_minus_9_dot_1_dot_2_dot_0.lib.ruby.gems.shared.gems.bundler_minus_1_dot_12_dot_5.lib.bundler.index_dot_rb.RUBY$method$search_by_spec$0(/Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared/gems/bundler-1.12.5/lib/bundler/index.rb:178)
      java.lang.invoke.LambdaForm$DMH/1554547125.invokeStatic_LLLLLLL_L(LambdaForm$DMH)
      java.lang.invoke.LambdaForm$MH/1849201180.invokeExact_MT(LambdaForm$MH)
      org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:245)
      org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:110)
      org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
      org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
      org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
      Users.ari.$_dot_rbenv.versions.jruby_minus_9_dot_1_dot_2_dot_0.lib.ruby.gems.shared.gems.bundler_minus_1_dot_12_dot_5.lib.bundler.index_dot_rb.invokeOther0:search_by_spec(/Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared/gems/bundler-1.12.5/lib/bundler/index.rb)
      Users.ari.$_dot_rbenv.versions.jruby_minus_9_dot_1_dot_2_dot_0.lib.ruby.gems.shared.gems.bundler_minus_1_dot_12_dot_5.lib.bundler.index_dot_rb.RUBY$block$use$0(/Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared/gems/bundler-1.12.5/lib/bundler/index.rb:119)
      java.lang.invoke.LambdaForm$DMH/617901222.invokeStatic_LLLLLLLL_L(LambdaForm$DMH)
      java.lang.invoke.LambdaForm$MH/1858609436.invokeExact_MT(LambdaForm$MH)
      org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
      org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:122)
      org.jruby.runtime.BlockBody.yield(BlockBody.java:110)
      org.jruby.runtime.Block.yield(Block.java:167)
      org.jruby.RubyArray.each(RubyArray.java:1593)
      org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
      org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)
      org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
      Users.ari.$_dot_rbenv.versions.jruby_minus_9_dot_1_dot_2_dot_0.lib.ruby.gems.shared.gems.bundler_minus_1_dot_12_dot_5.lib.bundler.index_dot_rb.invokeOther1:each(/Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared/gems/bundler-1.12.5/lib/bundler/index.rb)
      Users.ari.$_dot_rbenv.versions.jruby_minus_9_dot_1_dot_2_dot_0.lib.ruby.gems.shared.gems.bundler_minus_1_dot_12_dot_5.lib.bundler.index_dot_rb.RUBY$block$each$0(/Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared/gems/bundler-1.12.5/lib/bundler/index.rb:99)
      java.lang.invoke.LambdaForm$DMH/617901222.invokeStatic_LLLLLLLL_L(LambdaForm$DMH)
      java.lang.invoke.LambdaForm$MH/1858609436.invokeExact_MT(LambdaForm$MH)
      org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
      org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:122)
      org.jruby.runtime.BlockBody.yield(BlockBody.java:110)
      org.jruby.runtime.Block.yield(Block.java:167)
      org.jruby.RubyArray.each(RubyArray.java:1593)
      org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
      org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:497)
      org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:273)

Environment

  Bundler   1.12.5
  Rubygems  2.6.4
  Ruby      2.3.0p0 (2016-05-26 revision 52539) [java]
  GEM_HOME  /Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared
  GEM_PATH  /Users/ari/.rbenv/versions/jruby-9.1.2.0/lib/ruby/gems/shared:/Users/ari/.gem/jruby/2.3.0
  Git       2.9.0

Gemfile here.

@ari
Copy link

ari commented Jul 11, 2016

Note that the line it dies on in my version of bundler is this:

spec = @specs[spec.name][spec.full_name]

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants