"Too many open files" when building large jars inside VM/chroot environment

[mbooth101]

I am trying out the latest plexus archiver dc873a4

When I am building a large jar file inside a mock chroot environment inside a VM, I see a "Too many open files" exception, as shown below.

This seems to be a problem with using ParallelScatterZipCreator from commons-compress 1.10 because my problem went away when I patched it out: http://pkgs.fedoraproject.org/cgit/plexus-archiver.git/tree/0001-Avoid-using-ParallelScatterZipCreator.patch

I encounter this when building the Eclipse platform in Fedora Linux build infrastructure, but I will try to generate a more concise test case...

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-packaging-plugin:0.23.0:package-plugin (default-package-plugin) on project org.eclipse.platform.doc.isv: Error assembling JAR: Problem creating jar: /builddir/build/BUILD/R4_platform-aggregator-I20150601-2000/eclipse.platform.common/bundles/org.eclipse.platform.doc.isv/reference/api/org/eclipse/core/databinding/observable/class-use/Observables.html (Too many open files) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.eclipse.tycho:tycho-packaging-plugin:0.23.0:package-plugin (default-package-plugin) on project org.eclipse.platform.doc.isv: Error assembling JAR
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error assembling JAR
        at org.eclipse.tycho.packaging.PackagePluginMojo.createPluginJar(PackagePluginMojo.java:243)
        at org.eclipse.tycho.packaging.PackagePluginMojo.execute(PackagePluginMojo.java:154)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 20 more
Caused by: org.codehaus.plexus.archiver.ArchiverException: Problem creating jar: /builddir/build/BUILD/R4_platform-aggregator-I20150601-2000/eclipse.platform.common/bundles/org.eclipse.platform.doc.isv/reference/api/org/eclipse/core/databinding/observable/class-use/Observables.html (Too many open files)
        at org.codehaus.plexus.archiver.AbstractArchiver.createArchive(AbstractArchiver.java:933)
        at org.apache.maven.archiver.MavenArchiver.createArchive(MavenArchiver.java:589)
        at org.eclipse.tycho.packaging.PackagePluginMojo.createPluginJar(PackagePluginMojo.java:238)
        ... 23 more
Caused by: java.io.FileNotFoundException: /builddir/build/BUILD/R4_platform-aggregator-I20150601-2000/eclipse.platform.common/bundles/org.eclipse.platform.doc.isv/reference/api/org/eclipse/core/databinding/observable/class-use/Observables.html (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.codehaus.plexus.components.io.resources.PlexusIoFileResource$1.getContents(PlexusIoFileResource.java:99)
        at org.codehaus.plexus.components.io.resources.PlexusIoFileResource.getContents(PlexusIoFileResource.java:124)
        at org.codehaus.plexus.archiver.ArchiveEntry.getInputStream(ArchiveEntry.java:126)
        at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.zipFile(AbstractZipArchiver.java:492)
        at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.addResources(AbstractZipArchiver.java:365)
        at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.createArchiveMain(AbstractZipArchiver.java:321)
        at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.execute(AbstractZipArchiver.java:222)
        at org.codehaus.plexus.archiver.AbstractArchiver.createArchive(AbstractArchiver.java:916)
        ... 25 more

[krosenvold]

It's definitely the parallel algorithm consuming more file handles in parallel. How many file handles do you have available ?

[mbooth101]

My hard and soft open files limits are 4096 and 1024, respectively.

The jar has >10,000 files and this has never been a problem before. Increasing these limits is not a solution since not only is this a regression in behaviour, but the number of files in this jar is likely to increase over time.

However, in my tests I only see this in a virtualised environment, not on a real system. My initial thought is that the slower IO could be keeping files open longer so the number open concurrently is higher than it would be on real hardware. Does this sound plausible?

[mbooth101]

Please see pull request #7 that adds a test case to illustrate this problem. (Simple test to add 15000 files to a compressed zip archive.)

This reproduces the failure for me on real hardware, not only VM

[krosenvold]

Issues fixed on 2.x branch. Will be merged to master (3.x) "soon". Thanks for the excellent testcase !