Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ensure input stream is closed when remapping classes #587

Merged
merged 1 commit into from
Oct 5, 2020
Merged

ensure input stream is closed when remapping classes #587

merged 1 commit into from
Oct 5, 2020

Conversation

roxchkplusony
Copy link
Contributor

@roxchkplusony roxchkplusony commented Jul 11, 2020 •
edited
Loading

One more time, @johnrengelman. I hope this solves once and for all the input stream issue that has plagued us over the years trying to stay current on Gradle. Please take a look - I'm not sure how you would approach testing this since it would require enough remapped classes to open sufficiently many files.

Note the culprit of our build failure this time is: https://github.com/johnrengelman/shadow/blob/master/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/tasks/ShadowCopyAction.groovy#L331

My exception (not present with the fix):

[04:50:59][Step 1/3] org.gradle.api.GradleException: Could not add file '/home/teamcity/teamcity/work/728364bcea5b3a67/pipeline/platforms/db240/rdd/build/classes/scala/main/com/paxata/spark/ast/SampleByAST$$anonfun$76.class' to ZIP '/home/teamcity/teamcity/work/728364bcea5b3a67/pipeline/platforms/db240/rdd/build/libs/pipeline.jar'.
[04:50:59][Step 1/3] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[04:50:59][Step 1/3] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[04:50:59][Step 1/3] 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[04:50:59][Step 1/3] 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:257)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction$StreamAction.visitFile(ShadowCopyAction.groovy:246)
[04:50:59][Step 1/3] 	at sun.reflect.GeneratedMethodAccessor635.invoke(Unknown Source)
[04:50:59][Step 1/3] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[04:50:59][Step 1/3] 	at java.lang.reflect.Method.invoke(Method.java:498)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction$BaseStreamAction.processFile(ShadowCopyAction.groovy:184)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.NormalizingCopyActionDecorator.lambda$execute$0(NormalizingCopyActionDecorator.java:68)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.DuplicateHandlingCopyActionDecorator.lambda$execute$0(DuplicateHandlingCopyActionDecorator.java:64)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopyFileVisitorImpl.processFile(CopyFileVisitorImpl.java:64)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopyFileVisitorImpl.visitFile(CopyFileVisitorImpl.java:48)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.DefaultDirectoryWalker$PathVisitor.visitFile(DefaultDirectoryWalker.java:113)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.DefaultDirectoryWalker$PathVisitor.visitFile(DefaultDirectoryWalker.java:66)
[04:50:59][Step 1/3] 	at java.nio.file.Files.walkFileTree(Files.java:2670)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.DefaultDirectoryWalker.walkDir(DefaultDirectoryWalker.java:60)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:149)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.DirectoryFileTree.visitFrom(DirectoryFileTree.java:127)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.DirectoryFileTree.visit(DirectoryFileTree.java:112)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.collections.FileTreeAdapter.visit(FileTreeAdapter.java:110)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.CompositeFileTree.visit(CompositeFileTree.java:93)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopySpecActionImpl.execute(CopySpecActionImpl.java:40)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopySpecActionImpl.execute(CopySpecActionImpl.java:24)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.DefaultCopySpec$DefaultCopySpecResolver.walk(DefaultCopySpec.java:683)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.DefaultCopySpec$DefaultCopySpecResolver.walk(DefaultCopySpec.java:685)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.DefaultCopySpec.walk(DefaultCopySpec.java:477)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopySpecBackedCopyActionProcessingStream.process(CopySpecBackedCopyActionProcessingStream.java:39)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.DuplicateHandlingCopyActionDecorator.lambda$execute$1(DuplicateHandlingCopyActionDecorator.java:43)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.NormalizingCopyActionDecorator.lambda$execute$1(NormalizingCopyActionDecorator.java:60)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopyActionProcessingStream$process.call(Unknown Source)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction$2.execute(ShadowCopyAction.groovy:111)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction$2$execute.call(Unknown Source)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction.withResource(ShadowCopyAction.groovy:153)
[04:50:59][Step 1/3] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[04:50:59][Step 1/3] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[04:50:59][Step 1/3] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[04:50:59][Step 1/3] 	at java.lang.reflect.Method.invoke(Method.java:498)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:149)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:100)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:216)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction.execute(ShadowCopyAction.groovy:108)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.NormalizingCopyActionDecorator.execute(NormalizingCopyActionDecorator.java:59)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.DuplicateHandlingCopyActionDecorator.execute(DuplicateHandlingCopyActionDecorator.java:43)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.file.copy.CopyActionExecuter.execute(CopyActionExecuter.java:40)
[04:50:59][Step 1/3] 	at org.gradle.api.tasks.AbstractCopyTask.copy(AbstractCopyTask.java:186)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar.copy(ShadowJar.java:115)
[04:50:59][Step 1/3] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[04:50:59][Step 1/3] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[04:50:59][Step 1/3] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[04:50:59][Step 1/3] 	at java.lang.reflect.Method.invoke(Method.java:498)
[04:50:59][Step 1/3] 	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:42)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:721)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:688)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:539)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:524)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:507)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$300(ExecuteActionsTaskExecuter.java:109)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:258)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:247)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$1(ExecuteStep.java:33)
[04:50:59][Step 1/3] 	at java.util.Optional.orElseGet(Optional.java:267)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:63)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:35)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:49)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:34)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:43)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CatchExceptionStep.execute(CatchExceptionStep.java:34)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:44)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:54)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:38)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.executeWithoutCache(CacheStep.java:153)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.executeAndStoreInCache(CacheStep.java:129)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.lambda$executeWithCache$2(CacheStep.java:107)
[04:50:59][Step 1/3] 	at java.util.Optional.orElseGet(Optional.java:267)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.lambda$executeWithCache$3(CacheStep.java:107)
[04:50:59][Step 1/3] 	at org.gradle.internal.Try$Success.map(Try.java:162)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.executeWithCache(CacheStep.java:76)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:66)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:41)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:49)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:44)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:33)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:38)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:24)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:92)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:85)
[04:50:59][Step 1/3] 	at java.util.Optional.map(Optional.java:215)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:39)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:76)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:37)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:36)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:26)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:94)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:49)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:79)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:53)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:74)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:78)
[04:50:59][Step 1/3] 	at java.util.Optional.orElseGet(Optional.java:267)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:78)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:39)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:40)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:28)
[04:50:59][Step 1/3] 	at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:174)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:166)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
[04:50:59][Step 1/3] 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
[04:50:59][Step 1/3] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
[04:50:59][Step 1/3] 	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
[04:50:59][Step 1/3] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:374)
[04:50:59][Step 1/3] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:361)
[04:50:59][Step 1/3] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:354)
[04:50:59][Step 1/3] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:340)
[04:50:59][Step 1/3] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
[04:50:59][Step 1/3] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
[04:50:59][Step 1/3] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
[04:50:59][Step 1/3] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
[04:50:59][Step 1/3] 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
[04:50:59][Step 1/3] 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
[04:50:59][Step 1/3] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[04:50:59][Step 1/3] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[04:50:59][Step 1/3] 	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
[04:50:59][Step 1/3] 	at java.lang.Thread.run(Thread.java:748)
[04:50:59][Step 1/3] Caused by: java.io.FileNotFoundException: /home/teamcity/teamcity/work/728364bcea5b3a67/pipeline/platforms/db240/rdd/build/classes/scala/main/com/paxata/spark/ast/SampleByAST$$anonfun$76.class (Too many open files)
[04:50:59][Step 1/3] 	at java.io.FileInputStream.open0(Native Method)
[04:50:59][Step 1/3] 	at java.io.FileInputStream.open(FileInputStream.java:195)
[04:50:59][Step 1/3] 	at java.io.FileInputStream.<init>(FileInputStream.java:138)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.ResourceGroovyMethods.newInputStream(ResourceGroovyMethods.java:2139)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.dgm$1026.invoke(Unknown Source)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction$StreamAction.remapClass(ShadowCopyAction.groovy:331)
[04:50:59][Step 1/3] 	at sun.reflect.GeneratedMethodAccessor643.invoke(Unknown Source)
[04:50:59][Step 1/3] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[04:50:59][Step 1/3] 	at java.lang.reflect.Method.invoke(Method.java:498)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
[04:50:59][Step 1/3] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
[04:50:59][Step 1/3] 	at com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction$StreamAction.visitFile(ShadowCopyAction.groovy:242)
[04:50:59][Step 1/3] 	... 155 more42)

@roxchkplusony
Copy link
Contributor Author

@johnrengelman it's been a couple months. Are you able to look at this or is there another authorized contributor?

Armaxis
Armaxis approved these changes Sep 28, 2020
View reviewed changes
Copy link
Contributor

@Armaxis Armaxis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this! We've stumbled upon the same leak, and this fix worked!
Previously, the number of opened file descriptors climbed all the way to 10k, unless GC kicked off and closed those. Occasionally this cause a crash as described.
With this fix the number of opened file descriptors is stabilized around 1470 👍
@johnrengelman can we please have this fix included in next release?

@johnrengelman johnrengelman added this to the 6.1 milestone Oct 5, 2020
@johnrengelman johnrengelman merged commit 05e954a into GradleUp:master Oct 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Armaxis Armaxis Armaxis approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
6.1
Development

Successfully merging this pull request may close these issues.
3 participants
@roxchkplusony @Armaxis @johnrengelman