Jenkins OOMS for Java 17 + Eclipse 4.27

[cdietrich]

Jenkins OOMS for Java 17 + Eclipse 4.27
https://ci.eclipse.org/xtext/job/xtext-eclipse/job/master/3214/console

12:00:36  testCamelCase_03(org.eclipse.xtext.xbase.ui.tests.editor.ContentAssistReturnTest2)  Time elapsed: 0.081 s
12:00:36  Running org.eclipse.xtext.xbase.ui.tests.editor.ContentAssistFinallyTest
12:01:23  Exception in thread "Active Thread: Equinox Container: 9a2421ac-9094-44c9-88c2-23556f9abd43" java.lang.OutOfMemoryError: Java heap space
12:01:27  Exception in thread "Java indexing" java.lang.OutOfMemoryError: Java heap space
12:01:45  Exception in thread "Java indexing" java.lang.OutOfMemoryError: Java heap space
12:03:21  
12:03:21  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Java indexing"
12:04:08  
12:04:08  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-3-thread-3"
12:05:04  
12:05:04  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-4"
12:05:16  
12:05:16  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-3-thread-1"
12:06:13  
12:06:13  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-5"
12:06:39  Tests run: 74, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 355.228 s <<< FAILURE! - in org.eclipse.xtext.xbase.ui.tests.editor.ContentAssistFinallyTest

.log file is here https://ci.eclipse.org/xtext/job/xtext-eclipse/job/master/3214/artifact/org.eclipse.xtext.xbase.ui.tests/target/work/data/.metadata/.log

[cdietrich]

TODO:

• can this be reproduced locally
• can we obtain a heapdump
• is this a regression in platform?

[cdietrich]

can be reproduced locally.

[cdietrich]

have the feeling closing the editors does not longer work

[cdietrich]

no the document seems to leak somewhere else

[cdietrich]

@szarnekow do you have any platform change in mind that could cause this?

[iloveeclipse]

Since which IBuild is it? There were few PR's for platform text merged recently.

• Add extension point for configuring ITextDoubleClickStrategy. eclipse-platform/eclipse.platform.text#118
• Register IReconcilingStrategy with GenericEditor extension point.  eclipse-platform/eclipse.platform.text#115

I haven't checked what is inside, but they seem to be big enough to introduce a leak somewhere.

[cdietrich]

the first time i observed it is today. need to check older logs

[iloveeclipse]

Regarding JDT: jdt.core tests run with 1 GB heap, and that was a good memory leak detector in the past, and there were no significant changes in JDT UI AFAIK

[cdietrich]

ok a lot of backtraces looks like

so there seems to be a job with a link to the reconciler

[cdietrich]

of course we dont see a previous run on jenkins cause the history was pruned

[szarnekow]

Looks like the jobmanager keeps a reference around?

[cdietrich]

dump can be found at
https://drive.google.com/file/d/1qnFc-upkSQzK09sl68DOY3JWiWhGHa1-/view?usp=share_link

[cdietrich]

to me it looks like the pending jobs dont really reduce. but i also dont see a difference to older versions.
i wonder if we need to call something in the test to let the notifyListeners run.
maybe there is also something that takes the listeners to run longer, so that the trottler actually throttles

[cdietrich]

adding an additional read and dispatch loop to setup the the tests seems to help. but what has changed ...

[cdietrich]

and of course i cannot oomph against older target :(
#1960

[cdietrich]

now it gets funny. i see the job list also grow in 2022-03. will take a snapshot there too.
there seem to be no reconciler leaks ... :(

[cdietrich]

there seem to be no job infos with reconciler in older versions. need to debug who creates them on latest.

[cdietrich]

stacktrace where the job infos are created

JobInfo.<init>(Job) line: 57	
ProgressManager.lambda$14(Job) line: 613	
0x0000000800f77580.apply(Object) line: not available	
HashMap<K,V>.computeIfAbsent(K, Function<? super K,? extends V>) line: 1220	
ProgressManager.progressFor(Job) line: 613	
ProgressManager$1.updateFor(IJobChangeEvent) line: 502	
ProgressManager$1.scheduled(IJobChangeEvent) line: 474	
0x0000000800db9138.notify(IJobChangeListener, IJobChangeEvent) line: not available	
JobListeners.sendEvent(JobChangeEvent) line: 131	
JobListeners.sendEventsAsync(InternalJob) line: 118	
JobListeners.waitAndSendEvents(InternalJob, boolean) line: 78	
JobManager.withWriteLock(J, Function<J,T>) line: 524	
JobManager.schedule(InternalJob, long) line: 1373	
XtextReconciler(InternalJob).schedule(long) line: 396	
XtextReconciler(Job).schedule(long) line: 684	
XtextReconciler.resume() line: 345	
XtextReconciler$DocumentListener.assistSessionEnded(ContentAssistEvent) line: 180	
ContentAssistant.lambda$12(IContentAssistProcessor) line: 2521	
0x000000080188dd20.accept(Object) line: not available	
Collections$SingletonSet<E>.forEach(Consumer<? super E>) line: 4905	
ContentAssistant.fireSessionEndEvent() line: 2518	
CompletionProposalPopup.hide() line: 1110	
ContentAssistant.hide() line: 2267	
ContentAssistant.uninstall() line: 1609	
XtextSourceViewer(SourceViewer).uninstallTextViewer() line: 1346	
XtextSourceViewer(SourceViewer).unconfigure() line: 729	
XtextSourceViewer(SourceViewer).handleDispose() line: 789	
XtextSourceViewer(ProjectionViewer).handleDispose() line: 1303	
XtextSourceViewer(TextViewer).lambda$1(DisposeEvent) line: 1741	

[cdietrich]

on older versions
1st: org.eclipse.jface.text.contentassist.ContentAssistant.removeCompletionListener(ICompletionListener)
2nd : org.eclipse.jface.text.contentassist.ContentAssistant.fireSessionEndEvent()

on latest the remove seems not come before the fire

[LorenzoBettini]

At least in SWTBot tests, it helps (at least it reduces flakyness) to wait for jobs to finish, by looking at boolean idle = Job.getJobManager().isIdle();. SWTBot tests don't run in the UI tests, while these tests do, so the wait should be adapted.

[cdietrich]

to me it looks like the order of swt events has changed
i have zero clue why.

[cdietrich]

bingo: the old code removes the reconciler first, content assistant 2nd
the new does remove the content assistant first.

@iloveeclipse @angelozerr is this an intentional change with
eclipse-platform/eclipse.platform.text@81a4465

[cdietrich]

@iloveeclipse @angelozerr should this be a linked hashset in platform instead?

[angelozerr]

@iloveeclipse @angelozerr is this an intentional change with
eclipse-platform/eclipse.platform.text@81a4465

no, we wanted that all feature like reconciler, content assist use the same pattern lifecycle.

@iloveeclipse @angelozerr should this be a linked hashset in platform instead?

I think it can be a good idea, is it working better with linked hash map?

[cdietrich]

i will have a look later. we might then also have to iterate reverse

@szarnekow can you also check if there is something bogus with our reaction to the content assistant uninstall?
or is the content assistant also created while the editor stays open?

[szarnekow]

I think it can be a good idea, is it working better with linked hash map?

I'd rather use a list and walk it backwards during uninstall.

[angelozerr]

I'd rather use a list and walk it backwards during uninstall.

I created a PR eclipse-platform/eclipse.platform.text#121

[cdietrich]

pr looks good