-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix schedule task memory leak in ManagedLedgerFactoryImpl #13134
Fix schedule task memory leak in ManagedLedgerFactoryImpl #13134
Conversation
/pulsarbot run-failure-checks |
Can you share more details about the symptoms? Which Pulsar versions are impacted? |
@lhotari In current branch we maintained, version 2.8 and version 2.9 are impacted. |
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Good catch @Shoothzj |
@@ -353,6 +353,17 @@ public void asyncOpen(final String name, final ManagedLedgerConfig config, final | |||
log.warn("[{}] Attempted to open ledger in {} state. Removing from the map to recreate it", | |||
name, l.getState()); | |||
ledgers.remove(name, existingFuture); | |||
l.asyncClose(new CloseCallback() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, if the managedLedger in Fenced
or Closed
state, call asyncClose
just
pulsar/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
Lines 1316 to 1328 in ee90406
public synchronized void asyncClose(final CloseCallback callback, final Object ctx) { | |
State state = STATE_UPDATER.get(this); | |
if (state == State.Fenced) { | |
factory.close(this); | |
callback.closeFailed(new ManagedLedgerFencedException(), ctx); | |
return; | |
} else if (state == State.Closed) { | |
if (log.isDebugEnabled()) { | |
log.debug("[{}] Ignoring request to close a closed managed ledger", name); | |
} | |
callback.closeComplete(ctx); | |
return; | |
} |
So, should we just call ManagedLedgerFactoryImpl::close
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I guess calling asyncClose is sufficient. However there seems to be a bug in the asyncClose method when the ledger is fenced. The method call to cancel the scheduled tasks is missing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
badb576
to
f40684c
Compare
@@ -1192,6 +1192,7 @@ private long consumedLedgerSize(long ledgerSize, long ledgerEntries, long consum | |||
@Override | |||
public synchronized void asyncTerminate(TerminateCallback callback, Object ctx) { | |||
if (state == State.Fenced) { | |||
cancelScheduledTasks(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems that similar addition would be needed to asyncClose when the ledger is fenced
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, yesterday,I found It has been fixed in #12565, it's better to cancel the schedule task when the ledger is fenced. I will test if it fix the memomry leak.
Motivation
The memory is leak due to the ManagerLedgerImpl can't be garbage collected.
Modifications
Before remove the
managedLedgerImpl
from map, callclose()
to cacel the schdule job first.Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
no-need-doc
Fix a memory leak bug