Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][broker] Use shrink map for message redelivery. #15342

Merged
merged 2 commits into from
Apr 28, 2022

Conversation

Technoboy-
Copy link
Contributor

Motivation

Sometimes the messages sent to consumers are delayed over 100ms and we find that the CPU is wasted on :

"BookKeeperClientWorker-OrderedExecutor-4-0" #64 prio=5 os_prio=0 cpu=2447920216.81ms elapsed=3636758.63s tid=0x00007f48bdc07290 nid=0xae runnable  [0x00007f47ee2e2000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.bookkeeper.util.collections.ConcurrentLongLongPairHashMap.forEach(ConcurrentLongLongPairHashMap.java:197)
	at org.apache.pulsar.broker.service.persistent.MessageRedeliveryController.containsStickyKeyHashes(MessageRedeliveryController.java:97)
	at org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumers.getRestrictedMaxEntriesForConsumer(PersistentStickyKeyDispatcherMultipleConsumers.java:335)
	at org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumers.sendMessagesToConsumers(PersistentStickyKeyDispatcherMultipleConsumers.java:232)
	at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.readEntriesComplete(PersistentDispatcherMultipleConsumers.java:480)
	- locked <0x0000100117401718> (a org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumers)
	at org.apache.bookkeeper.mledger.impl.OpReadEntry.lambda$checkReadCompletion$2(OpReadEntry.java:156)

Then we find there are many empty datas in the map from dump :
image

As #14515 has introduced shrink map, it's better to use it to avoid iterator the empty data.

Documentation

  • no-need-doc

@Technoboy- Technoboy- self-assigned this Apr 27, 2022
@Technoboy- Technoboy- changed the title Use shrink map for message redelivery. [improve][broker] Use shrink map for message redelivery. Apr 27, 2022
@Technoboy- Technoboy- added area/broker doc-not-needed Your PR changes do not impact docs labels Apr 27, 2022
@Technoboy- Technoboy- added this to the 2.11.0 milestone Apr 27, 2022
@codelipenghui
Copy link
Contributor

I have added release labels, it's a serious problem that will cause the thread BookKeeperClientWorker-OrderedExecutor to 100% CPU usage.

@Technoboy- Technoboy- merged commit 615f05a into apache:master Apr 28, 2022
codelipenghui pushed a commit that referenced this pull request Apr 28, 2022
codelipenghui pushed a commit that referenced this pull request Apr 28, 2022
@codelipenghui codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Apr 28, 2022
codelipenghui pushed a commit that referenced this pull request Apr 29, 2022
@codelipenghui codelipenghui added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Apr 29, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request May 4, 2022
(cherry picked from commit 615f05a)
(cherry picked from commit b4762d1)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request May 9, 2022
(cherry picked from commit 615f05a)
(cherry picked from commit 9352feb)
@Technoboy- Technoboy- deleted the replace-to-common-long-map branch August 10, 2022 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants