Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add JMH benchmark #88

Closed
viartemev opened this issue Feb 13, 2019 · 8 comments
Closed

Add JMH benchmark #88

viartemev opened this issue Feb 13, 2019 · 8 comments
Assignees
@viartemev
Labels
enhancement New feature or request
Milestone
0.0.2

Comments

@viartemev
Copy link
Owner

Add JHM benchmark and compare it with https://github.com/reactor/reactor-rabbitmq
@viartemev viartemev added the enhancement New feature or request label Feb 13, 2019
@viartemev viartemev self-assigned this Feb 13, 2019
@viartemev viartemev mentioned this issue Feb 25, 2019
Merged
@viartemev viartemev added this to the 0.0.2 milestone Mar 6, 2019
@viartemev
Copy link
Owner Author

viartemev commented Mar 7, 2019 •
edited
Loading

Configuration:

@Warmup(iterations = 5, time = 5, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 1, time = 5, timeUnit = TimeUnit.SECONDS)
@Fork(value = 2)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)

The White Rabbit results:

Benchmark                                         (numberOfMessages)  Mode  Cnt        Score   Error  Units
ConfirmPublisherBenchmark.sendWithPublishConfirm                   1  avgt    2      822,326          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                  10  avgt    2     1669,531          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                 100  avgt    2    12271,187          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                1000  avgt    2    71214,000          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm               10000  avgt    2   672646,545          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm              100000  avgt    2  6533520,122          us/op

Reactor RabbitMQ results:

Benchmark                                (nbMessages)  Mode  Cnt       Score   Error  Units
SenderBenchmark.sendWithPublishConfirms             1  avgt    2     3886,563          us/op
SenderBenchmark.sendWithPublishConfirms            10  avgt    2     5633,654          us/op
SenderBenchmark.sendWithPublishConfirms           100  avgt    2    17744,681          us/op
SenderBenchmark.sendWithPublishConfirms          1000  avgt    2    86137,186          us/op
SenderBenchmark.sendWithPublishConfirms         10000  avgt    2   696657,341          us/op
SenderBenchmark.sendWithPublishConfirms        100000  avgt    2  7524017,626          us/op

@viartemev viartemev mentioned this issue Mar 7, 2019
Merged
@viartemev
Copy link
Owner Author

A link to Reactor RabbitMQ benchmark: https://github.com/reactor/reactor-rabbitmq/blob/master/src/jmh/java/reactor/rabbitmq/SenderBenchmark.java

@asm0dey
Copy link
Collaborator

asm0dey commented Mar 7, 2019

@acogoluegnes we would like to ask you to review this test if you'll have time and desire. Critics is absolutely welcome.
Also if you'll find incorrect usage of anything rabbit-related — we will be happy to try to fix it

@acogoluegnes
Copy link

acogoluegnes commented Mar 8, 2019 •
edited
Loading

@asm0dey @viartemev Thanks for bringing up this topic.

Here are the results I got:

$ uname -a
Linux acogoluegnes-inspiron 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# JMH version: 1.21
# VM version: JDK 11.0.2, OpenJDK 64-Bit Server VM, 11.0.2+7-LTS

The White Rabbit

Benchmark                                         (numberOfMessages)  Mode  Cnt        Score   Error  Units
ConfirmPublisherBenchmark.sendWithPublishConfirm                   1  avgt    2      104.172          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                  10  avgt    2      598.625          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                 100  avgt    2     3845.833          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                1000  avgt    2    36108.709          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm               10000  avgt    2   392132.353          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm              100000  avgt    2  4098567.349          us/op

Reactor RabbitMQ

Benchmark                                (nbMessages)  Mode  Cnt        Score   Error  Units
SenderBenchmark.sendWithPublishConfirms             1  avgt    2      678.301          us/op
SenderBenchmark.sendWithPublishConfirms            10  avgt    2     1122.558          us/op
SenderBenchmark.sendWithPublishConfirms           100  avgt    2     4809.979          us/op
SenderBenchmark.sendWithPublishConfirms          1000  avgt    2    39819.682          us/op
SenderBenchmark.sendWithPublishConfirms         10000  avgt    2   387274.332          us/op
SenderBenchmark.sendWithPublishConfirms        100000  avgt    2  4055719.579          us/op

The White Rabbit is faster for small number of messages (10-), maybe because Reactor RabbitMQ initializes a bunch of staff compared to the amount of actual processing. For higher number of messages, both libraries behave roughly the same.

Both libraries use the RabbitMQ Java client, so I guess it all comes down to the respective extra layer (Kotlin/Reactor) and the publish confirms handling.

Note I got a few errors when running the White Rabbit benchmarks (during warm-up, I don't think it affects the results):

# Run progress: 0.00% complete, ETA 00:06:00
# Fork: 1 of 2

119.738 us/op
# Warmup Iteration   2: 104.683 us/op
# Warmup Iteration   3: 103.852 us/op
# Warmup Iteration   4: 102.921 us/op
# Warmup Iteration   5: 102.284 us/op
Iteration   1: 103.689 us/opG [26s]
<JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 24 seconds more...>

Non-finished threads:

Thread[pool-4-thread-1,5,main]
  at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  at [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  at [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at [email protected]/java.lang.Thread.run(Thread.java:834)

Thread[DestroyJavaVM,5,main]


<JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 19 seconds more...>

Non-finished threads:

Thread[pool-4-thread-1,5,main]
  at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  at [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  at [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at [email protected]/java.lang.Thread.run(Thread.java:834)

Thread[DestroyJavaVM,5,main]


<JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 14 seconds more...>

...

<shutdown timeout of 30 seconds expired, forcing forked VM to exit>

@acogoluegnes
Copy link

Gave another try after changing the publishing handling in Reactor RabbitMQ to use the same algorithm and data structure as in the White Rabbit:

The White Rabbit

Benchmark                                         (numberOfMessages)  Mode  Cnt        Score   Error  Units
ConfirmPublisherBenchmark.sendWithPublishConfirm                   1  avgt    2      104.172          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                  10  avgt    2      598.625          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                 100  avgt    2     3845.833          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                1000  avgt    2    36108.709          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm               10000  avgt    2   392132.353          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm              100000  avgt    2  4098567.349          us/op

Benchmark                                (nbMessages)  Mode  Cnt        Score   Error  Units
SenderBenchmark.sendWithPublishConfirms             1  avgt    2      697.424          us/op
SenderBenchmark.sendWithPublishConfirms            10  avgt    2     1306.490          us/op
SenderBenchmark.sendWithPublishConfirms           100  avgt    2     4819.441          us/op
SenderBenchmark.sendWithPublishConfirms          1000  avgt    2    39597.671          us/op
SenderBenchmark.sendWithPublishConfirms         10000  avgt    2   373226.865          us/op
SenderBenchmark.sendWithPublishConfirms        100000  avgt    2  3900685.520          us/op

Pretty much the same conclusion for small number number of messages, but the White Rabbit publish confirms algorithm makes Reactor RabbitMQ a bit faster for large number of messages (10K+).

@asm0dey
Copy link
Collaborator

asm0dey commented Mar 8, 2019

@acogoluegnes cool, may I see please changes you used in Reactor RabbitMQ? Just curious what was written better
Also any ideas on nature of those exceptions?

@viartemev I tend to think that it's the difference between Linux and Mac hosts. Will benchmark on my laptop too.

@acogoluegnes
Copy link

@asm0dey I pushed the changes reactor/reactor-rabbitmq@9404592

It may be due a thread that is spawned but no properly stopped. This doesn't happen in the Reactor RabbitMQ benchmark run on Linux, so it must something related to the White Rabbit library.

@asm0dey
Copy link
Collaborator

asm0dey commented Mar 8, 2019 •
edited
Loading

@acogoluegnes I'm happy to read that my idea on confirmed publication helps your implementation too :)

@viartemev viartemev mentioned this issue Mar 12, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@viartemev viartemev

Labels
enhancement New feature or request
Projects
None yet
Milestone
0.0.2
Development

No branches or pull requests
3 participants
@acogoluegnes @asm0dey @viartemev