Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More optimizations for InFlightRequest #362

Merged
merged 12 commits into from
Jan 30, 2020

Conversation

jasper-d
Copy link
Contributor

Based on the optimizations done by @danielwertheim I further optimized InFlightRequest for use cases w/o a timeout (i.e. when using a client provided cancellation token).

Notable changes:

  • Moved InFlightRequest into a seperate file (no technical reason, just personal preference)
  • Made InFlightRequest internal
  • Added unit tests for InFlightRequest
  • Added some optimizations to reduce resource consumption if no timeout is specified. Now, a (linked) CTS is only created if a timeout is specified.
  • Further reduced allocations by avoiding closure over the TCS
  • Changed the cancellation behaviour to set a NATSTimeoutException only if cancellation wasn't caused by a client provided CT. This is a potentially breaking change. Before this, a NATSTimeoutException was always set as the TCS's exception when a timeout > 0 was passed in, even when a client provided CT caused cancellation (which may happen for any reason). Hence, clients that always used a timeout may not expect/handle the TaskCanceledException that now occurs if the client provided CT cancelled the request. However, I think that's better than throwing an unrelated exception.

Benchmark results

Source: https://github.com/jasper-d/nats.net/blob/benchmarks/inflightrequest/src/Benchmarks/MicroBenchmarks/InFlightRequestBenchmark.cs
The Categories column indicates whether CancellationToken.None (DefaultToken) or a "hot" CT (ClientToken) was passed to InFlightRequest. The Timeout column refers to the timeout arguments value passed to InFlightRequest.

The suffix _Upstream indicates that the current (less optimized) version of InFlightRequest was used.

Windows

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18362
Intel Core i7-4712MQ CPU 2.30GHz (Haswell), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-UIGYTS : .NET Framework 4.8 (4.8.4121.0), X64 RyuJIT
  Job-YMFKNL : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Method Runtime Categories Timeout Mean Error StdDev Median Ratio RatioSD Gen 0 Gen 1 Gen 2 Allocated
InFlightRequest .NET 4.6.2 DefaultToken 0 81.81 ns 0.171 ns 0.160 ns 81.78 ns 0.22 0.00 0.1019 - - 321 B
InFlightRequest_Upstream .NET 4.6.2 DefaultToken 0 369.31 ns 0.417 ns 0.370 ns 369.39 ns 1.00 0.00 0.2193 - - 690 B
InFlightRequest_ClientToken .NET 4.6.2 ClientToken 0 235.39 ns 0.873 ns 0.817 ns 235.30 ns 0.46 0.00 0.1197 - - 377 B
InFlightRequest_ClientToken_Upstream .NET 4.6.2 ClientToken 0 516.48 ns 0.474 ns 0.443 ns 516.59 ns 1.00 0.00 0.2623 - - 826 B
InFlightRequest .NET Core 3.1 DefaultToken 0 71.51 ns 0.145 ns 0.121 ns 71.53 ns 0.32 0.00 0.0969 - - 304 B
InFlightRequest_Upstream .NET Core 3.1 DefaultToken 0 226.44 ns 2.522 ns 2.236 ns 226.53 ns 1.00 0.00 0.1938 - - 608 B
InFlightRequest_ClientToken .NET Core 3.1 ClientToken 0 155.03 ns 3.099 ns 2.899 ns 153.63 ns 0.46 0.01 0.0968 - - 304 B
InFlightRequest_ClientToken_Upstream .NET Core 3.1 ClientToken 0 337.62 ns 6.852 ns 6.074 ns 335.84 ns 1.00 0.00 0.2089 - - 656 B
InFlightRequest .NET 4.6.2 DefaultToken 999999 822.57 ns 5.931 ns 4.953 ns 820.88 ns 1.00 0.02 0.2594 - - 818 B
InFlightRequest_Upstream .NET 4.6.2 DefaultToken 999999 823.71 ns 16.364 ns 16.072 ns 818.51 ns 1.00 0.00 0.2651 - - 834 B
InFlightRequest_ClientToken .NET 4.6.2 ClientToken 999999 1,026.69 ns 19.878 ns 23.664 ns 1,027.80 ns 1.01 0.04 0.3033 - - 955 B
InFlightRequest_ClientToken_Upstream .NET 4.6.2 ClientToken 999999 1,012.73 ns 19.380 ns 18.128 ns 1,005.26 ns 1.00 0.00 0.3071 - - 971 B
InFlightRequest .NET Core 3.1 DefaultToken 999999 440.01 ns 8.882 ns 21.109 ns 433.87 ns 0.96 0.07 0.2193 - - 688 B
InFlightRequest_Upstream .NET Core 3.1 DefaultToken 999999 459.33 ns 10.002 ns 21.955 ns 454.94 ns 1.00 0.00 0.2241 - - 704 B
InFlightRequest_ClientToken .NET Core 3.1 ClientToken 999999 579.46 ns 13.371 ns 38.579 ns 566.98 ns 1.01 0.07 0.2346 - - 736 B
InFlightRequest_ClientToken_Upstream .NET Core 3.1 ClientToken 999999 577.11 ns 11.588 ns 26.626 ns 571.71 ns 1.00 0.00 0.2394 - - 752 B

Linux

BenchmarkDotNet=v0.12.0, OS=fedora 31
Intel Core i7-3612QM CPU 2.10GHz (Ivy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-KAVYNT : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Runtime=.NET Core 3.1
Method Categories Timeout Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
InFlightRequest DefaultToken 0 114.2 ns 0.23 ns 0.20 ns 0.34 0.0968 - - 304 B
InFlightRequest_Upstream DefaultToken 0 338.2 ns 2.31 ns 1.93 ns 1.00 0.1936 - - 608 B
InFlightRequest_ClientToken ClientToken 0 219.8 ns 1.11 ns 0.99 ns 0.47 0.0968 - - 304 B
InFlightRequest_ClientToken_Upstream ClientToken 0 465.3 ns 1.42 ns 1.33 ns 1.00 0.2089 - - 656 B
InFlightRequest DefaultToken 999999 537.0 ns 4.44 ns 4.15 ns 0.92 0.2193 - - 688 B
InFlightRequest_Upstream DefaultToken 999999 585.1 ns 0.37 ns 0.31 ns 1.00 0.2241 - - 704 B
InFlightRequest_ClientToken ClientToken 999999 674.1 ns 0.67 ns 0.62 ns 0.98 0.2346 - - 736 B
InFlightRequest_ClientToken_Upstream ClientToken 999999 688.4 ns 5.09 ns 4.51 ns 1.00 0.2394 - - 752 B

Coalesced CT results

Not part of this PR but used to validate the rationale. Benchmarks to investigate if coalesced CTs can provide a benefit (depends on the actual scenario). Depending on the use case, this might also help in cases like #299.
Code: https://github.com/jasper-d/nats.net/blob/benchmarks/inflightrequest/src/Benchmarks/MicroBenchmarks/CoalescedTokenBenchmark.cs
InFlightRequest uses the optimized InFlightRequest with a timeout > 0.
InFlightRequest_Upstream_Coalesced uses the current implementation with coalesced CTs.
InFlightRequest_Coalesced uses the optimized version with coalesced CTs.

Windows

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18362
Intel Core i7-4712MQ CPU 2.30GHz (Haswell), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-UIGYTS : .NET Framework 4.8 (4.8.4121.0), X64 RyuJIT
  Job-YMFKNL : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Method Runtime Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
InFlightRequest .NET 4.6.2 805.8 ns 6.38 ns 5.33 ns 1.00 0.2594 - - 818 B
InFlightRequest_Upstream_Coalesced .NET 4.6.2 564.4 ns 1.46 ns 1.37 ns 0.70 0.2623 - - 826 B
InFlightRequest_Coalesced .NET 4.6.2 290.6 ns 0.87 ns 0.82 ns 0.36 0.1197 - - 377 B
InFlightRequest .NET Core 3.1 417.1 ns 3.74 ns 3.50 ns 1.00 0.2193 - - 688 B
InFlightRequest_Upstream_Coalesced .NET Core 3.1 361.5 ns 0.93 ns 0.82 ns 0.87 0.2089 - - 656 B
InFlightRequest_Coalesced .NET Core 3.1 191.4 ns 0.58 ns 0.54 ns 0.46 0.0968 - - 304 B

Linux

BenchmarkDotNet=v0.12.0, OS=fedora 31
Intel Core i7-3612QM CPU 2.10GHz (Ivy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-FCUXKY : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Runtime=.NET Core 3.1
Method Mean Error StdDev Ratio Gen 0 Gen 1 Gen 2 Allocated
InFlightRequest 537.3 ns 5.43 ns 5.08 ns 1.00 0.2193 - - 688 B
InFlightRequest_Upstream_Coalesced 551.9 ns 0.89 ns 0.84 ns 1.03 0.2089 - - 656 B
InFlightRequest_Coalesced 285.7 ns 0.60 ns 0.47 ns 0.53 0.0968 - - 304 B

@watfordgnf
Copy link
Collaborator

I have no concerns with the original TaskCancellationException coming through for a client cancellation. @ColinSullivan1 @danielwertheim any objections?

Copy link
Collaborator

@watfordgnf watfordgnf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just one small question really.

src/NATS.Client/Internals/InFlightRequest.cs Show resolved Hide resolved
Copy link
Member

@ColinSullivan1 ColinSullivan1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

I'm usually resistant to breaking changes, but we're still pre 1.0, and it's the right thing to do. Expanding the exceptions thrown here shouldn't impact many users, if any at all.

@ColinSullivan1 ColinSullivan1 merged commit 3558232 into nats-io:master Jan 30, 2020
@jasper-d jasper-d deleted the inflightrequest branch January 30, 2020 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants