ensure performance of tools on Dart 2 VM is on par with Dart 1 #33257
Comments
mraleph
added
the
area-vm
|
The initial plan is two adapt AOT specific optimizations developed as part of #31798 to JIT:
|
For now we are limiting this to type checks against type parameter types. # Performance improvements In Dart 1 mode Dart2JS compiles itself in 28s when running from source and in 23s when running from ideal app-jit snapshot (trained on the same workload). Before this change in Dart 2 mode numbers were 51s and 57s respectively. After this change in Dart 2 mode numbers are 38s and 32s. Meaning that regression is reduced by 50%. Issue #31798 Issue #33257 Change-Id: I34bf5385a5cc3c7702dc281c6dfa89da85d3dde1 Reviewed-on: https://dart-review.googlesource.com/57601 Reviewed-by: Régis Crelier <[email protected]> Commit-Queue: Vyacheslav Egorov <[email protected]>
… types." This reverts commit 4be50d6. Reason for revert: Failures on SIMDBC64 and Analyzer bots. Original change's description: > [vm] Enable type stubs based type checks in JIT mode for some types. > > For now we are limiting this to type checks against type parameter types. > > # Performance improvements > > In Dart 1 mode Dart2JS compiles itself in 28s when running from source > and in 23s when running from ideal app-jit snapshot (trained on the > same workload). > > Before this change in Dart 2 mode numbers were 51s and 57s respectively. > > After this change in Dart 2 mode numbers are 38s and 32s. Meaning > that regression is reduced by 50%. > > Issue #31798 > Issue #33257 > > Change-Id: I34bf5385a5cc3c7702dc281c6dfa89da85d3dde1 > Reviewed-on: https://dart-review.googlesource.com/57601 > Reviewed-by: Régis Crelier <[email protected]> > Commit-Queue: Vyacheslav Egorov <[email protected]> [email protected],[email protected],[email protected] Change-Id: I85a30c962b0cd556310e19193f5993ab76ecf2e7 No-Presubmit: true No-Tree-Checks: true No-Try: true Reviewed-on: https://dart-review.googlesource.com/57840 Reviewed-by: Vyacheslav Egorov <[email protected]> Commit-Queue: Vyacheslav Egorov <[email protected]>
Relanding 4be50d6 with fixes to DBC and location summaries: AssertAssignable must save FPU registers. For now we are limiting this to type checks against type parameter types. In Dart 1 mode Dart2JS compiles itself in 28s when running from source and in 23s when running from ideal app-jit snapshot (trained on the same workload). Before this change in Dart 2 mode numbers were 51s and 57s respectively. After this change in Dart 2 mode numbers are 38s and 32s. Meaning that regression is reduced by 50%. Issue #31798 Issue #33257 Change-Id: Ifb55f86453bfdf36a2e03bcd7f3197cfde257103 Reviewed-on: https://dart-review.googlesource.com/57980 Commit-Queue: Vyacheslav Egorov <[email protected]> Reviewed-by: Régis Crelier <[email protected]>
|
Any further updates here, @mraleph ? |
|
The first fix has landed on Friday. The work on another one is starting. |
|
Any updates here? |
|
No updates at the moment. Still working on this. |
|
@handicraftsman please elaborate on the “hmm?” - leaving such comments is not really helpful |
|
Any news? |
|
Still working on this. Have another fix in the development that cuts another 50% of the regression. |
|
@handicraftsman is there any particular reason you are interested in this? does your application regress in Dart 2 mode? in this case it would be helpful if you shared some information - so that we could ensure that we address regressions in your applications as well. |
|
Sass's speed regresses considerably in Dart 2 mode as well. Here's some data, collected using the same techniques we use to collect cross-implementation performance data, tested against Dart 2.0.0-dev.62.0. Note that due to #28617, we have live users running against both script and application snapshots. Small Plain SCSSRunning on a file containing 4 instances of
This is effectively just testing startup speed. It looks like Dart 2 actually improves startup speed with a script snapshot, but app snapshot startup speed is about 50% slower. Large Plain CSSRunning on a file containing 2^17 instances of
This benchmark simulates a large-scale style system. We see about a 15% performance hit for script snapshots, up to about 30% for app snapshots. This is pretty concerning: one of the selling points of Dart Sass is its performance parity with the C++ implementation, but this will break that parity and thereby substantially hurt our value proposition. |
|
Thanks for trying this out @nex3 I will make sure to include Dart SASS in the set of tools I am benchmarking against. |
|
FYI, it looks like the performance of 2.0.0-dev.64.1 is slow enough that it's causing mass timeouts on Travis of tests that invoke Dart subprocesses. |
|
@nex3 are you sure it is performance related issue and not a legit timeout (e.g. due to sync-async semantics changes or something like that)? |
|
Pretty sure; the tests pass on my fast desktop computer, but time out on Travis. |
…ecking This forwarders are used at dynamic call-sites and perform type checking for all non-generic-covariant arguments. This allows to skip the same type checks in the actual method body. This yield on average 10% improvement in performance across the body of benchmarks including dart2js compilation times. Bug: #33257 Change-Id: If3fc94a2e0a6f496ec0633f0b379d053a54a40ca Reviewed-on: https://dart-review.googlesource.com/61244 Commit-Queue: Vyacheslav Egorov <[email protected]> Reviewed-by: Alexander Markov <[email protected]> Reviewed-by: Martin Kustermann <[email protected]>
This substantially speeds up the test time, and once we re-enable Dart 2 support is should work around dart-lang/sdk#33257.
This substantially speeds up the test time, and once we re-enable Dart 2 support is should work around dart-lang/sdk#33257.
This CL introduces, similar to RawClass::direct_subclasses_, a RawClass::direct_implementors field which is kept up-to-date. The generation of type testing stubs uses this cached reverse-relationship information for faster cid-range calculation (time spent in type testing stub generation is decreased by an order of magnitude). This CL also enable type testing stubs for any instantiated type. This seems to improve * analysis-server-cold-analysis by 6% * analysis-server-warm-analysis by 11% as well as other non-analysis benchmarks in JIT mode. Issue #33257 Change-Id: If01ee06ac08251b2ed27f79d6b82ad7354c8336e Reviewed-on: https://dart-review.googlesource.com/66576 Commit-Queue: Martin Kustermann <[email protected]> Reviewed-by: Alexander Markov <[email protected]> Reviewed-by: Vyacheslav Egorov <[email protected]>
|
@mraleph Any updates on this? |
|
The progress can be tracked on #31798: we are landing changes, there is currently one CL in flight. With all changes combined I measure this on compiling large Dart application with dart2js:
I have not measured other tools yet. |
kevmoo
added
P2
|
@mraleph any updates here? Is anything else happening on this for 2.1? |
|
We had to disable one of the key optimizations we did. @mkustermann is working on reenabling it. We are going to return to optimizations after it is re-enabled. |
|
Move to 2.2 because the reland did not make it in time. |
|
FYI, between this and the dropped support for |
|
After waiting for more than a week for our buildbot to stabilize (and we still have flaky crashes left), we decided to land the bugfix now (06f9a9e), which also re-enables the optimization. |
|
Can this be closed, @mraleph and @mkustermann ? |
1 similar comment
|
Can this be closed, @mraleph and @mkustermann ? |
|
As far as I know we are still slower than Dart 1 sometimes, but we are currently not actively working on anything to reduce the overhead further. Need to gather the numbers again. @sstrickl could you take a stub at measuring where we stand currently with default mode versus |
|
@sstrickl can we can an update on where we currently stand? |
|
Just coming back to this after finishing up the RegExp work I was doing, so I'll try and have some initial numbers in here later this week. My initial benchmarking plans went awry when dart2js crashed while compiling the large code bases I was going to test it on, so I need to pick some alternative codebases for benchmarking. (That, and actually file an issue re: the crashes--I did let a member of the team know about them, but I should still create the issue as well for tracking.) |
|
All the numbers below are using commit 0b40a46, since that's the last commit that could compile the codebases I've been using until an appropriate fix for issue #36516 is in place. However, that's only ~a month old (Mar 4), so the numbers shouldn't be too different on master. Below, if Checked is Yes, that means that dynamic checks were generated (so the experimental flag was not used). Here's the numbers from running the first handful of N's Sass benchmarks 10 times each, min and max included with the mean to give you an idea of the result range:
Here, the overhead of the strong type checks is ~5% on average, with the ranges on the checked vs. unchecked runs overlapping in almost all cases. For the two large codebases I tried compiling with dart2js (first compiling dart2js to a kernel snapshot and using that for the benchmark times, each codebase compiled five times each):
Here, there's a very minor overhead (~1%) to the runtime strong type checks. So while the dynamic checks do add a small bit of overhead, it seems much more reasonable than in prior testing. |
|
Just to double-check, I've profiled the codebase 1 run both with and without checks, and made sure that the without checks run really was without checks. In the profile, the type testing stubs were around 1% of the profile overhead for the checked version, which matches approximately what I saw in the averages. Thus, on the kinds of loads we were seeing here, the cost of dynamic type checks are no longer providing a significant percentage of overhead, and so I'm going to close this. |
|
@sstrickl How often are these kinds of performance tests done? Before each release or only when certain components are changed? |
|
@Stargator we have a suite of performance tests which is automatically run for each commit to SDK repo in various configurations (native JIT/AOT, dart2js on V8 etc) on different hardware (different Intel variants, ARM32 and ARM64 hardware, mobile phones). Due to technical complexities involved we don't track performance of some more real world scenarios there. However there is an expectation that performance on real world benchmarks is a function of performance on the benchmarks we are tracking. In general I would recommend anybody using Dart to also have at least some performance tracking on their side - so that you can catch potential regressions between releases and report them back to us for fixing - if we miss something. |
Currently known regressions:
Most regressions seem to boil down to the cost of strong mode type checks: with parameter checks and implicit down-casts ignored hit on dart2js goes down to 10%.
The text was updated successfully, but these errors were encountered: