Possible VM Memory Leak #24129
Comments
iposva-google
added
P1
|
Probably unrelated but I fixed a leak in ThreadRegistry yesterday (a Monitor was leaked for every isolate). Fix is here: https://github.com/dart-lang/sdk/blob/master/runtime/vm/thread_registry.cc#L30 |
|
@kaendfinger can you add the OS and the Dart VM version you were using? Thanks! |
|
Thanks @kaendfinger . Any chance you can try 1.12-dev ? |
|
@sethladd Sure, I should be able to try it in a few hours. I'll report the result ASAP. |
|
@sethladd I don't believe I am seeing the leak in the 1.12-dev. I'm getting a very stable memory use from the system stats. While I was investigating, I wanted to know what the relationship between what Observatory reports for memory usage and what the system reports for memory usage is. Should I expect these numbers to be close together? |
|
I was able to reproduce using 1.12.0-dev.5.5. I will continue investigating. |
|
@kaendfinger could you explain a little about what this program is doing? Is it just an http server responding to requests from the responder and requester, or is there also something else going on that could give us a hint? Thanks! |
|
@zanderso After running it for a bit longer, it started producing the leak again. Broker is an HTTP server that allows WebSocket connections and allows each WebSocket client to communicate with each other using our protocol. |
|
@kodandersson Please investigate the proper deallocation of the InterruptableThreadState objects. |
|
Related to this- an isolate's ThreadRegistry remembers every thread that an isolate ever runs on. This list is never pruned, so even after a thread exits- the ThreadRegistry is holding onto a ThreadRegistry::Entry for each thread that entered the isolate. We could get into a bad state where an isolate is run on more and more threads and memory usage will grow by a (small) amount for each thread. |
The Observatory reports the Dart heap capacity in use, separately for the new and old generations. There are C++ based allocations in the VM within your process which are not being exposed in the Observatory, but are included in the system reports. The difference between the two numbers comes from VM internal data structures, networking buffers, thread stacks, ... |
|
@iposva-google Thanks for the explanation. |
|
Tagging for 1.12 consideration |
|
This fixes leaks related to StoreBuffer: |
…locks. Add StoreBuffer::ShutDown to delete global cache of free blocks on VM shutdown. BUG=#24129 [email protected] Review URL: https://codereview.chromium.org//1309603003 .
Cherry-picks and associated conflict resolution for: 664742f d0b03ee 3aa62d0 74e0278 028c995 ===== Keep StoreBuffer::global_mutex_ and global_empty_ alive. At Dart::CleanUp time, there may still be threads around that access their store buffer, so we need to keep these around until @zanderso lands the clean shutdown CL. We should stop leaking them once we cleanly stop all threads at VM shutdown (see issue 23844). BUG=#24169 [email protected] Review URL: https://codereview.chromium.org//1305123003 . Conflicts: runtime/vm/dart.cc Fix reuse of free store buffer blocks from the global cache of free blocks. Add StoreBuffer::ShutDown to delete global cache of free blocks on VM shutdown. BUG=#24129 [email protected] Review URL: https://codereview.chromium.org//1309603003 . Conflicts: runtime/vm/dart.cc - Prevent interrupting the NULL thread when the isolate is exiting from within a thread. BUG= Review URL: https://codereview.chromium.org//1307833002 . - Initialize fields used in the profiler interrupts. BUG= [email protected] Review URL: https://codereview.chromium.org//1304043003 . Completely remove InterruptableThreadState - InterruptableThreadState is gone. - Moved InterruptableThreadState fields directly into Thread. - Iterate over all threads in an isolate when profiling. - Still only sample the mutator thread. Fix ThreadRegistry leak - When deleting a Thread, iterate over all isolates and remove it from the isolate's thread registry. [email protected] Review URL: https://codereview.chromium.org//1293253005 . Conflicts: runtime/vm/isolate.h runtime/vm/thread.cc runtime/vm/thread.h runtime/vm/thread_interrupter.cc runtime/vm/thread_interrupter_android.cc runtime/vm/thread_interrupter_linux.cc runtime/vm/thread_interrupter_macos.cc - Fix leak in TimelineEventRecorder. - Fix leak in ThreadRegistry. patch from issue 1305353003 at patchset 1 (http://crrev.com/1305353003#ps1)
|
I believe that this issue is fixed, and the fixes were merged into 1.12.0-dev.5.9. Please verify and close this issue. |
|
@whesse Ok I will check them out ASAP. |
|
The memory leak is verified as fixed. Thank you very much. There is another issue with same test perhaps unrelated; however we're seeing very high CPU usage and observatory is showing 0% CPU utilization. |
|
@johnmccutchan I believe this would be in your area. To elaborate on the CPU Usage at 0% thing, we are actually getting a Pi Chart that says 100% Idle, even though it's doing constant work. Also, when I open a CPU Profile, it's empty. |
|
What program can I use to reproduce this? |
|
@johnmccutchan This is profiling broker.dart |
|
@kaendfinger I need a local reproducible so I can diagnose the problem |
|
Can we start a different bug for the CPU usage issue - this 1.12 milestone issue about the VM memory leak should be closed. Post a link to the new issue to this bug, when it is created. |
|
@kaendfinger The CPU profile issue is likely a known bug with programs that mostly process events quickly. We schedule a profile tick and the isolate finishes processing messages before it is sampled. A local reproduction would be helpful to use while I figure out the correct heurestic for this type of program. |
|
@johnmccutchan I will open a new issue for this in just a little bit. |
We have found a possible memory leak in the Dart VM. I have created a ZIP file with the stuff needed to reproduce this issue.
ZIP file: http://files.directcode.io/dart-leak-broker.zip
Steps:
dart --observe broker.dartdart responder.dartdart requester.dartYou will need to run these commands in 3 terminal sessions at the same time.
If you look at the Observatory on broker.dart, you will see the memory use is large. However, when looking at the memory usage of the process on the Operating System side, it's much higher.
In one of our tests, our heap was 32mb. However, the process was actually using 700mb of memory.
Source: https://github.com/IOT-DSA/sdk-dslink-dart
The text was updated successfully, but these errors were encountered: