Possible memory leak since 9.0 #1936
Comments
github-actions
bot
added
the
community
|
Hi @fcheung 👋 Thank you for sharing this issue! You mentioned 9.0.0 enabling thread instrumentation by default and we're wondering if this is related to what you're seeing. To help narrow it down, would you be willing to try disabling thread instrumentation while using Ruby agent 9.0.0? To do this, add the following to your # newrelic.yml
instrumentation.thread.tracing: falseAlternatively, you can disable thread instrumentation with the following environment variable: NEW_RELIC_INSTRUMENTATION_THREAD_TRACING=falseWe're also wondering if you have long-running/indefinite transactions? Thank you for mentioning that you're working on a reproduction. That should help us get closer to a resolution! |
|
@fcheung I have experienced the same. |
|
Hi, I've set NEW_RELIC_INSTRUMENTATION_THREAD_TRACING=true on our app still running new relic 8.16 to see whether thread tracing is the culprit - leaving that to settle for a while. In the mean time I've got this demo app that seems to reproduce the issue: https://github.com/fcheung/leak_app I've been running this demo app locally, in production mode with a valid license key and the data is showing up in newrelic The app has one page at if you run the app with the TRACE_ALLOCATIONS environment variable set, then it will record object allocations and visiting checking the earlier dumps show the counts associated with I've also run another copy of the app with NEW_RELIC_INSTRUMENTATION_THREAD_TRACING=false. In this copy there is nothing from newrelic in the heapy dump of top allocated/object count/ref count and the heap diff over 10 minutes or so doesn't seem to have much that isn't just normal churn. |
|
I can confirm that I get the same behaviour with 8.16.0 and NEW_RELIC_INSTRUMENTATION_THREAD_TRACING=true |
|
I can confirm that we had the same problem with newrelic 9.0.0, ruby 3.2.1, rails 7.0.4.3, puma 6.1.1. We've disabled newrelic to solve the problem. |
|
Thanks for confirming @fcheung! @bendilley - it looks like thread tracing is contributing to this issue. While we work on this issue, you can disable thread tracing and continue to use the agent by adding the following to your # newrelic.yml
instrumentation.thread.tracing: falseAlternatively, you can disable thread instrumentation with the following environment variable: NEW_RELIC_INSTRUMENTATION_THREAD_TRACING=false |
|
Hey guys, for what it's worth, after I re-enabled thread tracing with 9.2.0 being released, while my processes don't hang anymore (yay! issue #1910 is fixed) I am now seeing this as well. |
|
I've prepared a tag that appears to fix the issue for me when using the reproduction @fcheung provided (thank you so much for providing that! It's a huge help and made debugging the issue much easier). Here are some screenshots of the memory usage: This one is multiple instances of the repro app, each using a different agent tag. The tags were 9.2.0, 9.2.0 with thread tracing disabled, and a couple of debugging/bugfix attempts. After running for about 24 hours, you can see that the original 9.2.0 memory usage is continuously increasing, however the debug/bugfix attempt tags all show flat memory usage, same as with thread tracing being disabled! This screenshot is 9.2.0, 9.2.0 with thread tracing disabled, and my final bugfix (same as in previous tags but without the extra debugging logs/code). It's for a shorter period of time, but you can see that 9.2.0 is beginning to increase but the other 2 are remaining flat. This is the tag I have ready for others to try out. It would be great to see if this result is seen by others as well, so you can try this change out using this in your gemfile. We'd appreciate any feedback on this fix and definitely want to know if anyone is still seeing increasing memory usage with this tag. |
|
I've deployed this @tannalynn - will keep you posted with what we see |
|
Hi @fcheung, @bendilley, @travisbell, @cirdes! 👋 A fix for this issue is available in version 9.2.2 of |
|
@kaylareopelle @tannalynn FYI the fix has been looking good overhere for the last few days. |
|
That's great news! We appreciate you circling back to us. |
Description
After updating from 8.16 to 9.0 we noticed our ruby processes had steady memory growth. This occurs only in one of our applications, despite all of them having similar environments in terms of ruby version, rails version etc.
This screenshot show ruby process usage overtime on one of our dev servers over a weekend (ie no traffic other than loadbalancer health checks, maybe actioncable pings if anyone had left a window open)
after we rolled back to 8.16 this issue went away.
A thing that is different about this application that springs to mind is that is does use actioncable, ie. has extra threads running in the background and 9.0 defaulted thread instrumentation to on, so maybe that is relevant?
Steps to Reproduce
working on it ...
Your Environment
Additional context
I did find one other person mentioning this: #1842 (comment)
For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):
The text was updated successfully, but these errors were encountered: