-
Notifications
You must be signed in to change notification settings - Fork 178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(api): do not cache removed modules in thread_manager #7690
Conversation
Codecov Report
@@ Coverage Diff @@
## edge #7690 +/- ##
=======================================
Coverage ? 82.67%
=======================================
Files ? 321
Lines ? 21588
Branches ? 0
=======================================
Hits ? 17848
Misses ? 3740
Partials ? 0 Continue to review full report at Codecov.
|
1fc9969
to
a0916f6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Away from my robot right now (and don't have a tempdeck anyway) but this code looks solid
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great, cool use of weakref.
Tested with a tempdeck & a thermocycler. Threads clean up as expected. |
Closes #5359
Overview
Currently, when we unplug an OT module, a series of events ultimately leads to
API._unregister_modules
being called. This function then calls the__delete__
function on theAbstractModule
instances, which is supposed to stop the associated module's pollers and disconnect cleanly. But the python garbage collector does not execute the object's destructor (__delete__
), unless all references to that object are removed.This is where the issue in #5359 comes into picture. The behavior we notice is that if a module is unplugged when the app has polled the
/modules
endpoint, the temperature polling thread doesn't stop. This happens because the module endpoints in robot server talk to the modules via theThreadManager
. The thread manager wraps every instance of the attached modules in aCallBridger
and caches it usinglru_cache
. This cached instance doesn't leave the cache unless it's removed to make space for a new instance; which happens very rarely given the size of the cache. Which means that the ThreadManager keeps holding onto theAbstractModule
instance, which prevents the module from being garbage-collected, resulting in the polling threads staying alive even when the module is unplugged.To fix this behavior, we decided to implement our own caching of wrapped modules such that stale instances are removed timely. Additionally, we will explicitly call a
cleanup
function on all modules to stop any threads spawned by them instead of relying on the garbage collector.Changelog
lru_cache
with aWeakKeyDictionary
that doesn't prevent objects from being garbage collected.cleanup
method to all modules, which gets called inAPI._unregister_modules
.update_temperature
in order to catch exceptions better.Review requests
Risk assessment
Medium. Could affect communication with modules if implemented poorly.