Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: Input/output errors are repeatedly logged after removing a Temperature Module #5359

Closed
SyntaxColoring opened this issue Apr 2, 2020 · 1 comment · Fixed by #7690
Closed

bug: Input/output errors are repeatedly logged after removing a Temperature Module #5359

SyntaxColoring opened this issue Apr 2, 2020 · 1 comment · Fixed by #7690
Assignees
@sanni-t
Labels
api Affects the `api` project bug robot-svcs Falls under the purview of the Robot Services squad (formerly CPX, Core Platform Experience). software-investigate Our Software team needs to look into this so we can understand more about it.
Milestone
CPX Sprint 31

Comments

@SyntaxColoring
Copy link
Contributor

SyntaxColoring commented Apr 2, 2020 •
edited by sync-by-unito bot
Loading

Overview

After a Temperature Module is removed from an OT-2, the API server will still try to poll it indefinitely, throwing an error on each attempt. The error does not go away after reconnecting the module.

I have no idea if there's a user-facing effect to this, but it's probably not good? It makes logs noisy, at the very least.

Tested on v3.16.1.

Steps to reproduce

  1. SSH in to an OT-2.
  2. journalctl --boot --no-pager --follow --output=short-monotonic
  3. Connect and turn on a Temperature Module. You should see opentrons-api-serial polling for the current temperature once per second.
  4. Upload and run this protocol: 
    metadata = {"apiLevel": "2.2"}
def run(protocol):
    module = protocol.load_module("Temperature Module", 7)
    module.set_temperature(20)
  5. Let the protocol complete.
  6. Turn off the Temperature Module.
  7. After a few seconds, you should start to see errors like this: 
    [  283.981680] <serial number> kernel: usb 1-1.3: USB disconnect, device number 7
[  283.984597] <serial number> kernel: cdc_acm 1-1.3:1.0: failed to set dtr/rts
[  283.995124] <serial number> opentrons-api[339]: Module Removed: ModuleAtPort(port='/dev/ot_module_tempdeck1', name='tempdeck')
[  283.997421] <serial number> opentrons-api[339]: Module tempdeck detached from port /dev/ot_module_tempdeck1
[  292.636349] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.636349] <serial number> python[339]: Traceback (most recent call last):
[  292.636349] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.636349] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.636349] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.636349] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.636349] <serial number> python[339]: Traceback (most recent call last):
[  292.636349] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.636349] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.636349] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.636349] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.636349] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.643388] <serial number> python[339]: Traceback (most recent call last):
[  292.643388] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.643388] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.643388] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.643388] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.643388] <serial number> python[339]: Traceback (most recent call last):
[  292.643388] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.643388] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.643388] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.643388] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.643388] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.643388] <serial number> python[339]: Traceback (most recent call last):
[  292.648828] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.648828] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.648828] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.648828] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.648828] <serial number> python[339]: Traceback (most recent call last):
[  292.648828] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.648828] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.648828] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.648828] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.648828] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.648828] <serial number> python[339]: Traceback (most recent call last):
[  292.654254] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.654254] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.654254] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.654254] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.654254] <serial number> python[339]: Traceback (most recent call last):
[  292.654254] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.654254] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.654254] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.654254] <serial number> python[339]: termios.error: (5, 'Input/output error')
[  292.654254] <serial number> python[339]: Exception in thread Tempdeck recursive update temperature:
[  292.654254] <serial number> python[339]: Traceback (most recent call last):
[  292.659401] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[  292.659401] <serial number> python[339]:   File "/buildroot/output/target/usr/lib/python3.7/threading.py", line 865, in run
[  292.659401] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 285, in _recursive_update_temperature
[  292.659401] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 250, in _send_command
[  292.659401] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/temp_deck/driver.py", line 269, in _recursive_write_and_return
[  292.659401] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line111, in write_and_return
[  292.659401] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/opentrons/drivers/serial_communication.py", line53, in clear_buffer
[  292.659401] <serial number> python[339]:   File "usr/lib/python3.7/site-packages/serial/serialposix.py", line 595, in reset_input_buffer
[  292.659401] <serial number> python[339]: termios.error: (5, 'Input/output error')
    The errors repeat every 10 seconds or so.
  8. Turn the Temperature Module back on. The errors will not go away, but successful temperature polls will be intermixed with them.

┆Issue is synchronized with this Wrike Task by Unito
@SyntaxColoring SyntaxColoring added bug api Affects the `api` project hmg hardware, motion, and geometry labels Apr 2, 2020
@SyntaxColoring
Copy link
Contributor Author

SyntaxColoring commented Apr 7, 2020 •
edited
Loading

We think that this should be resolved in v3.17. We should close this ticket when that's confirmed. Edit: It was not!

@SyntaxColoring SyntaxColoring added the software-investigate Our Software team needs to look into this so we can understand more about it. label May 2, 2020
@sanni-t sanni-t added robot-svcs Falls under the purview of the Robot Services squad (formerly CPX, Core Platform Experience). and removed hmg hardware, motion, and geometry labels Apr 20, 2021
@sanni-t sanni-t self-assigned this Apr 20, 2021
@mattwelch mattwelch added this to the CPX Sprint 31 milestone Apr 20, 2021
sanni-t added a commit that referenced this issue Apr 20, 2021
@sanni-t
fix(api): do not cache removed modules in thread_manager
3fe1d90 
Closes #5359
@sanni-t sanni-t mentioned this issue Apr 20, 2021
Merged
sanni-t added a commit that referenced this issue Apr 21, 2021
@sanni-t
fix(api): do not cache removed modules in thread_manager
4686b0e 
Closes #5359
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sanni-t sanni-t

Labels
api Affects the `api` project bug robot-svcs Falls under the purview of the Robot Services squad (formerly CPX, Core Platform Experience). software-investigate Our Software team needs to look into this so we can understand more about it.
Projects
None yet
Milestone
CPX Sprint 31
Development

Successfully merging a pull request may close this issue.

fix(api): do not cache removed modules in thread_manager Opentrons/opentrons
3 participants
@mattwelch @SyntaxColoring @sanni-t