bug: Controlling Lights Works In Jupyter Notebook But Not In Python Script

[gfleetwood]

Overview

Two weeks ago I filed #6045 which was recently closed via pull request #6091. However, controlling the robot's light worked running a script in a Jupyter notebook, but gave this error - mentioned in the pull request - when I ran the same script on the command line:

Failed to initialize character device, will not be able to control gpios
(lights, button, smoothie kill, smoothie reset). If you need to control gpios, 
first stop the robot server with systemctl stop opentrons-robot-server. 
Until you restart the server with systemctl start opentrons-robot-server, 
you will be unable to control the robot using the Opentrons app.

I tried a number of permutations of os.system("systemctl stop opentrons-robot-server") and os.system("systemctl start opentrons-robot-server") and nothing worked. The latter snippet was not needed in the Jupyter notebook - only the former. The consistent failing was the unresponsive of the lights when running the script from the command line. However, sometimes the rest of the script would also fail to execute, seemingly dependent on the placement of os.system("systemctl start opentrons-robot-server"). This placement was all over the script but always below os.system("systemctl stop opentrons-robot-server").

Steps to reproduce

I'm on v3.19.0 and opentrons==3.19.0. This is a minimal reproducible example:

import os
os.system("systemctl stop opentrons-robot-server")
#os.system("systemctl start opentrons-robot-server")

import opentrons.execute
from opentrons import protocol_api
from opentrons import types

def function(protocol):
    
    protocol.home()

    for plate in plates:
        
        loc = types.Location(plate.calibrated_offset, plate)     
        pipette.move_to(location = loc)      

    protocol.home()

slots = [5, 6] 
protocol = opentrons.execute.get_protocol_api('2.5')
tiprack = protocol.load_labware('opentrons_96_tiprack_300ul', '11')
pipette = protocol.load_instrument("p300_single", mount = 'right', tip_racks = None)  
plates = [
    protocol.load_labware('corning_24_wellplate_3.4ml_flat', i)
    for i in slots 
]

protocol.set_rail_lights(on = True)
function(protocol) 
protocol.set_rail_lights(on = False)

Current behavior

A script controlling the lights affects the lights in the Jupyter notebook but doesn't affect the lights when run as a Python script.

EDIT: I now have the additional problem of scripts hanging instead of ending when they're done running.

Expected behavior

Consistent behavior when interacting with the lights in the Jupyter notebook and as a script.

[theosanderson]

Is it possible you still had Jupyter open when you ran the script? Key point is that only one thing (robot server / Jupyter notebook/ script) can be connected to GPIOs at a time so once something connects you have to disconnect it before you do something else. If you get the Failed to initialize character device, message something else is connected to the GPIOs.

[gfleetwood]

That was probably it as I don't usually shutdown my prototyping notebook. Will see if that works. Thanks!

[gfleetwood]

@theosanderson It worked! I suspect the problem was twofold, i.e having a Jupyter notebook open but also trying to run two scripts at once. The latter stems from me wanting to run two separate experiments on a schedule. Is there a recommended way of doing that for experienced Opentrons users? If I had access to systemd I would just run the script on a schedule according to different config files. I tried to install systemd from source but gave up out of frustration. Right now I'm using a threading hack I bumped into:

import os
os.system("systemctl stop opentrons-robot-server")
#os.system("systemctl start opentrons-robot-server")
import threading

import opentrons.execute
from opentrons import protocol_api
from opentrons import types 
    
def func2(protocol):
   pass

def func1(protocol):

    # Run every 2 hours
    time_interval = 7200
    threading.Timer(time_interval, func1, args = (protocol,)).start()
    
    func2(protocol)

protocol.home() 
func2(protocol)
protocol.home()

Which I tried to move away from using ssh, i.e having the scheduler on a normal raspberry pi, and then sshing into the robot when its time to run the script. That didn't work because for some reason the usb connection isn't working and then I tried to turn off the wifi thinking that it would help. Bad idea. Now I can't use the robot at all. I was hoping you could comment on the issue I submitted: #6238.

Also, can you recommend some good documentation on the purpose of the robot server, and how to gracefully connect/disconnect to/from the GPIO pins? Is this a uniquely opentrons things or is it coming from the raspberry pi itself?

[theosanderson]

@theosanderson It worked! I suspect the problem was twofold, i.e having a Jupyter notebook open but also trying to run two scripts at once. The latter stems from me wanting to run two separate experiments on a schedule. Is there a recommended way of doing that for experienced Opentrons users? If I had access to systemd I would just run the script on a schedule according to different config files. I tried to install systemd from source but gave up out of frustration. Right now I'm using a threading hack I bumped into:

You could have a launcher script that launches the python protocols at the right schedule with subprocess (and ensures two are never launched at once). Then each Python protocol can do its own thing without interfering with others.

Which I tried to move away from using ssh, i.e having the scheduler on a normal raspberry pi, and then sshing into the robot when its time to run the script. That didn't work because for some reason the usb connection isn't working and then I tried to turn off the wifi thinking that it would help. Bad idea. Now I can't use the robot at all. I was hoping you could comment on the issue I submitted: #6238.

That's odd - if you take the case off the Pi you will be able to see the ethernet port. You could potentially connect that directly to a local router and view what devices are connected to it to find the IP so you can get in and turn WiFi back on etc and recover a bit. Similarly you can connect a USB-to-ethernet connection to the Pi's USB ports to do the same sort of thing without having to mess with the case.

Also, can you recommend some good documentation on the purpose of the robot server, and how to gracefully connect/disconnect to/from the GPIO pins? Is this a uniquely opentrons things or is it coming from the raspberry pi itself?

The robot server is the thing that the app interacts with to do:

• calibration
• running protocols
• turning lights on and off in the app

that sort of thing. Once you connect to the GPIO pins you essentially disconnect from them by killing the process. There is probably some way you can do this more gracefully by killing the hardware controller but I haven't played with it. It's coming from libgpiod itself.

[gfleetwood]

@theosanderson Got everything working. Thanks!