Robot freezes when running steps longer than 30s #2822
Comments
|
Here is the error from the developer console: During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): (anonymous) @ logger.js:43 During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): (anonymous) @ logger.js:43 |
|
Hi, I also wrote a python program directly to do this protocol and I am getting the same error During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Seems no programs can work at all. What is going on? |
utimcraig
changed the title
|
@utimcraig thanks for this report! This is a bug with the robot's safety features to catch bugs in its motor controller board. It uses a fixed 30-second timeout. If a command takes more than 30 seconds to complete - so, for a move command, more than 30 seconds to start and stop moving - you get the error you see. In your protocol, on line 28 you set the flow rate of the pipette to 3 uL/s: p300.set_flow_rate(aspirate=3, dispense=3)Later, on line 42, you aspirate 160 uL: p300.aspirate(160, lysate_plate.cols('1'))Because of the flow rate restriction, this takes (160/3) 53.33 seconds, longer than the 30 second timeout, and so the error is raised.; To be clear, this is not your mistake - we intend to fix this, and will use this issue to track the work to do so. However, as a workaround until then, you have two options:
On our part, we will add dynamic timeouts for serial commands based on a projection of how long the command will take. Again, thank you for this report! |
|
Thanks @sfoster1 , it would be nice to not have to worry about a step being under a time limit. Especially mixing steps. This kind of aspirate-dispense with low flow rate is critical for phytip purifications: I think a lot of people may be interested in this application. This time limit especially hurts in .mix() operations |
|
@sfoster1 It appears this time limit is not only for single steps, the aspirate+dispense time needs to be less than 30s. So the example you gave is incorrect, each aspirate/dispense needs to not be 89 uL, but rather (89/2) = 44.5 uL. I think the best thing is to dynamically calculate a timeout, though this gets complicated because you also have to include head moving times. Perhaps a heuristic upper limit could be set for the steps taking into account the slowest flow rates, largest volumes, largest moving distances, and head movement speeds. For mix() commands I'm not sure how you could set it other than dynamically. That really is the best option. This also is good advertising for my feature request posted before I knew about the time limit: Feature request: Time to next step #2821 |
utimcraig
changed the title
|
I have a bit of a workaround for this issue. If you use a pipette.delay(seconds=0) in-between aspirate steps it seems to re-start the 30s timer. So in order to aspirate 100ul at 1ul/sec you can do something like this- pipette.aspirate() The best solution is still to dynamically calculate the limit on the time of a step though |
|
Hope everyone had a good holiday. Is anyone working on this bug? Thanks |
|
Any updates on the dynamic time calculation? |
|
HI @utimcraig! We're busy working on some core problems in the code base that are preventing us from tackling this problem. You'll see updates in this thread when that work is complete and we have engineering resources available to tackle this one. Thank you for your patience! |
Currently, after every command we send an M400, which delays until the smoothie's queue is empty. We couple this with retrying unacked commands. The problem is, this couples command acknowledgement with command execution. The timeouts need to be short enough to be responsive if a command isn't acknowledged, but short timeouts cause problems with commands that take a long time - like long motions at low speeds. Instead, write the command and retry it if not acknowledged, but do not do an M400 until the command is acknowledged. Once that's done, you can use a very long timeout for the M400 without worrying about having to retry. Closes #3300, closes #2822
overview
When I am using a protocol from protocol designer, the robot freezes after the first aspirating/dispense step. The steps after the first one use the "New pipette tip "never"" option, I'm not sure if that is related to this issue. This protocol is from the PD ignores aspirate/dispense speeds However it also occurs when using a hand-written .py protocol (attached at bottom of chain)
current behavior
Robot stops working, run timer also stops.
The api.log mentions a serial response error. I also have run this protocol using only USB, USB+Wifi, and Wifi connections and it happens on every type of connection between the computer and the robot. I'm guessing maybe this serial connection doesn't really have anything to due with the wifi vs usb connection but you guys can help hopefully.
Additionally, after getting this error I have to power cycle the robot to get it to communicate with the computer again.
steps to reproduce
I tried running either the test.json file (renamed here so I can upload it to github). The api and serial logs are uploaded here:
api.log
serial.log
Test-json-renamed to txt for github upload.txt
expected behavior
The robot should perform the protocol from PD without freezing.
The text was updated successfully, but these errors were encountered: