EMU: MercuryiTC communication failures #6286
Comments
|
The device supports RS232 flow control - could we use that? Or is our baud rate too high for specifics of the connection? |
|
For help with prioritising from the IS: "We have to say that this needs to be fixed before the Cycle because we recently found that it’s much more than just annoying i.e. it caused a failure of switching on the booster heater. The reason for this is that the booster heater script refers to the Temp_Sample block for determining whether it should switch the heater on, and if the target temperature is higher than Temp_Sample, it will start using the booster heater by csetting the same block. If the block is temporarily in ‘invalid’, then it certainly fails to read and set the temperature (and just carries on). The booster heater is a crucial kit for carrying out the user programme in the next Cycle on EMU and MUSR. We have already seen this happened twice during this short test period (about a week). So, this problem has a potential to cause lost times. I can describe more details if this needs a quicker response" |
|
OK. Well, perhaps we should bring this into the sprint (possibly at the expense of something else). Some questions:
|
|
|
The decision has been made to do this next sprint |
|
EMU and likely MuSR may not be able to run this cycle until we fix this so muons are quite concerned. |
|
Do we know yet if a Mercury iTC on MuSR shows the same issues? |
|
In the meeting today they said they had not noticed anything, but they are concerned that if it hits EMU it might trouble MuSR too. I am sure we will hear if they do find the same issue |
|
What is the situation regarding setup and testing on EMU, can they put something online for us? |
JamesKingWork
added
ready
and removed
bucket
|
IS: "The cryostat was on the instrument and showing the “invalid” error continuously until a few days ago. However, the cryostat is now gone and it’s a bit tricky how we can reproduce the state. The Mercury itc is still connected to the instrument, but the errors are much less frequent. My feeling is that these errors appear when there is a change in the blocks. With all sockets disconnected, the values do not change much. We are looking for a way that we can connect the controller to something – can be an empty cryostat or just some probes, so that it reads some meaningful number. This will happen probably sometime in the next week. Will keep you posted." |
|
We have just tested the RIKEN mercury plugged into MuSR as that also had problems. In the mercury logs there are various issues: We managed to get this to happen with these steps in the genie python console: r.settemp(1)
r.settemp(2)
r.settemp(1)
Block cycle is in alarm: INVALIDThe logs: |
|
There are two sources of It looks like the first is due to a lock timeout, which is only recorded in steam debug mode The lock timer starts as soon as the record wishes to be processed and access the device, if it times out the operation is abandoned and the PV is put into timeout alarm. This is the cause of a majority of the The second issue, that does leave messages in the log file, may just be a response taking a bit too long. I've increased reply timeout from 1 to 2 seconds, but too early to say if that has resolved the issue. However these messages occurred far less frequently that the ones caused by the lock timeout. It seems replies get out of sync in this case, I wondered if we might be able to use a I think stream device should log lock timeouts to file so will submit a PR for that. |
|
Tests on EMU look to be going well, also now deployed to MUSR |
|
Confirmed on MUSR that with the increase in locktimeout we no longer see invalid alarms appearing. The soak test on MUSR is still ongoing so will confirm soon whether we see the less frequent logged mismatch errors. |
As an EMU scientist, I would like to be able to talk reliably to a MercuryiTC. The scientists initially raised concerns about intermittent invalid alarms and it appears this is due to communication failures to the device.
Acceptance criteria:
Notes:
The text was updated successfully, but these errors were encountered: