Change behaviour when device disconnects

[coretl]

• Add top level STATE PV that can be
  • DISCONNECTED
  • CONNECTING
  • CONNECTED
• At startup set to CONNECTING, then CONNECTED if it works
• If it fails to connect then immediately fail with a traceback and let Kubernetes restart the process
• While CONNECTED if the TCP connection drops then set to DISCONNECTED and halt the polling process
• Add a liveness probe on the STATE PV
• Any other error should log to the console, but continue the process

[AlexanderWells-diamond]

If it fails to connect then immediately fail with a traceback and let Kubernetes restart the process

I suspect I'm missing key information here, but that sort of thing sounds like it could end up in an infinitely failing loop, if the connection is impossible for some external reason?

[coretl]

Kubernetes has a backoff mechanism, so it will wait longer and longer between restarts. This is more visible than anything we could invent ourselves...