ConductorServiceTimeoutException

[neverfox]

We have a 24/7 job that listens to Datomic logs for new segments. It sometimes goes for a while without any work. Twice now, after long periods of idle time (on the order of days), the following has happened to one of our peer groups when work finally resumed, i.e. constantly spitting out in the logs:

16-11-21 18:17:31 onyx-datomic-902846474-4bh91 WARN [onyx.messaging.aeron:95] -
                                  java.lang.Thread.run           Thread.java: 745
    uk.co.real_logic.agrona.concurrent.AgentRunner.run      AgentRunner.java: 105
         uk.co.real_logic.aeron.ClientConductor.doWork  ClientConductor.java: 113
         uk.co.real_logic.aeron.ClientConductor.doWork  ClientConductor.java: 293
uk.co.real_logic.aeron.ClientConductor.onCheckTimeouts  ClientConductor.java: 338
uk.co.real_logic.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns

[lbradstreet]

@neverfox this should be fixed by 0.9.14. Please let me know if you see anything similar when you're on 0.9.14.

[lbradstreet]

Closing until we hear of this re-occuring.

[neverfox]

This is still happening with 0.9.15 and from what I can tell it's when an always-running job receives new work to do after a long hiatus (days/weeks) of doing nothing.

[lbradstreet]

Thanks for the report. I assume it doesn't come back up after spitting out the errors? It should have re-established the subscriber. Could you please paste the errors from the lastest logs?

…

On 3 Jan 2017, at 11:29, Roman Pearah ***@***.***> wrote: This is still happening with 0.9.15 and from what I can tell it's when an always-running job receives new work to do after a long hiatus (days/weeks) of doing nothing. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[mariusz-jachimowicz-83]

After reading real-logic/aeron#256 it seems that Aeron uses this timeout to set publication as dead after 5s of inactivity. Solution might be increase this timeout - https://github.com/real-logic/Aeron/wiki/Configuration-Options see Publication Connection Timeout.

[thenonameguy]

This occurs to our code as well. Our job runs fine for 8-15 minutes then all of the tasks fail with this:

io.aeron.exceptions.ConductorServiceTimeoutException: Exceeded (ns): 5000000000
                          clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle :lifecycle/write-batch. Killing the job. -> Exception type: io.aeron.exceptions.ConductorServiceTimeoutException. Exception message: Exceeded (ns): 5000000000