[help] Any argument to delay "'errorValue' int 5 | Timed out"?

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

My pipeline encounters 'errorValue' int 5 | Timed out (error below). Although the pipeline (10K+ targets) is erroring, if I restart it, it proceeds for a while before erroring again.

I would like to avoid restarting the pipeline repeatedly Is there any argument I can sset up to make the condition that causes this error more permissive?

To be clear, I know that the error is related to memory, but despite using "retry" with incremental resources, I still encounter this problem. So, rather than trying to solve the cause further, I would like to find a universal solution to avoid premature failure.

Thanks

── Last error message ───────────────────────────────────────────────────────────────────────────────────────────────
    'errorValue' int 5 | Timed out
 A {mirai} dispatcher process is running at pid 3462289 with status "sleeping". There may be a network connection issue, or the dispatcher is simply busy managing lots of tasks or uploading/downloading large datasets. Maybe try increasing seconds_timeout in your {crew} controller.
 Please also try upgrading R packages {nanonext}, {mirai}, and {crew} to their latest versions on CRAN. (Likewise with {targets} if you are using it.) Upgrading these packages solves many kinds of errors. 
Another possibility is an out-of-memory error. The dispatcher process can run out of memory if it is overwhelmed with data objects too large or too many to comfortably fit inside a single R process. Please read https://wlandau.github.io/crew/articles/logging.html to learn about proactive resource usage logging which can provide useful data in the event of a crash. 
If memory consumption is a problem, then as a workaround, you could make each task save/load files instead of using/returning large R objects in memory. If you are using {targets}, you might consider storage = "worker", retrieval = "worker", and/or cloud storage as documented at https://books.ropensci.org/targets/performance.html and https://books.ropensci.org/targets/cloud-storage.html.

[wlandau]

It looks like the mirai dispatcher is still alive (sleeping) in your case, so maybe this is a transient network error.

When mirai::status() queries the dispatcher, the network request has a fixed non-configurable timeout of 5 seconds: https://github.com/shikokuchuo/mirai/blob/785aac10922a878f1553fba586ec82652fcba230/R/mirai-package.R#L111. Sometimes that request may take longer if the dispatcher is busy, so crew retries every seconds_interval seconds until seconds_timeout is reached. So you could try raising seconds_timeout in the controller to a higher value so crew attempts mirai::status() more times before giving up.

By the way, mirai is currently migrating to a new network protocol that should avoid the need for a dispatcher process at all. I don't know for certain if it will be totally feasible to use with crew, and it will probably take months of implementation to factor in. But if it is feasible, then crew will be more robust, and these errors should go away on their own.