Managing long-running processes in Dandihub

[magland]

I'm looking for some guidance on managing long-running processes in dandihub. Could anyone share what happens if I close the browser tab or switch off my computer while a process is running? Also, does the instance automatically shut down after a certain period? Ideally, I'd like to start a job and close the tab while letting the process run. Any insights on this would be greatly appreciated!

[CodyCBakerPhD]

I've had a lot of experience with this one. Longest I've ever run a single instance is close to 2 weeks, though I was doing more ETL for upload purposes than download/processing like I'm sure you're interested in

JupyterHub instances in general have gotten better about this over time, BUT; yes, exiting the tab in any way essentially tells the server to close down after X minutes (feels about X is about ~15; it's not immediate, probably because that would be annoying if it was a random disconnection). I just leave at least one device (usually an older, less used one) running with that window open. Note that you can then reconnect to the same server and window from a different device to monitor progress or run other operations; it seems to only care that at least one window is open somewhere

It's not just closing the tab though; a similar timeout can occur randomly when keeping it open for a very long period of time. tmux is your friend here, for whatever reason... controlling your terminals/screens with this particular tool keeps the timeout from cancelling your spawn

But even that isn't perfect and so for whatever operation you're running, I strongly recommend building a safe 'resumeable' iteration architecture where if there is a crash, you can pick it right back up where it was (or close to it); assuming you're asking for Dendro reasons, such an architecture would also be to your great benefit for utilizing spot pricing discounts in general, which can have the same issue

Just a friendly reminder that the DANDI Hub platform is community-driven and following best practices (such as only leaving an instance open for as long as is actually required to complete the task; cleaning up data the moment you no longer need it, etc.) is what allows it to remain free of charge. So if you plan on using it in this way, it might be better to use a smaller instance rather than the GPU/larger ones

[magland]

This is really helpful, thanks Cody!