wish: throttle "restartall"

[markstos]

I have several produces managed by forever which work as a pool. The reverse proxy in front of them can gracefully handle some members of the pool being offline.

However, it appears that when 'forever restartall' is used, all my node instances are going offline at once, and the app temporarily becomes unavailable. This behavior is nice in some ways in that it's fast and keeps all the nodes together in a consistent state.

Still, it would be nice to have an option for a throttled version of restartall that restarts one at a time, perhaps with a pause.

As a workaround, I could write a script to query forever list and then call forever restart on the node instances one by one.

[jcrugzz]

@markstos interesting concept. Seems like maybe a --limit option on restartall? this would execute for-each-limit type functionality rather than restart them all at once.

[markstos]

@jcrugzz Something like that sounds right. To me, something like --max-at-once would be more evocative, but I think I'd use it by any name if it existed.

[indexzero]

@markstos I think the best about for this would be as a configuration value and not a CLI option.

e.g.

  forever config set rolling true # enables rolling restart
  forever config set rolling 5    # enables rolling restart with a concurrency of 5 

[markstos]

That would be OK with me too. I can see why a config value might be
better-- Someone who's familiar with the difference might set up the
'forever' configuration but later other devs are assigned to "just restart"
something and not think about the details-- or may know which method is
used.

Thanks for considering this!

 Mark

On Thu, Sep 11, 2014 at 4:07 AM, Charlie Robbins [email protected]
wrote:

@markstos https://github.com/markstos I think the best about for this
would be as a configuration value and not a CLI option.

e.g.

forever config set rolling true # enables rolling restart
forever config set rolling 5 # enables rolling restart with a concurrency of 5

Reply to this email directly or view it on GitHub
#603 (comment).

Mark Stosberg
Senior Systems Engineer
RideAmigos

[Tjatse]

I have the same situation, my apps are running like a queue, for example:

forever start app1
forever start app2
forever start app3

app1 is the levelup sever and provides restful apis for app2 & app3, when I using forever restartall, all the monitors have been restarted out of order, It seems app2 is running, but app1 is still restarting, the bad news is: app2 crashed.

So, i think maybe all the processes could be sorted by index when you invoke getProcesses method, and restart one by one.

[markstos]

I'm not sure what happened to the commit above. Here's an alternate pull request which implements just the rolling-restart request:

#665

[markstos]

Could I have some updated feedback on the PR I proposed above. It proposes always using a rolling restart for the "restart" cases with multiple nodes, to avoid downtime when all the nodes have stopped, but none have started yet.

This mirrors the zero downtime upgrade approach that Nginx uses. I'll note that Nginx does not offer configuration options for how a restart happens. The rolling zero-downtime approach is the only option. I recommend emulating Nginx's successful design here.

If you really want to take the entire cluster offline before bring up workers with a new configuration, there is already an option to do this, just like there is in Nginx: Just issue a stop command followed by a start command.

[markstos]

I'll note that the StrongLoop slc tool uses rolling starts. Search for "rolling restart" on this page to see it mentioned.

Also note this chart comparing pm2 with StrongLoop Process Manager and forever. There's a line item for "Rolling Restarts" as a desirable feature which the two competing options have, but forever does not: http://strong-pm.io/compare/

pm2 offers several options:

• stop/start
• restart (sounds like it's implemented like stop/start
• reload (sounds like a rolling restart)
• gracefulReload (sounds like waiting for connections to die before starting)

Implementing the rolling-start as "reload" instead of "restart" as option, but I think it could just be considered an improvement to "restart", without requiring people to figure which of 4 different options for restarting their app is best.