[💡 FEATURE REQUEST]: Parallelize the pipeline create and destroy process

[Kaspiman]

Plugin

JOBS

I have an idea!

I ran into a problem with Jobs plugin. It appears when i use a large number of pipelines, for example we have 80.

The RR takes a long time to destroy and serve, because each pipeline (with driver) stops and starts sequentially.

• https://github.com/roadrunner-server/jobs/blob/master/plugin.go#L140
• https://github.com/roadrunner-server/jobs/blob/master/rpc.go#L141

As a result, the restart takes a few minutes, and in case of failure (for examle, network problems with AMQP driver), a few more minutes.

Tasks

Beta Give feedback

1. Speed up destroy command
2. Speed up serve command

[rustatian]

@Kaspiman You may download nightly build with this change from the GitHub actions: https://github.com/roadrunner-server/roadrunner/actions/runs/5881379978

[Kaspiman]

@rustatian I have a few questions about this code, especially in errorgroup waiting

[rustatian]

@Kaspiman Sorry, missed your comments. Didn't get a notification about them.