Two analyses at a time

[brianlball]

#717
#753

Running 2 analyses Resque queues in the web-background container can cause out of order ResqueJobs::RunAnalysis jobs.
For example, when an LHS analysis is submitted to OSAF, it is actually two jobs. The first is analysis_type='lhs' followed by analysis_type='batch_run'. see batch_run_methods. They are both Resque.enqueue'd right after each other, but there are no hooks or requirements that one goes before the other. With only 1 Resque Worker in the web-background container, only one job in the analyses queue is run at a time, the first being the non-batch_run analysis_type.

If there are 2 Resque workers in the web-background container, either by changing start-web-background.sh or helm-chart, batch_run.perform can start before lhs.perform can complete which results in datapoints trying to be run before they are made.

This is how it should work, where LHS generates the datapoints and batch_run then data_point.submit_simulation:

VS

This is what can happen with 2 analysis queues right now, batch_run has no datapoints to submit because lhs has not created them yet:

[brianlball]

[brianlball]

parallel analyses with the new test

[brianlball]

#618

[nllong]

wow, this is fancy! Good work figuring out and fixing!