Two analyses at a time doesnt work

[brianlball]

Running 2 analyses Resque queues in the web-background container can cause out of order ResqueJobs::RunAnalysis jobs.
For example, when an LHS analysis is submitted to OSAF, it is actually two jobs. The first is analysis_type='lhs' followed by analysis_type='batch_run'. see batch_run_methods. They are both Resque.enqueue'd right after each other, but there are no hooks or requirements that one goes before the other. With only 1 Resque Worker in the web-background container, only one job in the analyses queue is run at a time, the first being the non-batch_run analysis_type.

If there are 2 Resque workers in the web-background container, either by changing start-web-background.sh or helm-chart, batch_run.perform can start before lhs.perform can complete which results in datapoints trying to be run before they are made.

This is how it should work, where LHS generates the datapoints and batch_run then data_point.submit_simulation:

VS

This is what can happen with 2 analysis queues right now, batch_run has no datapoints to submit because lhs has not created them yet:

[brianlball]

A solution is to use resque hooks to set a redis flag in the first analysis_type:

before_perform_assign_started
if analysis_type != 'batch_run'
Resque.redis.set("analysis:#{analysis_id}:started", true)

Then remove it when the job is done:

self.after_perform_remove_started
if analysis_type != 'batch_run'
Resque.redis.del("analysis:#{analysis_id}:started")

And have batch_run check that the right conditions are met before proceeding; if not, then requeue the job without error.