enable replicas for web-background for parrell project processing #24
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #23
It turns out this feature request is actually more of a helm/k8s deployment issue than the original issue filed here: NREL/OpenStudio-server#618
The web-background pod spawns a resque worker that services the background and analyses queues. When a user submits a job to openstudio-server (e.g. cli command below), this action creates a new job which in turn gets put into the analysis resque queue. Then, when the web-background receives a queue message and processes it, it loops through all the data points associated for that analysis and adds these to the simulation queue for processing by the worker nodes. It waits for all of those simulations to complete before closing the analysis job and then begins processes the next analysis in the queue.
Conceptually, it looks like this.
Currently, there is only one resque worker that services the analyses queue, which results in one analysis being ran at a time. By increasing the number of web-background k8s pods, you can increase the number of resque analyses jobs which translates into parallel analysis processing. This is especially useful when you have a lot of idle workers (e.g. close to the end of a large batch run and waiting on a few long running data points). This will maximize cpu usage on worker nodes and should reduce costs by avoiding idle resources.
This PR makes the default of 2 web-background workers and reduces the cpu and memory as the jobs are not computational or memory intensive.
The simulations queue is actually first-in-first-out, so openstudio-server will process stimulations based on the time they were submitted, but if there are available workers and no jobs are left in that analysis queue, it'll begin servicing the next analysis' data points.
Currently, PAT only supports one job at a time but you can submit multiple analyses uses the cli command as show below.
openstudio_meta" run_analysis --ruby-lib-path="/Applications/ParametricAnalysisTool-3.2.1/ParametricAnalysisTool.app/Contents/Resources/OpenStudio/Ruby" "/Users/tcoleman/git/OpenStudio-PAT/sample_projects/Office_HVAC/Office_HVAC.json" "http://a02d1056e36dd4598bb372de92ce27c0-224809964.us-west-2.elb.amazonaws.com"openstudio_meta" run_analysis --ruby-lib-path="/Applications/ParametricAnalysisTool-3.2.1/ParametricAnalysisTool.app/Contents/Resources/OpenStudio/Ruby" "/Users/tcoleman/git/OpenStudio-PAT/sample_projects/Office_HVAC/Office_HVAC.json" "http://a02d1056e36dd4598bb372de92ce27c0-224809964.us-west-2.elb.amazonaws.com"