Implement server & worker init & final scripts

[rHorsey]

From @asparke2 on May 11, 2017 16:5

Necessary for ResStock and ComStock

Copied from original issue: NREL/OpenStudio#2608

[rHorsey]

Worker is implemented and tested. Server will get implemented next week.

[rHorsey]

@shorowit found a lovely bug with *.sh files coming from windows. This has been addressed using dos2unix in 7dc6836

[rHorsey]

Worker looks to be mostly stable. There are some paradoxes in when to apply worker finalization in the workflow, but otherwise the implementation appears solid.

[rHorsey]

Worker is being merged in. Sever remains to be done.

[rHorsey]

From @asparke2 on August 10, 2017 1:12

Sweet!

[rHorsey]

Defining server finalization as major, and server initialization as minor.

[anyaelena]

Per Pivotal, it looks like this should be assigned to me. It would be helpful if I could get a concrete example of what we are trying to do. I discussed with @nllong running code on only the server docker container as part of the deployment process - presumably this is "initialization". Is "finalization" a final stage of setup or part of the destroy process for the server container? And are we only talking about Docker here?

[anyaelena]

@nllong @rHorsey Notes from our call:

Analysis finalization should be triggered from web_background but run on web_1, which has superior resources and memory management.

Consider triggering this via a HTTP action for the analysis. Since this may take longer than a standard HTTP request and we don't want it to time out, we will still need the actual script(s) to run in the background on web_1, meaning that we need to use Resque on web_1.

So:

1. Analysis completes
2. background task calls http method on web
3. web adds "run_finalization_scripts" to queue for background tasks that run on web.

Now we can all think what's wrong with this...

[anyaelena]

I guess there is no reason for the http call since we are going to need to use background task queue regardless. So:

1. Analysis completes
2. web_background puts run_finalization_scripts in queue
3. web_1 runs "run_finalization_scripts" in background