Project Meeting 2018.11.30
- Ran 100% sample full scale example in 59 minutes on one Azure Linux machine with 64 processors and 432GB RAM
- Ran the same example in 2 hours on an Azure Windows box
- The Linux example includes tweaks to the low level Intel MKL settings used by OpenMP and pandas/numpy
- We plan to experiment a bit more with the same MKL environment variables settings on Windows to decrease runtimes
- Cost of the Linux run was $5 and max RAM usage was 242GB (see below)
- Added three new example configurations, see dev:
- mp - Multi-processing setup
- azure - Azure cloud based setup
- stride - Separate machines setup (but not really ready for use since incomplete)
- Coding largely complete, just inline docs and some settings clean-up required
- @RSG still todo:
- Do some local machine Windows runs with smarter Intel MKL environment variable settings
- Run a cloud based 128 processor 2TB RAM setup as well
- Revise user documentation and then do a pull request (and release)
- Plan to complete by TRB
- @PSRC and SFCTA to experiment with mp as well
- PSRC has mp running locally and on AWS; both with Windows
- Started reviewing existing CT-RAMP and DaySim code and draft plan, see Task 3
- Drafted a rough prototype this week that works in a single process
- For mp, need to collate results after each iteration to calculate shadow price
- For the user configuration, a new concept is added - shadow price segment
- This allows for specifying shadow price segments, worker, elem student, high student, etc. and an expression file to identify it + what size terms to use
- For geography constraints, these would be handled by adding an expression to the sampling expression file (such as home school district)
- Will allow for multiple types of convergence - iterations, goodness-of-fit, etc.
- Planning to complete shadow pricing by TRB as well