Project Meeting 2024.04.18

Agenda

• Project Admin
• Phase 9a Updates

Action Items

• Consultants to email Joe with current status on budget for Phase 9a work.
• Joe will reach out to Ben Stabler, who is still on that Zephyr group that is doing something with omx, to understand possibilities of getting vendors to add Blosc compression capabilities.

Meeting Notes

Project Admin

• Recent partners only meeting discussed Phase 9b SOW.
• Joe will continue talking to partners and consultants to get Phase 9b paperwork underway (needs to be done this fiscal year).

Phase 9a Updates

• Jeff recently created some new PRs and merged others.
  • The BayDAG contributions have been reviewed and about ready to be merged.
  • A couple of new PRs for things to fix small bugs.
• Main thing that got updated and changed a little bit is on the PR that was originally fixing the school escorting model in sharrow and has evolved into a being a common way to set sharrow compute settings, adding compute controls to components that apply to sharrow and non-sharrow versions of the computations. The team saw things that had a similar flavor (for example, 2 8-bit integers multiplied and causing an overflowed 8-bit integer, which popped up in school escorting) and fixed with pandas eval compute engines. The controls over dropping the columns is a similar thing that you want to be able to turn on/off for different components. All these things are now a single common interface for every model component to do things optimally for each component – if it’s doing something weird, you can change it for that component without doing it for the whole system. Additionally, related to the dropped columns, Jeff found a corner case in the joint tour participation function where there was one check that was hard coded that identified that joint tour participants needed to include one non-preschool age person. The check for whether or not that was happening relies on a column that wasn’t otherwise present in that spec (total number of people that are not preschoolers that are on the joint tour), this was dropped but is needed later on for a check. The PR adds a layer of column names that need to be protected even if they are not included in the model spec, doing it in the model config file and not hard coding in the code.
• Tests are running now and then PR will be merged.
• Anticipate by next Tuesday that everything will have been accepted and then able to do an end-to-end run of ABM3, sharrow on and sharrow off, as well as the MTC example with all the new changes, sharrow one and sharrow off.
• School escorting optimization (outside of the sharrow stuff) is still failing tests, the progressive tests, which may just need be regenerated. The team wants to finish the core fixes first, and then regenerate the progressive tests, and then retest the school escorting optimization.
• Sijia opened an issue on the unpredictable memory release.
• Omx compression issue
  • ABM3 model has many skims, which take about 25 minutes to read in.
  • The key piece of why it’s really slow is that the omx standard calls for hdf5 zlive compression which is widely available to run on anything but it is slow. We can use higher compression libraries since we are running on higher end machines. There is a new library (Blosc) and you can run the skims in HDF5 format. The change to the compression algorithm doesn’t impact the memory used but the read time improves by a factor of 10. No other changes are needed. You aren’t saving time if you are converting from zlive to blosc every time, so it would be good to use blosc from the beginning. We may want to ask the proprietary vendors to add the capability if it’s not possible (Jeff has done it in emme, not sure about others, pretty sure Transcad can just do this but need to get into gisdk to make it happen).
  • What vendors are using: Cube (MWCOG, SFCTA, MTC, ARC), emme (PSRC, CMAP, SANDAG), PTV (Oregon), Transcad (SEMCOG)
  • Joe will reach out to Ben, who is still on that Zephyr group that is doing something with omx
• Planning to meet on Tuesday for technical meeting