Project Meeting 2024.01.16

Agenda

• Phase 9 Contracting

Action Items

• Bench contractors to send revisions to proposed SOWs and budget estimates to Joe before Thursday's 1/18 meeting.
• Joe to cancel partner's only meeting on Thursday 1/8

Meeting Notes

Admin: Phase 9 Contracting

• Bench contractors received proposed Scopes of Work for early Phase 9 work
• Concerns about the performance of the implementations/examples with sharrow as well as performance of past merges/updates without sufficient performance evaluation is driving the currently proposed SOW.
• Through partner-only discussions, the consortium decided to prioritize performance of current models/implementations by focusing on a couple of examples and really optimizing them: identifying key issues people have and working through them.
• There may be a disconnect between the core engineered code and the way the models were implemented. This is an opportunity to clarify/improve how users are expected to configure and code the models so that it’s straightforward, clear, and appropriate, as well as an opportunity to provide guidance for future users. Improving usability is a good goal for ActivitySim, as well as performance. Also, we want ensure that the experience of those with implementations ahead of others are positive.
• The goal is to make this next phase of work collaborative and expedient.
• Scope comments
  • BayDAG pull requests should be merged prior to starting Phase 9 work, meaning Phase 8 merges need to be completed first; then, bring in the BayDAG merges and test to make sure it works and matches the SANDAG results. It’s a little out of sync right now, since BayDAG was pivoting off of a pre-Phase 8 branch. It will require a little bit of effort to make those work, not code review, to make it up to date to where we are today. Need to conduct testing once those changes are made and updated, and re-validate to make sure everything is correct. That’s a good starting point for performance tasks of Phase 9. Need to make sure it’s working before making it run fast. RSG has some proposed SOW to do this.
  • MTC, SANDAG, and RSG are in agreement that most agencies would want to incorporate these updates to make sure future testing would benefit from the updates developed under the BayDAG branch.
  • However, CS can start on some tasks (such as vehicle choice and school escorting models) before the BayDAG example is fully merged, as long as the BayDAG code updates don’t touch those. RSG to confirm.
• RSG made a presentation to MWCOG on challenges of implementing sharrow . MWCOG to send the presentation to the consortium and then decide if it’s worth meeting time to present it.
• We need to know if there are problems manifesting in the MWCOG model but not the BayDAG model. We should make the BayDAG model have that problem/feature in order to be able to fix it. There could be an additional task to replicate the MWCOG problems in the BayDAG model, for example, the district level-constants in workplace location choice. Conceptually, this is a good idea, but worried it could spin out of control when we want to limit the work that’s being done to just these implementations/examples.
• Request to look into data type updates in the next round of work, after this optimization effort.
• An outcome of this effort is to make sure the optimization is clear across different contractors and other potential users and increase communication. * For RSG and WSP, their contributions to this effort could include identifying all the problems they’ve had in implementation, identifying pain points of implementation, not necessarily limited to performance issues. RSG and WSP tasks could also include code review and documentation.
• For CS scopes, there is text about sharrow and estimation mode. In sharrow’s documentation, it clearly says clearly that it doesn’t work comprehensively with tracing and estimation – not because it can’t, but because it wasn’t prioritized. The estimation mode code was written to do the work without consideration about performance, since estimation datasets were smaller that a full model population, so it didn’t seem a high priority. There was very little effort towards making estimation mode efficient. For this optimization task, the goal is not to comprehensively optimize estimation mode but to not ignore it. We'll want to document whether or not optimization tasks do/do not impact estimation mode and understand implications. CS to revise the scope text as appropriate. Keep it (estimation mode) in mind, think about it, and not implement fixes that make estimation a lot worse, for example. Estimation mode won’t fall off the map but doesn’t need to be resolved.
• Next steps
  • Have updated SOWs by Thursdays call, along with Phase 8 update.