Research Request - RT Predictions / Real-Time Information POC metrics

[tiffanychu90]

Complete the below when receiving a research request, and continue to add to this issue as you receive additional details and produce deliverables. Be sure to also add the appropriate project-level label to this issue (eg gtfs-rt, DLA).

Research Question

Single sentence description:
Ahead of RFPs/procurement going on, we need to figure out conceptually what performance metrics we might be interested in, and do a proof-of-concept with those calculations. Slack thread.

This goes into rt_predictions analytics-driven warehouse development.

Detailed description:

• From "what we collect" to "what can we learn/measure that's useful", use this POC to capture our learnings. Draft metrics
• Inform future full-scale implementation
• We haven't yet touched TripUpdates tables too much, so let's just grab a workable amount. For this MVP, stick with 2 operators, and we can even pick 2 routes for each operator and get at the calculations across trips.
• Test out the various aggregations.
• Metrics are: update completeness, wait time, prediction inconsistency, and reliable prediction accuracy

How will this research be used?

Inform future tables to store in our warehouse that are more aggregated than the raw, but also provide enough granularity. We can't go reaching into TripUpdates regularly because it's way too big.

Data sources

• Cal-ITP data sources:
• RT TripUpdates
• schedule stop_times

Deliverables:

1. preprocessing - cutoff for trip updates before trip start
2. update completeness - trip updates dataset
3. update completeness - vehicle positions dataset
4. expected wait time
5. prediction inconsistency
6. reliable prediction accuracy

Timeline of deliverables:

End of April

[tiffanychu90]

Related task to include in pre-processing step

• Research Task - When to start measuring RT prediction accuracy? #726

Tweaks to be made: Slack thread

[tiffanychu90]

Tweak how metric 3 (prediction inconsistency / jitter) is calculated by simplifying it to be dependent only on prediction. Slack thread

[tiffanychu90]

Slack thread about the mart_ad_hoc tables and Laurie's Google Doc with notes about all the tables needed any time we answer stop_time_updates questions (set of 3 tables with individual prediction (stop_time_update), the final update (the arrival at the stop), and combo of scheduled stop times + scheduled trips.