stop_times reguarization routine

[mpadge]

@dhersz There are no GTFS standards for how stop_times.txt should represent trips starting before and after midnight, other than to "enter a value greater than 24:00:00 for trips starting after midnight". I often need to "regularize" a stop_times table by converting all entries to have departure_time entries between 0 and 23:59:59. Among other reasons, this is important to extract statistics on actual trip frequencies and the like, which are best standardized to estimates within a single 24 hour period.

Another reason this is important is because systems which effectively run round the clock must nevertheless cut their stop_times somewhere to give a representation within their GTFS feed of a single "day". These kinds of feeds often repeat trips early in the morning at the ends of feeds (with times > 24:00:00), so contain redundancies which must be removed for accurate statistical analyses.

A stop_times regularization function would perform the two simple steps of:

1. Converting all trips to equivalent departure (or arrival) times of <24:00:00; and
2. Removing any redundant trips to leave unique trips only

Do you think that gtfstools would be a good home for a routine like that? If so, I could easily open an issue there, an help get the routine together.

Thoughts @polettif @rafapereirabr @tbuckl ?

[dhersz]

Great idea @mpadge!

Some thoughts that came up when reading your post:

• Only trips whose first departure time exceeds 24:00:00 should be converted (i.e. if the trip departs from the first stop at 23:59:59 it should not be converted, even if it departs from subsequent stops later than 24:00:00). Does it sound right?
• I wonder if this may bring some issues with services (as specified in calendar.txt). Let's say a trip is specified to run only on saturday, but it departs from the first stop at 25:00:00. This is, effectively, sunday 01:00:00. If we convert 25:00:00 to 01:00:00 in this case, while keeping the service unaltered, the trip would run only at saturday 01:00:00, rather than sunday 01:00:00. I'm not sure if we should treat these cases or not (and neither if I have made myself clear lol), as they seem rather poorly specified from the beginning, but it's some food for thought.

Edit:

• A good way to prevent this second potential issue is to specify a new service for the trips that were regularised, that is identical to the original one but shifted one day forward. Using the same example above, we would have to set a new service that runs on sundays for that trip. In the same spirit, if the original service runs from monday to friday, the new would have to run from tuesday to saturday.
• But then another issue: this would probably mess up the frequencies-and-the-like statistics that you mentioned in the first place, I guess.

Edit2:

And btw, I think {gtfstools} is a great place for a function of that nature! I and the {gtfs2gps} team will be working in the following month or so on bringing some of the {gtfs2gps} features to {gtfstools}, but feel free to open an issue there or start a PR so we get this moving!

[rafapereirabr]

Hi all. This is a really tricky problem we've also faced with the gtfs2gps package when we compute travel speeds between stops. We went around this problem by doing the following:

• Converts departure_times to seconds
• Subtract consecutive stops, like t2 - t1, t3 - t2, t4 - t3, ...
• For the negatives values of the above operation, we add 86400 seconds (equivalente of 24 hours).

Here is the code.

This is a solution tailored to our need of calculating speeds between stops, but I would be glad to hear how the others have been addressing the midnight trips problem.

[polettif]

For my use cases it wasn't necessary to normalize times and I think it might introduce more problems than it might solve. A standard "public transit day" starts/ends at ~3am so there's always some overlap. tidytransit's set_hms_times converts times to hms/difftime objects which are basically a time in seconds. They're not limited to a 0-24h range, so values after midnight are just >84600 (see hhmmss_to_seconds).

This allows for simple calculations (no need to check for negative values) and ggplot handles hms scales quite nicely, as you can see in the timetable vignette. However, for calculations the date specified refers to the "pt day" a trip belongs to rather than the actual date it happens. That's not really an issue though as trips after midnight are generally not that important and they "belong" to the previous day anyways, in Switzerland at least.