The 'gtfs' class

[dhersz]

Hello all.

In this thread I'll discuss the gtfs class. I don't have an extensive experience with S3 classes and methods, so I tried to stick with the recommendations made by Hadley in the S3 chapter of his Advanced R book. I'll provide some notes on where I diverged from them, and why.

According to Hadley, it's good practice to create a constructor, a validator and a helper for an S3 class. The constructor function is meant to be used by people with more experience with the objects in question, so it doesn't include that many checks (thorough checks are included in the validator function, which I'll show later). The constructor for the gtfs class is new_gtfs(). It receives a list and assigns the gtfs class to it, as well as any other attribute you may also want to include.:

library(gtfsio)

path <- system.file("extdata/ggl_gtfs.zip", package = "gtfsio")
temp <- import_gtfs(path)

not_gtfs <- unclass(temp)
class(not_gtfs)
#> [1] "list"
attributes(not_gtfs)
#> $names
#>  [1] "calendar_dates"  "fare_attributes" "fare_rules"      "feed_info"      
#>  [5] "frequencies"     "levels"          "pathways"        "routes"         
#>  [9] "shapes"          "stop_times"      "stops"           "transfers"      
#> [13] "translations"    "trips"           "agency"          "attributions"   
#> [17] "calendar"

gtfs <- new_gtfs(not_gtfs)
class(gtfs)
#> [1] "gtfs"
inherits(gtfs, "gtfs")
#> [1] TRUE

also_gtfs <- new_gtfs(not_gtfs, random_att = "hello")
class(also_gtfs)
#> [1] "gtfs"
attributes(also_gtfs)
#> $names
#>  [1] "calendar_dates"  "fare_attributes" "fare_rules"      "feed_info"      
#>  [5] "frequencies"     "levels"          "pathways"        "routes"         
#>  [9] "shapes"          "stop_times"      "stops"           "transfers"      
#> [13] "translations"    "trips"           "agency"          "attributions"   
#> [17] "calendar"       
#> 
#> $class
#> [1] "gtfs"
#> 
#> $random_att
#> [1] "hello"

It also receives the subclass argument, which is used to specify a "more specific" class, while still inheriting from gtfs:

also_gtfs <- new_gtfs(not_gtfs, subclass = "dt_gtfs")
class(also_gtfs)
#> [1] "dt_gtfs" "gtfs"
inherits(also_gtfs, "gtfs")
#> [1] TRUE

Why is that useful? Because now we can start differentiating the objects created in our packages. For example, it might be very useful to differentiate objects created by {gtfstools} and {tidytransit}, because data.table syntax doesn't work on tibbles. Then we can do something like:

gtfs_gtfstools <- new_gtfs(
  not_gtfs, 
  subclass = "dt_gtfs", 
  validation_result = "very_complex_object_here"
)
class(gtfs_gtfstools)
#> [1] "dt_gtfs" "gtfs"

not_gtfs_tbl <- lapply(not_gtfs, tibble::as_tibble)

gtfs_tidytransit <- new_gtfs(
  not_gtfs_tbl,
  subclass = "tbl_gtfs",
  validation_result = "another_complex_obj_here"
)
class(gtfs_tidytransit)
#> [1] "tbl_gtfs" "gtfs"

As you can see, a gtfs object may be composed either by data.tables or tibbles, it really doesn't matter. new_gtfs() will only raise a few errors when:

• The object passed to it is not of typeof() list;
• The object passed to it contains an unnamed element;
• The object passed to subclass is not a character vector;

new_gtfs(1:3)
#> Error in new_gtfs(1:3): 'x' must be a list.

new_gtfs(list(shapes = 1:5, 1:6))
#> Error in new_gtfs(list(shapes = 1:5, 1:6)): Every element in 'x' must be named.

new_gtfs(not_gtfs, subclass = 17)
#> Error in new_gtfs(not_gtfs, subclass = 17): 'subclass' must be a character vector.

As you can see, these are not very thorough checks. That's because, again, this function is meant to be used by us, package maintainers, and not final users. A basic usage, for example, is import_gtfs(), which does a lot of checks itself, so a situation like any of the above is highly unlikely to occur. Another situation food use, for example, would be inside our own custom GTFS reading functions. See an example workflow below:

gtfstools_read_gtfs <- function(path) {
  
  # ....
  
  gtfs <- gtfsio::import_gtfs(path)
  gtfs <- convert_from_standards(gtfs)
  val_obj <- validate_gtfs(gtfs)
  gtfs <- gtfsio::new_gtfs(
    gtfs, 
    "dt_gtfs", 
    validation_result = val_obj
  )
  
  # ....
  
}

More complicated checks are made in the validator function, called assert_gtfs(). (here is where I first diverge from Hadley's suggestions, because he suggests that these functions should take the name of validate_...() - but as you can see in the code above, {gtfstools} (and {tidytransit} as well) already includes a validate_gtfs() function, so I thought it would be wiser to prevent yet another name clash)

So what does this assert_gtfs() check?

• Yet again if all elements are named;
• If all elements inherit from data.frame;
• An exception to the rule above is when a . element is present. If it is, then it must be a list;
• And every element inside this . sub-list must also be named and inherit from data.frame.

The content of these data.frames is never checked. If all these checks are successful, then assert_gtfs(x) invisibly returns x:

bad_gtfs1 <- new_gtfs(list(a = data.table::data.table(a = 1), b = 1:5))
class(bad_gtfs1)
#> [1] "gtfs"
assert_gtfs(bad_gtfs1)
#> Error in assert_gtfs(bad_gtfs1): Every element in a GTFS object must inherit from 'data.frame'. The following elements do not: 'b'

bad_gtfs2 <- new_gtfs(list(a = tibble::tibble(a = 1), b = list(a = 1, b = 2)))
assert_gtfs(bad_gtfs2)
#> Error in assert_gtfs(bad_gtfs2): Every element in a GTFS object must inherit from 'data.frame'. The following elements do not: 'b'

bad_gtfs3 <- new_gtfs(list(a = data.frame(a = 1), . = matrix(1:4)))
assert_gtfs(bad_gtfs3)
#> Error in assert_gtfs(bad_gtfs3): The '.' element of a GTFS object must be a list.

bad_gtfs4 <- new_gtfs(
  list(
    a = data.frame(a = 1),
    . = list(
      hello = data.frame(b = 2),
      data.frame(c = 3)
    )
  )
)
assert_gtfs(bad_gtfs4)
#> Error in assert_gtfs(bad_gtfs4): Every element inside '.' must be named.

bad_gtfs5 <- new_gtfs(
  list(
    a = data.frame(a = 1),
    . = list(
      hello = data.frame(b = 2),
      goodbye = matrix(1:4)
    )
  )
)
assert_gtfs(bad_gtfs5)
#> Error in assert_gtfs(bad_gtfs5): Every element inside '.' must inherit from 'data.frame'. The following elements do not: 'goodbye'

gtfs <- new_gtfs(not_gtfs)
finally_good_gtfs <- assert_gtfs(gtfs)
identical(gtfs, finally_good_gtfs)
#> [1] TRUE

Ok, so where does this validator function should be used? Hadley recommends using it inside helper functions. Helper functions are functions that final users should use to create these objects. So, for example, tibble::tibble() is a helper, while tibble::new_tibble() is a constructor. So it feels like very natural to create a helper called gtfs().

But, in fact, I didn't lol. This is because I feel like the gtfs() function would not be that useful. Instead, I think that each of our own packages should include helper functions that are specific to our needs. So for example, {gtfstools} could very well benefit from a dt_gtfs() function that might resemble something like:

dt_gtfs <- function(...) {
  
  tmp_gtfs   <- gtfsio::new_gtfs(..., "dt_gtfs")
  
  # .... my own set of checks and stuff here
  
  val_object <- validate_gtfs(tmp_gtfs)
  gtfs <- new_gtfs(tmp_gtfs, "dt_gtfs", validation_result = validation_result)
  
  # now use assert_gtfs() to make sure gtfs remains a valid 'gtfs' object
  
  gtfs <- gtfsio::assert_gtfs(gtfs)
  
}

And the user would use it somewhat like this dt_gtfs(shapes = data.table(.....), trips = data.table(.....), *andsoon*). What do you think of this? Should {gtfsio} include a helper function? Or you see it as a better fit for our packages?

Methods

The gtfs class has a few, though not many, custom methods. These can be found in the gtfs_*.R files.

print.gtfs() is very similar to print.default(), but it doesn't print the class attribute:

print(gtfs["shapes"])
#> $shapes
#>    shape_id shape_pt_lat shape_pt_lon shape_pt_sequence shape_dist_traveled
#> 1:    A_shp     37.61956    -122.4816                 1              0.0000
#> 2:    A_shp     37.64430    -122.4107                 2              6.8310
#> 3:    A_shp     37.65863    -122.3084                 3             15.8765

print.default(gtfs["shapes"])
#> $shapes
#>    shape_id shape_pt_lat shape_pt_lon shape_pt_sequence shape_dist_traveled
#> 1:    A_shp     37.61956    -122.4816                 1              0.0000
#> 2:    A_shp     37.64430    -122.4107                 2              6.8310
#> 3:    A_shp     37.65863    -122.3084                 3             15.8765
#> 
#> attr(,"class")
#> [1] "gtfs"

But don't worry. print.gtfs() preserves the print.default()'s property of invisibly returning the very same object passed to it.

tmp <- print(gtfs["shapes"])
#> $shapes
#>    shape_id shape_pt_lat shape_pt_lon shape_pt_sequence shape_dist_traveled
#> 1:    A_shp     37.61956    -122.4816                 1              0.0000
#> 2:    A_shp     37.64430    -122.4107                 2              6.8310
#> 3:    A_shp     37.65863    -122.3084                 3             15.8765
class(tmp)
#> [1] "gtfs"

Please note that any other attributes will not be stripped from the print. For example:

gtfs <- gtfs["shapes"]
attr(gtfs, "validation_result") <- "hello, world!"
print(gtfs)
#> $shapes
#>    shape_id shape_pt_lat shape_pt_lon shape_pt_sequence shape_dist_traveled
#> 1:    A_shp     37.61956    -122.4816                 1              0.0000
#> 2:    A_shp     37.64430    -122.4107                 2              6.8310
#> 3:    A_shp     37.65863    -122.3084                 3             15.8765
#> 
#> attr(,"validation_result")
#> [1] "hello, world!"

This preserves the default behaviour of {tidytransit} and {gtfstools} today of always printing the validation_result alongside the gtfs object itself. I'm not sure if {gtfsrouter} and {gtfs2gps} use any other custom attributes as well.

To be honest I'm not sure if I like the current {gtfstools} behaviour of always printing the validation result object, and I'd very happily change the printing method to hide any custom attributes. If you agree with this please let me know and I'll change it. Otherwise I'll probably change it directly on {gtfstools}.

{gtfsio} also include custom methods for subsetting GTFS objects. The first, which was actually shown above, is that subsetting it with [ preserves the gtfs class ([.default would strip the class from the object):

class(gtfs)
#> [1] "gtfs"

class(gtfs["shapes"])
#> [1] "gtfs"

class(gtfs[c("shapes", "trips")])
#> [1] "gtfs"

The $ and [[ operators were also tweaked to raise warnings when attempting to subset missing elements (this warning aside, they behave exactly like the default operators).:

gtfs["ola"]
#> Warning: This GTFS object doesn't contain the following element(s): 'ola'
#> $<NA>
#> NULL

gtfs$ola
#> Warning: This GTFS object doesn't contain the following element(s): 'ola'
#> NULL

gtfs[["ola"]]
#> Warning: This GTFS object doesn't contain the following element(s): 'ola'
#> NULL

Perhaps an error would be more useful here? I tried sticking to default behaviour as much as possible, but maybe an error is more telling here than a warning.

I also think it might be useful to include some custom methods to [<-, $<- and [[<- to prevent non data.frame-like objects to be assigned to the GTFS object (exception: . sub-lists), but I haven't done it. What do you think?

Sorry for the long post, but I wanted to be very thorough here. Cheers!

[mpadge]

Hi @dhersz! Thanks again so much for your brilliant work on this package. I am finally able to ditch all former gtfsrouter import routines in favour of this package. Everything works pretty much straight out of the box, and even highlighted a couple of deficiencies in the ways that gtfsrouter was doing things. One very minor different is in the class structure, which is almost identical except that gtfsrouter follows "standard practice" in class definitions across languages in appending inherited class structures to any new definitions. In practice, that means that gtfsio just has class = "gtfs", whereas gtfsrouter has class = c ("gtfs", "list"). These things really are just lists, and so it is good practice to keep that to enable default expectations of list behaviour to apply.

(Kinda technical reasons why, and they're not even directly important for now, because R can only implement method dispatch on the first listed class entry, but that will hopefully change sometime relatively soon, after which time things like this will matter. So you can also see it as future-proofing, if nothing else.)

[dhersz]

Hi @mpadge. I'm very happy to hear that you'll start using gtfsio inside your package! Unfortunately I've been very busy with my thesis (who needs this anyway??) recently and haven't been able to implement it in gtfstools myself yet :|

Regarding the change you suggest, I was under the impression that any gtfs objects would preserve the list methods because mode(gtfs) == "list" and is.list(gtfs) == TRUE). But I just checked it and you are 100% right:

a_list <- list(ola = 2)

sloop::s3_dispatch(print(a_list))
#>    print.list
#> => print.default

gtfs <- structure(a_list, class = "gtfs")

sloop::s3_dispatch(print(gtfs))
#>    print.gtfs
#> => print.default

mode(gtfs)
#> [1] "list"
is.list(gtfs)
#> [1] TRUE

gtfs <- structure(a_list, class = c("gtfs", "list"))

sloop::s3_dispatch(print(gtfs))
#>    print.gtfs
#>    print.list
#> => print.default

(which by the way looks a bit counter-intuitive for me, but I don't have that much experience with this)

I'll make the proposed changes and report back here.

[dhersz]

Fixes in 322576d.

Any other suggestions? I'll wait a bit before submitting it to CRAN again so we can make any changes that may be required upon gtfsio implementation in gtfstools and gtfsrouter.

[mpadge]

Yes, i think it'd be a good idea to wait just a bit longer until I've gone through the whole process of integrating into gtfsrouter. Hoping to have that done this coming week.

[dhersz]

Hi @mpadge. Have you managed to look into this? I feel like we have made some good progress since the last submission and I think it is worth a new CRAN version already. If you plan to work on this integration in the next couple days I could happily wait until you're done, otherwise I would be equally happy to submit it to CRAN as is. Cheers!

[mpadge]

Thanks for the ping @dhersz. I'm on holidays at present, and won't have much time for gtfsrouter for next few weeks, so please feel free to update CRAN version in the meantime. The changes already implemented will help a lot - thanks!