WIP: Date/DateTime support through the Time module

[quinnj]

As discussed in #3524 and @karbarcca/Datetime#12.

This is most of the Datetime package reworked to simplify, speed up, and polish some functionality (e.g. date parsing). Notably missing is full Timezone support, which I've also been reworking. I figured I'd submit this PR and things can get ironed out a little more before all the Timezone code is finalized. I also have the Datetime manual and function reference which can be folded in fairly easily once things settle down.

I've included a few RFC, TODOs, and some comments on certain pieces that I'd appreciate feedback on. Also any feedback on how to fold this into Base would be appreciated as I've never pushed anything like this before.

Cheers!

[StefanKarpinski]

I'm excited about getting this into Base. That is a lot of new exports, however. What I'd propose is that Base can export only the bare minimum to work with dates and times plus the Time module so that if you want, e.g. the day and month constants you can do using Time and have them available. Which makes using Time sound a bit weird. Maybe using Dates to import all of those names would make more sense.

[quinnj]

Yeah, I wasn't sure on the exports, so I should re-exported everything from the module. I think it makes sense to keep a lot of it inside though. I just pushed some export reductions.

[JeffBezanson]

Fully agree: great functionality, way too many exports in Base :) I think there are still too many. It should be a small handful, or just Time. We will probably do this more to other parts of Base as well.

Do we really need both Calendar and CALENDAR etc.? The latter doesn't seem to be used.

As usual, I think UTC should be an instance of something, not a type, so we'd have isa(UTC,Timezone).

How does this relate to our old friends strptime, strftime, time, etc.? I'd really like to remove any duplication, including removing/deprecating the existing functions and types if possible.

[JeffBezanson]

The <: Ranges declarations worry me a little. I'm pretty sure a lot of the functions of Ranges in Base won't work correctly on these, which is partly the fault of the existing code. For example, I don't think there's a generic way to construct a range given a start, step, and length (unlike colon, which takes start, step, stop, which is the wrong approach; oh the things one learns :) ).

What is the fun in DatetimeRange1 for?

This makes me think we should take another crack at making Range work for ordinal types. We tried once for Char and concluded it wasn't worth it, but TimeType is another example where it's needed.

[quinnj]

Some answers:

-I took the exports down to just Date and Datetime.
-CALENDAR and TIMEZONE were relics of a former structure, I've taken them out.
-For UTC, the only tricky is that we're not only dispatching on UTC, but it's serving as a parameter tag for Datetime. So a UTC instance doesn't seem to be able to fit the parameter tag role; at least as far as I understand things. Is there another way to look at it? I used the Sort module as my guide here, but it seemed like I needed an abstract subtype to fulfill the tag role.
-I think we can get rid of/at least unexport strtime and strptime as there's some date string parsing code in the Time module. time() is actually used by the now() function to convert the unix milliseconds to a Datetime instance, so perhaps it can just be moved over. The TmStruct type et al. can probably go since their functionality is pretty well subsumed.
-Yeah, I'll admit I wasn't super comfortable with the Range code, but I think it's definitely functionality that can be convenient. The first time I did it, it was entirely separate. This time I was trying to be clever by hooking in with Ranges.
-The fun in DatetimeRange1 is an "inclusion" function. Basically, it allows you to not only specify a range of dates, but also an anonymous inclusion function that indicates whether a given date should be "included" in the range or not. This was a stab at implementing the bulk of the runt library; though perhaps using a Range type isn't appropriate here since you pretty much always want the resulting [x] dates instead of a range type. I'll look into refactoring that to see if it can stand-alone without too much hassle. The functionality allows usage like:

dt = Date(2009,1,1)
dt2 = Date(2013,1,1)
ff = recur(dt:dt2) do x
        # Get dates of Thanksgiving between two dates
    dayofweek(x)==Thursday && 
    month(x)==November && 
    dayofweekinmonth(x)==4
end

[JeffBezanson]

Ah, that is cool. I'm not sure what to call it, but it seems more like a set or sieve of dates. Probably doesn't qualify as a Ranges.

I see your point about UTC. Maybe it should be an empty immutable type instead, just to prevent it from being subtyped.

Feel free to hack and slash as necessary to remove subsumed stuff. Nothing worse than crufty stuff lying around that nobody's sure whether it's ok to use. Of course the help text will have to be updated.

I appreciate all the tests; they look great.

[quinnj]

Not sure if this could be more generalized and live in the sort.jl file or if it's special-cased enough to just stay here. It's not really Datetime specific. It could probably also use a better name.

[milktrader]

Watching this development. Great stuff!

[quinnj]

Yeah, maybe someone else can opine as to the TmStruct stuff. My feeling is we'd probably be ok getting rid of it, but I'm really not sure if it's used by anyone or not, or how a deprecation would work there. The one use I see is if people are specifically looking for the standard C <time.h> way of dealing with time, with the strftime/strptime functions, etc. Then it could maybe not be exported or at least the help text could specify that they just wrap the C API. Then the other question is if we should have any additional support for TmStruct/Datetime conversion as proposed by @catawbasam in quinnj/Datetime.jl#23.

Sorry to be a bit rambling. My vote would be to remove the TmStruct type and related functions and if someone so desires, a CTime.jl package could house that code (and probably flesh it out some more).

[StefanKarpinski]

Can we keep strptime/strftime and have them return/take DateTime objects instead of TmStruct objects? That way the interface remains familiar and it's easy to port C code, but most of the cruft goes away.

[quinnj]

Yeah, we would have to keep the TmStruct, as that's what strptime/strftime take as inputs, but we could unexport it and have the interface be through Datetime types. Then we could merge @catawbasam's code mentioned above which basically provides the conversions.

[StefanKarpinski]

Wait, can't we just simulate the behavior of strptime and strftime instead of calling the C versions?

[quinnj]

Definitely. If you look down around line 558 or so, the interface is Datetime(datestring::String,formatstring::String) where the format string is similar to strptime. There isn't a strftime quite yet, but I don't think that's quite as useful as strptime anyway.

[quinnj]

Ok, I've set up the TimeZone package here: https://github.com/karbarcca/TimeZone.jl

Don't expect it to work quite right yet though. Still some bugs to iron out.

[nalimilan]

I'm not sure I follow. Here are two simple use cases: how would you handle them?

• I have parsed DateTime data in my local time zone from a file, and I want to take subset of them, let's say those before a given point in time. How can I create a DateTime object by giving it the local time and let it translate to UTC, so that I can compare it with the parsed data? AFAICT Datetime(2014,2,6,12,30) assumes I'm passing UTC time, but I don't want to compute it by hand from the local time.
• I have parsed DateTime data in my local time zone from a file, and I want to compute daily averages. How can I ensure the averages are done over days in local time and not over UTC time? Local office hours make a big difference so the time of day where you split the data it important.

In these two very common cases, it seems to me that you need both the UTC time and the time zone information. Am I missing something?

[quinnj]

@nalimilan, for the first case, if you were in the U.S. Eastern Time Zone, for example, you input as DateTime(2014,2,6,12,30,0,New_York). This takes the time local to you, 2014-02-06T12:30:00 EST, and will convert, store, and show you 2014-02-06T17:30:00 UTC. You're then free to compare this UTC DateTime with other UTC DateTimes. So you can input a DateTime from any time zone and the conversion using the appropriate daylight savings and offset rules will be used automatically to convert to UTC.

In the 2nd case, there are a few different options, one of which would be use of the ZonedDateTime I mentioned above. Personally, I just input local date time's as UTC and use the type as normal. This kind of usage fits what Noda/Joda call "unspecified" DateTimes, meaning when I do calculations, I don't really care what time zone it is, normally because I'm not doing any inter-time zone conversions or necessarily worrying about daylight savings transitions.

[quinnj]

Ok, I've pushed all changes I've talked about and finished the documentation (I ended up just doing a module page similar to Profile since you have to use Using Time to access the functions).

I'd appreciate any other code review. @StefanKarpinski, @JeffBezanson

[jakebolewski]

Maybe a more descriptive error message? "Month out of range (1:12)"
Also, give the recent discussions about better error handling, throw an ArgumentError instead of error?

[quinnj]

That sounds like a good idea. I haven't stayed completely up to date on the latest, but I'll see what I can push.

[quinnj]

Ok, just pushed changes.

[nalimilan]

@karbarcca Thanks, this makes sense for the first case.

For the second case, everything depends on how the date/time data with time zone information (like "2014-02-06T12:30:00 EST") is parsed by default (e.g. when importing a CSV file or data from the Web). Discarding time zone information and considering it as "unspecified" doesn't sound like a very good default (and that doesn't work if the input times are using several time zones). So the options are converting to UTC (but then you don't allow to user to find out what was the original time zone, so you cannot aggregate by local day), or use a ZonedDateTime object.

My point is, I think you will have to bite the bullet and support time zones (including arithmetic in local time) out of the box anyway at some point. From a quick inspection, Joda time, Numpy datetime64 and lubridate all store time zone information in DateTime objects. Only JSR-310 apparently distinguishes LocalDateTime (without time zone info) and ZonedDateTime. Do you know why JSR-310 made this choice? Do you think the first three are wrong?

[StefanKarpinski]

@nalimilan – I kind of suspect this may be the case. Let me lay out a philosophical perspective here.

It makes sense to talk about a point in time without any notion of timezone. Let's call such a thing a Timestamp. This is a representation of time that should be completely divorced from such things as days, weeks, points on the earth, etc. It is a bare, raw measurement of time and it makes just as much sense anywhere in the universe as it does on this planet. The fact that we represent it in terms of fractions of seconds is incidental; if we could represent all real numbers with perfect precision, the units of storage for a Timestamp would not matter at all – only roundoff error makes that choice significant. It makes no sense for a Timestamp to have a timezone – bundling a timezone with a Timestamp makes as little sense as bundling a measurement of distance with the timezone of the place you measured that distance – i.e. none. A meter is a meter, regardless of where it was measured. Likewise, relativity aside, a point in time is a point in time.

A particular time of day during a particular date at a particular location on the planet Earth is quite a different concept. That is, of course, inherently linked with a location. For convenience, we quantize those locations into timezones. Whenever we want to ask about the hour of the day or the day or the week when something occurred, we need to know what timezone we're talking about because without a timezone, those questions don't make sense. Let's call such a thing a DateTime. The very name implies that it has something to do with Earth and its days and thus entails some location information. It's important to note that a Timestamp is quite different than a DateTime that happens to be in UTC. The notion of a Timestamp is more fundamental: it makes sense anywhere in the universe, whereas explaining UTC requires bringing Earth and its rotation and timezones into the explanation. The fact that we may want to print Timestamp values in UTC is a red herring. That's just because we, as humans, are not very good at looking a large raw number of units of time and interpreting that number. The fact that we may want to represent a Timestamp as fractions of a second since a particular moment in time in the inertial reference frame of Greenwich, England is also a red herring. Yes, that implies that the transformation between a Timestamp value and a DateTime in UTC is the identify function, but that doesn't mean that they're the same concept. Timestamp is a more fundamental concept and needs to e separate from DateTime – it is not the same as "a DateTime that happens to be in UTC."

In particular, the kinds of operations you want to do with time fall pretty clearly into those that belong to Timestamp objects and those that belong to DateTime objects. Interval arithmetic with points in time falls clearly in the domain of Timestamp. "What day of the week did this occur on?" is clearly in the domain of DateTime values. I don't really think this is fuzzy at all – all operations split neatly between these two domains. I also believe that there is a great deal of value in clearly conceptually separating them. For convenience, it may make sense to automatically translate between the two – e.g. wrapping an underlying Timestamp in a DateTime object associated with the local timezone so that we can see it in our local time as we're accustomed to. Or allowing the subtraction of DateTime objects in different timezones to automatically perform a subtraction of the underlying Timestamp objects. Moreover, this separation can sometimes be used to force the programmer to be more careful about how they do things. If they are required to explicitly provide a timezone to turn a Timestamp into a DateTime, then it becomes immediately obvious where code that asks things like "what day of the week" depends on assumptions about the timezone. If a Timestamp were a DateTime in UTC, on the other hand, this would implicitly be tied to Greenwich, England, which might not be the right timezone at all.

Regarding representing the timezone of a DateTime as part of its type versus part of its value, code generation issues aside, I think it's much better to represent it as data and not as part of the type. Let's say you're storing a vector (or data frame column, or whatever) of DateTime values. The two cases to consider are: (a) they are all in the same timezone, or (b) they are in different timezones. In the "same timezone" case, I would argue that you should be storing Timestamp values instead of DateTime values. We already know this is best practice for e.g. recording events in a database. If you want to present the values in a particular timezone, that's fine – just keep that timezone as a single value somewhere and combine it with each Timestamp when presenting it. That way presenting the values in any timezone is just as easy – just choose a different timezone for presentation. The "different timezone" case is another matter – here you want to know what the local time was when something happened, which requires knowing not only when it happened absolutely, but also approximately where, down to the granularity of timezones. Representing this with the timezone as data requires storing at most an additional four bytes per value. Representing this with the timezone as part of the type means that DateTime is an abstract type, so the vector must be an array of pointers to individually heap-allocated, boxed DateTime objects. I think you can see where this is going. Total storage per value:

• timezone as data: 1 timestamp + 1 timezone = 12 bytes
• timezone as type: 1 pointer + 1 type tag + 1 timestamp = 3*8 bytes = 24 bytes.

Using the type to represent the timezone takes twice as much storage, and of course completely destroys data locality and makes it a nightmare to pass the data to another programming language system. And then there are the problems it causes for code generation.

tl;dr: There should be two separate levels of time values – Timestamp values, which are universal points in time (despite the incidental details of how they are represented), and DateTime values, which are composites, each combining a Timestamp with a Timezone, thereby representing a particular time on a particular date in a particular timezone. There should be some automatic conversion between these levels – especially in the direction of dropping the Timezone from a DateTime to get a bare Timestamp, e.g. for interval arithmetic – but this convenience should not be taken too far. Such conversions can make dangerous assumptions about location, which are better left explicit.

[quinnj]

@nalimilan, I'm not saying these other implementations are necessarily wrong, but I did point you to several of these package's author's posts about how uncomfortable/unsatisfied they are with how these things are dealt with, even in their own packages. The point is, handling timezones and operations you may want to perform with them is really tricky business and there is far from consensus with how to do it. Most packages I've seen basically take the approach of providing an 80% solution. Sure, you can create a DateTime object with a timezone, but the implementation isn't going to guard you against some of the gotchas I mentioned earlier or perhaps not even provide defined behavior. I happen to think this is unfortunate and really hope to find a way to do better, without sacrificing simplicity and performance. I've based all this work on these packages, so I'm hoping we're headed in the right direction!

For now though, instead of putting a half-baked ZonedDateTime out in the wild, I really like the idea of providing just enough timezone functionality to cover probably 75% of use-cases for timezones, and as more people bring concrete, real live use-cases that aren't covered, that can help push further development. So we're taking the strategy of starting small, and adding as we need to.

I think we really achieve a great balance of simplicity and conceptual correctness with the current implementation. From @StefanKarpinski's great philosophical layout, you can see the underlying Timestamp of any DateTime by accessing it's only field DateTime(2014,2,9).instant, and all DateTime types in Base will have the nice, clean, well-defined behavior of the human form of time in UTC (which, though rooted in GMT and Greenwich, England, is now considered conceptually distinct and represents a "universal" timezone of sorts).

[nalimilan]

@StefanKarpinski I compeltely agree with you points (except that Instant sounds like a more intuitive term for Timestamp). Indeed I had missed the performance problem with using different types to mark different time zones. Better store this in DateTime objects themselves.

@karbarcca It sounds indeed very reasonable to implement the most important features first, and leave the hardest cases for later. But what I don't understand is why you don't make DateTime include time zone information (of course defaulting to UTC). When using UTC, you would get the reliable behavior; when using a custom time zone, you'd get what is currently supported, and exceptions for what isn't. Or is the long-term goal to merge ZonedDateTime into DateTime? If so, then I've no objection.

As I said above, to me, even more important than arithmetic (which is clearly the tricky part), is the ability to parse (or create from Julia code) a date which contains time zone information, and to present it to the user (print, plot...) in the same time zone by default, because that's what makes sense to people. You don't need to support anything fancy for that: you may even require people to explicitly convert their DateTime objects to UTC if they want to do arithmetic until you know how to implement local time arithmetic correctly.

Anyway, you seem to master these things, so I'm not really worried. ;-)

[nolta]

Since you're converting UTC -> atomic time here, you should throw an error if the conversion is undefined, i.e., before 1972 or after the latest Bulletin C expires (currently 31 Dec 2014).

Although technically i suppose you could extend back to 1961, but that's a bit more complex.

[quinnj]

I'm not sure I follow you. Why is the conversion undefined? The date algorithms are based on rata die days/milliseconds and the proleptic gregorian calendar, so the conversions between date representations and "instants" is compatible from typemin(Date) to typemax(Date).

[quinnj]

And to further explain myself, the idea of the proleptic gregorian calendar, is that we take the current gregorian and apply it retroactively through time. This is spelled out clearly in the docs that we aren't messing with missing days in 1582 or anything when calendars were switched.

[nolta]

Proleptic gregorian exists, but proleptic UTC doesn't. You'd have to go back and define a fake schedule of leap seconds.

[quinnj]

I think it's silly to throw an error for dates before 1972. We could display 1970-01-01T00:00:00 GMT for dates before 1972, or even not display any abbreviation, but that all seems a little far-reaching. My vote is that we use proleptic UTC. I also don't see any reason to mess with things in the future. If using UTC makes anyone feel to0 queasy or uneasy, then we can change everything to UT or GMT to denote that we're using a sensible default.

[nolta]

That's not really relevant. The problem isn't with the time, it's with the conversion to a single number of SI seconds. Since we're not doing the conversion correctly, we should just be honest about it.

[quinnj]

Sorry, I'm not following you, you'll have to explain yourself in more detail. From what I understand, the current implementation is following ISO 8601 standards, using the proleptic gregorian calendar and proleptic UTC (if you will) to represent instants of time retroactively. I don't see where we're being dishonest if we tell people that's the implementation.

[quinnj]

I dug into this a little more and it turns out this is pretty much the behavior everyone else follows as well (proleptic UTC). Checking Joda, Noda, and lubridate all showed they use UTC, even for pre-1972 dates.

[nolta]

No worries, this stuff is a bit complicated.

Roughly speaking, there are two major time standards: UT & TT. UT is "astronomer" time, based on the rotation of the earth. TT is "physicist" time, based on ticking clocks. They have different units: a UT second is really an angle, and a TT second is the usual SI second.

UTC is a clever attempt to bridge these two systems.

When Joda says "leap seconds are not supported", what they mean is "we implement UT".

When you say "leap seconds are supported", what you mean is "we implement TT". As a result, you can define

GPSTime(t::UTCDateTime) = t.instant.ms - 62451648009000

Passing a timestamp like "1939-10-28 09:00:00 GMT" to Joda works because GMT is part of the UT family. Passing the same timestamp to your code requires a conversion from UT -> TT. This correction is called ΔT, and it isn't being applied.

[quinnj]

So what would we need to change? It seems like we just need to further document the kind of construction/conversion we're doing so that if people are really interested, they can find out what exactly we're supporting. From a routine user's perspective though, I don't think they're going to think twice that we call our DateTimes UTC, support leap seconds, and inputting dates pre 1972 isn't anything different. Again, I'm definitely open to changing the default abbreviation here if that simplifies things. Or not have an abbreviation at all since it seems like it could lead to confusion. Maybe that's best: no abbreviation and we clearly document how construction/conversion works.

[nalimilan]

A small possible enhancement: why not have a constant years equal to Year(1), and the same for all periods? This would allow writing the quite neat 1 years + 2 months + 4 days.

[ivarne]

That looks kind of nice, but I would object to reserving such useful variable names for this kind of pretty syntax. You could hide them in a module that the user more might import with using, or just list the 6 const declarations in the documentation for users to copy into their code when they need it.

[quinnj]

Hmmm.......that is pretty nice, but I think agree with @ivarne that those are valuable variable names to take and pretty trivial to define on your own. The constructors for periods used to be years(1); months(1), but I actually changed them to be more in line with the thinking in #1470 (i.e. always use Proper-case names for type constructors). While this technically isn't constructing the Period types, I think it'd still be better to leave as is in Base.

[milktrader]

Agreed those const are cool but should probably be implemented in packages

[quinnj]

Closing for now.

[milktrader]

What happened? It doesn't appear that this was merged.

[quinnj]

Stefan said he'd like to take some more time to look things over. In the mean time, I've cleaned up some more code and it's currently available on the dev branch of the Datetime package. I'll bump it to master in a week or so with the name change of the package to Dates.jl. In the mean time, you can get the new code by

Pkg.add("Datetime")
Pkg.checkout("Datetime","dev")
include(Pkg.dir() * "/Datetime/src/Dates.jl")
using Dates

My thinking is also evolving with regards to what constitutes "Base" code vs. default packages. See this issue for more discussion. The main idea is that code like DateTime functionality wouldn't be in "Base", but could be included as a default package shipped with the main julia distro, but it's easier for people who don't need the functionality to deselect or exclude the code they don't need by creating their own distro with just the default packages they need (probably not as big a deal with this package because there are no binary deps or non-julia code involved, but much more useful for openblas code, fftw, mpfr, etc.).