-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clarification of the handling of leap seconds #313
Comments
Dear Tobias @d70-t Thanks for raising this issue. I agree with your idea of clarifying the current convention as a separate issue from adding a convention for leap seconds. As you have no doubt seen, that other discussion #148 was very long and has stalled, because we didn't seem able to understand one another or reach a consensus. But perhaps we can return to it. If you have a definite requirement and use-case that will help. You've labelled this ticket as a defect, and I agree that it arguably is simply an ambiguity or mistake, but in view of the history I don't think we should treat this as a defect issue. That would mean it would get accepted if no-one objected, whereas I think we should make sure there's some positive support, as required for an enhancement. Therefore I'll change the label. I hope that's OK with you. You suggest that this issue applies to all existing calendars but actually I think it only affects the default/ It seems to me that the clearest way to describe this is as an issue affecting the computation of the interval of time between two instants of time each identified by components of date-time (year month day hour minute second). When this calculation is done, it is assumed in CF calendars that 00:00 on any calendar day is 60 seconds after 23:59 on the previous calendar day. As you say, we could simply state that this is the case. It raises questions which logically I feel we should also make clear. However, in view of the previous discussion, I think they might be contentious.
Best wishes Jonathan |
Dear @JonathanGregory, thank you for this motivating reply! I am also fine with handling this as an enhancement. This will add some more pressure towards a good formulation, but that shouldn't be a bad thing 😄 I also have a use case for a new version of CF-Conventions which are able to handle leap seconds, but I want to get this one fixed first. My Idea was to keep it general, such that the sentence covers all the cases. I.e. if a calendar does not know about leap seconds anyways, it does not matter if the "ignore leap seconds" rule is applied or not. We could
If we do calendar-specific wording, we should still ensure that no ambiguity regarding leap seconds may occur for the others. That is,
Regarding the second part of your post on describing this as an issue of the computation of time-deltas. I agree with all what you have written. I would probably rephrase one sentence a little:
into
This will also cover the case in which the leap second is not inserted at midnight of the calendar (i.e. when using time-zone aware reference times). There's just one more subtlety which might or might not be worth noting (and I have great fear while writing it as I really hope that it does not trigger a long discussion):
This might be clear to everyone, but the thing is that in principle there would be another solution to the constraints you mentioned which would be to continue counting the component-wise date time which then would get out of sync with the "real" calendar. (This is what I am currently thinking of creating a term like If you like, I could draft a PR which could be used as a base to talk about the wording. All the best, |
Dear Tobi @d70-t I haven't found any evidence on the web of the Julian calendar using leap seconds, but for the sake of simplicity I agree with your original proposal that for the moment we can make a general statement, rather than calendar-specific ones. It can be revised again if leap-second-aware calendars are included. Like you, I'm nervous of your subtle point about civil time, because it feels like it might be treading on the toes of one of the controversies which prevented us from agreeing about the leap seconds before. However, I'm willing for us to try again! In my opinion, we should be able to avoid problems by clarifying the purpose of calendars as things stand in CF as solely for encoding and decoding date-times. They say nothing about what these date-times mean or might be used for. We would have to revise this interpretation if leap-second-aware calendars are introduced. This is one of the things which caused problems before, if I remember correctly. Section 4.4 on "Time coordinate" starts by defining
I suggest that we elaborate this paragraph, for instance:
After that, we describe the values it can currently have. Issue #148 may modify this set but would not be inconsistent with the above, I believe. What do you think of that? Best wishes Jonathan |
Dear @JonathanGregory, I think this is really good 👍 I especially like that this description has a strong focus on describing the mapping between the tuple of #148 may require a rewrite of the 60 second rule into calendar specific assumptions, but at this point, I am pretty sure that this can be done without contradicting the rule as written in your post (the simplest would be to just copy the paragraph into all existing calenders and replace it by other wordings for new calendars). However, I think this rewrite should not be done now. I'd probably have a few more comments on the exact wording, but these are minor and I think it is easier to do it within a PR review than in an issue-message-stream. Do you agree? All the best, |
Dear Tobi @d70-t Our usual practice is to do as much discussion in the issue as possible. Once there is a PR, it's nonetheless still easier to follow the discussion if comments on it are made in the issue, as that means there's only one place to look. However, if you and I generally agree about the words, and since no-one else has yet expressed reservations, I think it would be fine to have a PR. Would you like to do that? Best wishes Jonathan |
Hello @d70 and @JonathanGregory, I have been hitherto silently following this, and also very much like Jonathan's suggested text (#313 (comment)). I think, in the absence of #148, it removes the current ambiguities and I would also welcome a PR. Thanks, |
Likewise, I have put off commenting, as the two of you have made good progress. Now that things are clearly getting serious, I would suggest to
I'll review the wording once the final proposal is in place, but I think most of the text suggested above is really quite good. thanks to both of you for this contribution. Maybe I'll find time to think again about #148 |
@davidhassell @JonathanGregory The proposed wording does seem to bring clarity to the CF text. However, I have a concern in that it seems to imply that ignoring leap seconds in UTC and the Gregorian calendar is acceptable. I recognise that it may have been, or still is, common practice. But UTC and the Gregorian calendar have leap seconds by definition. |
This is the suggested initial wording from cf-convention#313 as authored by @JonathanGregory.
I've prepared a PR at #316. It mostly contains @JonathanGregory's wording with two small changes identified by separate commits. Feel free to comment on the proposed wording. I wasn't entirely sure about how to handle the remaining things of the release checklist, namely:
Can you provide me some assistance on how this should be done? @taylor13 I agree with your suggestions and hope that the current PR is sufficient in not naming those things. @chris-little I also fully agree that leap seconds should not be ignored, and indeed I have use cases where I really want to represent times within those leap seconds. However an even bigger problem is that I want to unambiguously specify at least most of the date-times. Currently there's obviously some way people handle the conversion of time values into date-times, but it is not written down, so datasets which might cross leap seconds are ambiguous. Regarding this issue, I only see two options to treat the importance of leap seconds:
I strongly favor the second option. And thank you all again for being so positive about this thing 🎉 |
@chris-little said:
Be careful here. The real-world Gregorian calendar does not have leap seconds by definition. In general world usage, "UTC" is a precise timekeeping system which includes leap seconds. "Gregorian" is a system of labeling calendar dates only, and has nothing to do with length of day, or leap seconds in particular. In CF we use the calendar attribute "gregorian" in an extended sense, to identify a particular timekeeping system which combines both the real-world Gregorian calendar, and a fixed length day which does not use leap seconds. I support the general premise of this issue, all currently defined CF calendars have fixed length days and do not include leap seconds. If this notion is generally accepted, then it will become easier to start adding some of the more precise timekeeping systems later. |
I support this proposal and favor pushing forward on #148 rather than adding more about leap seconds here. |
Dear Tobi @d70-t Thanks for preparing the pull request. I notice that you prefer the word "count", in "A time coordinate value is a number which represents a date-time as a count", "the counting unit" (meaning the unit of time in the My idea in drafting that text was that we can avoid the problem which @chris-little raises at this stage by being clear that the Regarding your other questions
Best wishes Jonathan |
Dear @JonathanGregory, yes, I also wasn't entirely happy with the word "count". I assumed that something like a fractional count would not be uncommon, but that might well be due to a lack in my understanding of the english language. I am happy with the term number as well. It think the first instance of Then for the last part we could maybe try something like I'll have a look at the other points. All the best, |
Dear @JonathanGregory, I've added me as an additional author and added the note on the 60th second to the conformance document. Regarding the version, there are sill many occurrences of version 1.8 in the document as well as the conformance document. This is the reason why I didn't check the other items. I could update these, but it doesn't feel like all of these should be part of this PR. Tobi |
Dear Tobi @d70-t OK, thanks. Those changes look fine to me.
Maybe we could just depend on the word "value", which has that meaning.
Yes, I sympathise. I suggest rewriting the start in a different order:
Is that better?
Yes, I see what you mean. Maybe, "it is assumed that the coordinate value increases by exactly 60 seconds from the start of any minute." That sounds more definitely a statement about the numerical value, I think. I'm sure the version number will be updated when the next version is made. Jonathan |
Dear @JonathanGregory, I like your suggestions. I've included them into the PR and am also happy now with the current state of the PR. All the best, |
Dear Tobi @d70-t That looks fine to me. Since @davidhassell @taylor13 @Dave-Allured @sethmcg have also said they're in favour of it, we've got the required amount of support and no objections which haven't been addressed at present. If no-one raises any further concerns within three weeks from now, the proposal will be accepted. Thanks and best wishes Jonathan |
To be consistent with the single occurrence in the existing version of the standard, and also with #315, we should write "date/time" in the new text introduced by this issue, instead of "date-time", which I wrote. I assert that this isn't a material change to the proposal, so we don't need to "reset the clock". This proposal will be accepted on Friday 2nd April if there are no further concerns expressed by then. |
Thanks @JonathanGregory. I agree that this is not a material change and I also don't have a particular preference for either variant. But as you say in #298, "date/time" is what has been in the conventions before. I've updated the PR accordingly. |
Thanks for doing that, Tobi
|
🎉 |
* added example 6.1.2 to the list of examples; fixed cf-convention#284 * updated changes in history.adoc * removed fourth lines of third table in sect 9.3.1; fixed cf-convention#288 * updated history * Bring conformance doc in line with clarification to use of region names/area_types to allow use of flag_values and flag_meanings as per discussion in cf-convention#198 * Add support for variables of type string to conformance doc. See issue cf-conventions#139 * Revert "Bring conformance doc in line with clarification to use of region names/area_types to allow use of flag_values and flag_meanings as per discussion in cf-convention#198" This reverts commit f754457. * first draft of section 5.8 * format typo * rewording * rewording * rewording * New 'Do' Use value, and 'dimensions' entry * Domain construct * rewording * rewording * rewording * formatting of computed_standard_name entry * rewording * rewording * rewording * top-level * rewording * move fig 3 * rewording * span * rewording * data * rewording * rewording * rewording * conformance * recommended attributes * typo * dimensions * dimensions * format * typo * domain independence * domain optional * format * format * format * format * empty dimensions * long_name * UML * Update ch01.adoc * Update history.adoc * Add static assets to HTML check build * Add static assets to Travis upload job * Fix order of i/j in lon/lat bnds figure correct indices of neighbour cells in @d case * update/correct order of indices i/j in Fig 2 (2D lon/lat bounds) * update/correct order of indices i/j in caption of Fig 2 * rename "figure 1" to "figure 3" in Appendix i * correct indices of neighbour cells in @d case * update history Figures are generated from: https://github.com/neumannd/cell_bounds_figures_for_cf_conventions * updates arising from cf-convention#301 up to 2020-09-28 * correct label for 1.2 * format correction * reword empty dimensions example * comma * example links * long_name * formatting * missing 'construct' * term units * term units * standard names * typo * units conformance requirement * remove requirement for identical units * Copyedit * fixed typos * History * more text following 2020-11-27 discussions * bounds * tidy * tidy * tidy * tidy * reproducability * offset * indices * indices * indices * super * tie_point_dimension (1) * tie_point_dimension (2) * tie_point_dimension (3) * tie_point_dimension (4) * tie point * tie_point_dimension (5) * corrected interpolation_configuration description * zone/area rewording * zone/area rewording * multiple mappings * multiple mappings * multiple mappings * typos and some minor rewording suggestions * format * spell check * markup style * example formatting * example formatting * example formatting * example formatting * minor typesetting * interpolation_parameters * interpolation parameters variable dimensions * interpolation parameters variable dimensions * non-standard provision * interpolation parameters variable dimensions * captions, cdl * tidy * minumum size of interpolation zones * Appendix A attributes * interpolation -> sampling * Conformance - first draft * 2nd draft: better descriptions of allowed dimensions * typos * Correct 'is list' to 'is a list' * history cf-convention#304 * check on interpolation zone dimension size * Clarification of the handling of leap seconds This is the suggested initial wording from cf-convention#313 as authored by @JonathanGregory. * leap seconds: added the word "count" in some places The purpose of this change is to slightly highlight the difference between when seconds are used within the coordinate value for counting and the seconds which are part of the date-time. * leap seconds: minor wording extension * leap seconds: added reference to cf-convention#313 to history.adoc * add myself to the end of the list of additional authors * leap seconds: updated conformance text This change excludes values larger or equal to 60 for seconds in reference date-times in time unit attributes. Additionally, the reference time has been changed to reference date-time to agree with the wording in the proposed conventions text. * leap seconds: small rewording as discussed with @JonathanGregory Reasoning: counting may be associated with integral numbers, which is was not intended. We still like the idea of a little more separation between seconds as a unit of the value and seconds as in the date-times. * replace date-time with date/time * conformance changes for new interpolation variable * conformance changes for new interpolation variable * conformance changes for new interpolation variable * conformance changes for new interpolation variable * appendix A changes for new interpolation variable * appendix A changes for new interpolation variable * lat lon tie point definition * spelling * URI -> URL * lower resolution -> sampled * Use on domain variable * typo * Move 'interpolation dimension' definition to first occurence * Minor re-wording * Fix cross-reference * Re-wording * typesetting * tie point index re-wording * Rotation of interpolation axes for two dimensional methods and mino corrections * terminology: interpolation variable and tie point variable * typo * examples in toc * Replace expression for gsqr with equivalent, but numerically more accurate version * Update authors * Update history * Rename attribute tie_points to coordinate_interpolation (Change 2) * Reword section Interpolation and Non-Interpolation Dimensions (Cahnge 10) * Rename tie_point_dimensions attribute to tie_point_mapping (Change 2) * Change term 'tie point variable' to 'tie point coordinate variable' (Change 4) * Reword first paragraph of Section 8 (Change 6) * Remove sentence "This form of compression may also be..." (Change 7) * Update sentence: "A single interpolation dimension may be associated..." (Change 9) * Reword section "Interpolation and non-interpolation dimension" (Change 10) * Improve sentence "An interpolation zone must span at least two points..." (Change 11) * Correct sentence "....must be a subset of zero or more of the ..." (Change 12) * Reword text about the dimensions of interpolation parameter (Change 13) * Improve sentence "The bounds of a tie point must be the same..." (Change 14) * Reduce number of data variables in Example 8.5 (Change 16) * Rename "terms to continuous area" and "interpolation subarea" (Change 5) * Improve wording of "An interpolation subarea must span..." (Change 11) * Remove paragraph "The same interpolation variable may be multiply mapped ...." no longer relevant * Rename terms to: subsampled dimension, interpolated dimension and non-interpolated dimension * Combine the tie_point_dimensions and tie_point_indices attributes (Change 1) * Update figures to match new terms * Improve description of non-overlapping interpolation subareas * Improve description of non-overlapping interpolation subareas * Update Example 8.6 to correctly specify one dimension interpolation for X and Y * Improve wording of Tie Point Index Mapping (Change 8) * Clarify interpolation subarea size * Clarify dimensions in Figure 2 * Add new section 8.3.9, "Computational Precision" * Combine the tie_point_dimensions and tie_point_indices attributes (Change 1) * Remove paragraph "A single interpolated dimension may be associated with multiple ...." no longer relevant * Update ch08.adoc Co-authored-by: David Hassell <[email protected]> * Update ch08.adoc Co-authored-by: David Hassell <[email protected]> * Update ch08.adoc Co-authored-by: David Hassell <[email protected]> * Update ch08.adoc Co-authored-by: David Hassell <[email protected]> * Change sampl... to subsampl... * Rewrite section Interpolation of Cell Boundaries (Change 15) * Constrain interpolation parameters to support bounds interpolation * Update <<link>> names and figure names to new terms * Require tie points to be numeric type and have no missing values * Update Appendix J with new terms and names * Correct spelling mistake in Appendix J * Correct numbering mistake in Appendix J * Change "iz" (interpolation zone) to "is" (interpolation subarea) in App J (Change 3) * Correct "target dimension" to "interpolated dimension" (Change 17) * Introduce section numbering and remove table captions in Appendix J * Include interpolation argument s in figure 1 and 2 * Move Figure 1 and 2 in Appendix J futher down * State tht for linear interpolation, the coordinates of the interpolated points are evenly spaced. * Change "equivalently" to "similarly" in explanation of s1 and s2 in App J * Rename cofficeint "c" to "w" in Appendix J to avoid confusion with point C * Move "Common Conversions and Formulas" in front of "Interpolation Methods" in Appendix J * Add "s" to "each of the interpolated dimension" in Appendix J * Minor wording improvements arising from review * Conformance for bounds tie points * computational_precision conformance Co-authored-by: Daniel Neumann <[email protected]> Co-authored-by: Rosalyn Hatcher <[email protected]> Co-authored-by: JonathanGregory <[email protected]> Co-authored-by: Daniel Lee <[email protected]> Co-authored-by: Daniel Lee <[email protected]> Co-authored-by: David Blodgett <[email protected]> Co-authored-by: AndersMS <[email protected]> Co-authored-by: Tobias Kölling <[email protected]> Co-authored-by: Tobias Kölling <[email protected]>
Title
Clarification of the handling of leap seconds
Moderator
Requirement Summary
In order to correctly compute timestamps of measured variables where the reference date in the
units
attribute and the point in time of the measurement are separated by one or more added (or removed) leap seconds, it is required to know if leap seconds are to be counted or not.Technical Proposal Summary
Add a sentence like the following to §4.4 Time coordinate:
Benefits
Anyone who is working on high-frequency (sub-minute intervals) measurement data will benefit from this clarification. Without this change, time series wich cross an added (or removed) leap second can not be stored unambiguously using CF Conventions.
Furthermore, there there is considerable demand of specifying time in
seconds since 1970-01-01
for historical reasons or other reference date far from the measured point in time. This specification is also not exact if the handling of leap seconds is not defined.Errors due to different handling of leap seconds between data producers and data consumers lead to bugs which are hard to track and annoying. These errors can likely be reduced if handling of leap seconds is specified unambiguously.
Drawbacks
If leap seconds are ignored, measurements which happen exactly on the leap second can not be represented. This however is currently already the case in practice. In order to fix this additional issue, proper handling of leap seconds, TAI clocks and UTC clocks would have to be included. That topic is already being discussed in #148 and thus should not be part of this issue. Once #148 is settled (I hope that will be soon!), the added statement will not be true in general, but will be dependent on the specified
calendar
or other attributes and thus might need to be reworded slightly. I however don't think that this issue will affect the possibility to implement #148 in any negative way.Status Quo
Leap seconds are not mentioned anywhere in CF Conventions. Usual libraries like
cftime
ornumpy
ignore leap seconds, but there's no specification that this library must be used and that what these libraries do is how it is intended to be. Thus for all implementations which I am currently aware of, nothing will change in practice, but discussions about how to implement custom solutions will become much simpler.Associated pull request
#316
Detailed Proposal
(this is the same as above)
Add a sentence like the following to §4.4 Time coordinate:
This issue is intentionally separate from #148. My hope is that this issue can be settled a lot quicker. I also think that these really are two different concerns as #148 would add additional, desirable functionality while this issue only removes ambiguity from the current specification. It is incomplete in the sense of the mentioned drawbacks in order to make the scope smaller and simpler to implement.
The text was updated successfully, but these errors were encountered: