EDTF dates in BibLaTeX #590

JohnLukeBentley · 2016-10-24T12:54:10Z

New support for Extended Date/Time Format in Biblatex

Biblatex has recently added support for Extended Date/Time Format 1.0, conforming to levels 0 and 1. In brief, this includes, among other formats:

Added support for times. e.g. So one could identify a blog post as having (in Biblatex format) date={2016-07-18T20:26:06+10:00}, date=2016-07-18T20:26:06Z, or date={2016-07-18T20:26:06}
Negative date (occurring in the BCE/BC era). E.g. date={-0379}, which could output in biblatex, depending on the user chosen options, as "0380 BCE". Note the difference of 1 due to differences between astronomical V non-astronomical calendars.
Approximate dates. E.g. date={1567~}, which could output in biblatex, depending on the user chosen options, to "ca. 1567"
Uncertain dates. E.g. date={1567?}, which could output in biblatex, depending on the user chosen options, to "1567?"

For all the details, in one's latex install see:

The documentation biblatex.pdf p37, Database Guide > Usage Notes > Date and Time Specifications; and
latex/biblatex/examples/96-dates.tex ... for examples.

This is available in the production version of biblatex, 3.6. However, there are few bugs there so it might be easier to hold off until 3.7 is released before having a thorough look around.

I note also "PSA: Better BibTeX does not currently support Zotero 5.0 #555" which means you are in a state of flux ... and probably not wanting to write new code until Zotero 5.0 issues stablise.

Also perhaps it will be a rather large effort to verify zotero-better-bibtex properly handles exporting all the EDTF 1.0 date/time formats that biblatex now supports. It might be easier for you if I, and others, flag the relevant export problems one at a time.

That is, even if you want to delay attending to specific zotero-better-bibtex issues, in virtue of doing a major rewrite first, it might be better to feed you these issues one at a time. (Indeed you might prefer to attend to outstanding zotero-better-bibtex issues as you do your major rewrite).

Support times in the Zotero date field.

Consequently, the first problem I'm encountering is ...

Exporting a Zotero entry with a Zotero date in various extended formats (e.g. 2016-07-18T20:26:06+10:00, or 2016-07-18T20:26:06) produces unpredictable results. This is using "Better Biblatex" (not tested in "Better Bibtex").

2016-07-18T20:26:06+10:00

From ...

% Zotero
% Date: 2016-07-18T20:26:06+10:00

I get ...

% Biblatex entry via "Better Biblatex"
@article{barker_2016_turkey,
  author = {Barker, Anne},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  ...
  year = {2016-07-18T20:26:06+10:00},

I expect ...

@article{barker_2016_turkey,
  author = {Barker, Anne},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  ...
  date= {2016-07-18T20:26:06+10:00},

That is, the problem is exporting a date with time and timezone results in the date going to year rather than date.

2016-07-18T20:26:06

From ...

% Zotero
% Date: 2016-07-18T20:26:06

I get ...

@article{barker_2016_turkey,
  author = {Barker, Anne},
  date = {2016-07-18},

I expect ...

@article{barker_2016_turkey,
  author = {Barker, Anne},
  date = {2016-07-18T20:26:06},

That is, the problem is exporting a date with time and no timezone results in the date going to date, as expected, but the datetime value has the time portion lost.

2016-07-18T20:26:06Z

From ...

% Zotero
% Date: 2016-07-18T20:26:06Z

I get ...

@article{barker_2016_turkey,
  author = {Barker, Anne},
  date = {2016-07-18},

I expect

  author = {Barker, Anne},
  date = {2016-07-18T20:26:06Z},

That is, the problem is exporting a date with time and "Z" (for Zulu) timezone results in the time portion being lost.

I've sent the full entry to you as ReportID: IRFDJ8NT.

The text was updated successfully, but these errors were encountered:

retorquere · 2016-10-24T23:10:57Z

this one was fairly easy, for that reference I now get

@article{barker_2016_turkey,
  abstract = {Religious fundamentalism is a powerful force in Turkey where the military has a long history of intervening in politics to ensure the nation remains secular.},
  author = {Barker, Anne},
  date = {2016-07-18T20:26:06},
  journaltitle = {ABC News},
  langid = {australian},
  note = {Actual: 2016-07-18T20:26:06+10:00},
  rights = {http://www.abc.net.au/conditions.htm\#UseOfContent},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  url = {http://www.abc.net.au/news/2016-07-18/turkey-coup-attempt-shows-division-over-wish-for-islamist-rule/7639292},
  urldate = {2016-07-24}
}

JohnLukeBentley · 2016-10-25T03:36:27Z

Thanks! I'm glad this one was easy.

However, I'm not seeing the fix at my end. I get the same faulty results for the same inputs.

I suspect either my install is not properly recognizing the updated zotero-better-bibtex plugin or I'm misunderstanding something about updating it.

To update I:

Downloaded zotero-better-bibtex-1.6.78.xpi https://github.com/retorquere/zotero-better-bibtex/releases/latest
In Zotero > Tools > Addins > Cog icon > Install Add-on from File ... [Selected zotero-better-bibtex-1.6.78.xp]
Verified that the "Last Updated" Value is "Tuesday, 25 October 2016".

Perhaps you could increment the version number at your end (e.g. to 1.6.79)?

Or perhaps I'm failing to properly recognize how GitHub works. Specifically perhaps I should read

retorquere released this 4 days ago · 2 commits to master since this release
( From https://github.com/retorquere/zotero-better-bibtex/releases/tag/1.6.78)

... as meaning ...

1.6.78 doesn't contain the subsequent commits

? In which case I have to wait for your next release?

retorquere · 2016-10-25T06:55:42Z

You will either have to wait for the next release (which I hope will get done in a few days, I'm looking to include #585), or use https://github.com/retorquere/zotero-better-bibtex/releases/download/builds/zotero-better-bibtex-1.6.78-circle-3162.xpi in the interim.

JohnLukeBentley · 2016-10-25T07:06:42Z

zotero-better-bibtex-1.6.78-circle-3162.xpi fails to correct the errors. Given you have the hood open for #585 ... take all the time you need and I'll just wait for the next release. There's certainly no urgency from my end.

retorquere · 2016-10-25T07:12:51Z

Not sure which errors you mean -- given the reference you submitted in IRFDJ8NT, I get

@article{barker_2016_turkey,
  abstract = {Religious fundamentalism is a powerful force in Turkey where the military has a long history of intervening in politics to ensure the nation remains secular.},
  author = {Barker, Anne},
  date = {2016-07-18T20:26:06},
  journaltitle = {ABC News},
  langid = {australian},
  note = {Actual: 2016-07-18T20:26:06+10:00},
  rights = {http://www.abc.net.au/conditions.htm\#UseOfContent},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  url = {http://www.abc.net.au/news/2016-07-18/turkey-coup-attempt-shows-division-over-wish-for-islamist-rule/7639292},
  urldate = {2016-07-24}
}

If there are more errors, references that yield these errors would be appreciated.

retorquere · 2016-10-25T07:18:25Z

I'd actually rather bundle multiple feature requests/bug fixes in a single release rather than releasing a new version for each separately.

JohnLukeBentley · 2016-10-25T08:14:28Z

Not sure which errors you mean

Each of the three errors I mentioned remain. That is, using the entry which you've quoted back to me and trying each of following values, in turn, in Zotero Date:

2016-07-18T20:26:06+10:00
2016-07-18T20:26:06
2016-07-18T20:26:06Z

So, sticking with example you've quoted back to me (Input date 2.), when I use Zotero Date: 2016-07-18T20:26:06 I get date = {2016-07-18}, as originally detailed. This is via Better Biblatex Quick Copy (Ctrl + Shift + C).

Given, however, you see the change at your end, and I don't at my end, I'm suspecting:

Some kind of install issue at my end. However,
In Zotero > Tools > Add-ons > Extensions ... I see "Zotero Better Bib(La)Tex 1.6.78-circle-3162" (when I click on "more" to expand the entry for BBT);
And I've tested this on a separate machine which has never before had Zotero Better Bib(La)Tex installed.

or

The changes you've made didn't compile into 1.6.78-circle-3162; or
The issues you are experiencing with Error in automatic export #585 (I've only skimmed that thread) are repeating themselves here: some kind of platform funkiness. You wrote in Error in automatic export #585 ...

but it'd be super nice if this wasn't in some way a platform dependency. I don't have a windows system at hand, and debugging on windows is a major PITA in any case.

... I'm on Windows 10.
or
Some other bizarre thing.

I'm using Zotero 4.0, not Zotero 5.0.

Perhaps a third party could be kind enough to tell us what they see at their end (and what OS they are using)?

I'd actually rather bundle multiple feature requests/bug fixes in a single release rather than releasing a new version for each separately.

Yes indeed: that'd be the way to do it. Maybe you'd rather close #585 first (among other issues), and that'll automatically help here. But if there is any test I can perform to help, let me know.

retorquere · 2016-10-25T08:58:13Z

Ah, yeah, sorry, you need to go into about:config and set biblatexExtendedDateFormat to true to enable the new behavior. Since the support for this in BibLaTeX is so recent I can't make it default to enabled. I'll probably add a preference for this in the preferences pane, but I'm still contemplating whether I should merge it into a single preference with #566 (BBT already has too many preferences).

For those three dates (with the setting on) I get

@article{barker1_2016_turkey,
  abstract = {Religious fundamentalism is a powerful force in Turkey where the military has a long history of intervening in politics to ensure the nation remains secular.},
  author = {Barker1, Anne},
  date = {2016-07-18T20:26:06},
  journaltitle = {ABC News},
  langid = {australian},
  note = {Actual: 2016-07-18T20:26:06+10:00},
  rights = {http://www.abc.net.au/conditions.htm\#UseOfContent},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  url = {http://www.abc.net.au/news/2016-07-18/turkey-coup-attempt-shows-division-over-wish-for-islamist-rule/7639292},
  urldate = {2016-07-24}
}

@article{barker2_2016_turkey,
  abstract = {Religious fundamentalism is a powerful force in Turkey where the military has a long history of intervening in politics to ensure the nation remains secular.},
  author = {Barker2, Anne},
  date = {2016-07-18T20:26:06+10:00},
  journaltitle = {ABC News},
  langid = {australian},
  note = {Actual: 2016-07-18T20:26:06+10:00},
  rights = {http://www.abc.net.au/conditions.htm\#UseOfContent},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  url = {http://www.abc.net.au/news/2016-07-18/turkey-coup-attempt-shows-division-over-wish-for-islamist-rule/7639292},
  urldate = {2016-07-24}
}

@article{barker3_2016_turkey,
  abstract = {Religious fundamentalism is a powerful force in Turkey where the military has a long history of intervening in politics to ensure the nation remains secular.},
  author = {Barker3, Anne},
  date = {2016-07-18T20:26:06Z},
  journaltitle = {ABC News},
  langid = {australian},
  note = {Actual: 2016-07-18T20:26:06+10:00},
  rights = {http://www.abc.net.au/conditions.htm\#UseOfContent},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {Turkey Divided between Secular and {{Islamist}} Rule},
  url = {http://www.abc.net.au/news/2016-07-18/turkey-coup-attempt-shows-division-over-wish-for-islamist-rule/7639292},
  urldate = {2016-07-24}
}

retorquere · 2016-10-25T10:00:08Z

#585 just cleared so I'm prepping for release, running tests once more on each of the 3 branches (#585, #587 and #590) slated for this release before I merge them. It'd be good to have confirmation that the output above is what you want, or whether there are still test cases you'd want to have added.

JohnLukeBentley · 2016-10-25T13:26:21Z

That's solved, glad it was you not me :) (But I'm guilty of something else as I'll explain below).

Yes the three test cases now work at my end.

I ensured both of the following where true ...

extensions.zotero.translators.better-bibtex.biblatexExtendedDateFormat
services.sync.prefs.sync.extensions.zotero.translators.better-bibtex.biblatexExtendedDateFormat

If you are keen to have a crack at the other EDTF formats then: the EDTF 1.0 (level 0 and level 1) formats that biblatex supports include the following (I've essentially taken a representative sample from biblatex.pdf p37, Database Guide > Usage Notes > Date and Time Specifications):

Negative date: -0876
Uncertain: 1723~
Approximate: 1723?
Uncertain and approximate: 1723?~ (Note the other way around "1723~?" happens to be illegal under EDTF 1.0).
Date range: 1988/1992
Season: 2004-22 (The "month" position should allow 21 to 24 for Spring, Summer, Autumn, Winter respectively).
Unspecifieds
- 199u
- 19uu
- 1999-uu
- 1999-01-uu
- 1999-uu-uu

Other notes:

The above is a mere representation. Those marks [?, ~~, /, etc] can be combined with full dates and datetimes, e.g. For "1995-03-30/1995-04-05". And they can be combined with each other, E.g. "1934~~/1936~"; or "-0379~".
A year must be four digits. E.g. "0343-02-03" is legal, "343-02-03" is not.

You'd just need to pass these through without processing.

For example, from ...

% Zotero
Date: -0379

.... to ...

 % Biblatex
 date={-0379}

I notice that, at the moment, you decrement the date to arrive at ...

 % Biblatex
 date={-0380}

... which is entirely my fault because of my misleading prior remark ...

which could output in biblatex, depending on the user chosen options, as "0380 BCE". Note the difference of 1 due to differences between astronomical V non-astronomical calendars.

... these decrements, and other processing, will get applied by the biblatex system from raw *.bib entry fields, on their way to becoming a pdf. So we want ...

% Zotero
~~date={-379~}~~
Date: -0379

% Biblatex *.bib file
@book{plato_2004_republic,
  author = {{Plato}},
  ~~date = {-379~},~~
  date = {-0379~},
  title = {Republic},
  .....
}

% Biblatex (biber) processing results in pdf outputs
Plato (ca. 0380 BCE). Republic. Trans. by C. D. C. Reeve. 3rd edition. ...

For testing you could just use any entry (barker_2016_turkey would do) and just change the Zotero Date field every time.

Edit: Changed last example (~~ ... ~~ represent deletions):

Made Zotero example more Zotero like
Added leading zero.

retorquere · 2016-10-25T14:02:15Z

  date = {-877},
  date = {1723~},
  date = {1723?},
  date = {1723~?},
  date = {1988/1992},
  date = {2004-22},
  year = {199u}
  year = {19uu}
  year = {1999-uu}
  year = {1999-01-uu}
  year = {1999-uu-uu}

JohnLukeBentley · 2016-10-25T14:22:57Z

"-877" is illegal, "-0877" is legal ... years must have four digits.

I suppose expecting BBT to begin to perform complex validation, which rejecting 3 digit years might become, is probably too much. That might be a bit like performing email address validation on websites.

But at a minimum you want to ensure "-0877" passes through BBT, and should therefore be your test case.
"1723~?" is exactly the wrong way around

Uncertain and approximate: 1723?~ (Note the other way around "1723~?" happens to be illegal under EDTF 1.0).

From Extended Date/Time Format (EDTF) 1.0 the relevant BNF is

UASymbol = ("?" | "~~" | "?~~")

... and I learnt of this only because Biblatex enforces it.

retorquere · 2016-10-25T14:39:26Z

  date = {2016-07-18T20:26:06},
  date = {2016-07-18T20:26:06+10:00},
  date = {2016-07-18T20:26:06Z},
  date = {-0877},
  date = {1723~},
  date = {1723?},
  date = {1723?~},
  date = {1988/1992},
  date = {2004-22},
  year = {199u}
  year = {19uu}
  year = {1999-uu}
  year = {1999-01-uu}
  year = {1999-uu-uu}

retorquere · 2016-10-25T15:45:41Z

Got it -- good spot, that negative-date hack was specific to CSL (#360) but it leaked into the BibTeX export. That date is now date = {-0876}

retorquere · 2016-10-25T15:49:28Z

If date={-0379} and 0380 BCE are equivalent, I'm not adding an option for that. BBT has too many options as-is.

retorquere · 2016-10-25T16:25:34Z

BTW this is the last issue to close before I cut a release.

retorquere · 2016-10-25T16:29:00Z

(and email validation is actually not as hard as it's made to look - the rules on domain names are fairly rigid, and you basically should allow everything before the "@". Anything you forbid there is usually wrongly rejected)

JohnLukeBentley · 2016-10-26T03:33:58Z

In short, you can close the issue and cut your release.

If date={-0379} and 0380 BCE are equivalent, I'm not adding an option for that.

They are equivalent but you shouldn't allow, or at least not add an option for, "0380 BCE" in BBT for the reasons:

That it is illegal under EDTF 1.0;
And therefore it is illegal under Biblatex, as an input format.

... in addition to whatever other reasons you can point to.

The point of my example was for a related but separate purpose: to show that you shouldn't convert "-0379". So, yes, in addition to not converting "-0379" you don't need to allow for "0380 BCE".

All your test cases look good (and indeed it's a good idea to include the original three datetime formats).

If BBT passes those test cases I think that gets the lion's share of the EDTF formats that biblatex supports. That is, if you are agreeable I'm happy to consider the issue currently closed, and you can go ahead and cut your release, for the sake of "release early and often". That is, at a future point I probably will be able to identify some other test cases for a future release (but if you'd rather have me spend more time on the issue now, I can).

On email validation, to the limited extent I grasp the issue, I'm under the impression many webdevs attempt a rigorous validation of addresses, trying to implement the relevant RFC's, and frequently get their implementation wrong: wrongly rejecting valid addresses (as you allude to).

Even if I'm mistaken about that I think you get my points for our purpose here: the (strict) validation of EDTF date formats is complex; biblatex undertakes this validation; if you where to attempt that validation it would be a significant task; and, given, the significance of the task there's a liability to get a mismatch between what BBT accepts and what biblatex accepts.

The alternative, which I'm under the impression you've implemented, is to have a very permissive validation. So you, correct me if I'm wrong, essentially allow anything through that parses as a (edit: iso) date once you allow for the exceptional characters I've brought to your attention (~,?,/,u etc), with some additional exceptions. That is, your validation is basic rather than strict, to avoiding wrongly rejecting something (edit: and not being too fussed if something illegal under EDTF/Biblatex goes through: biblatex can be left to reject it). If I've broadly understood your approach correctly, then I think that's the right thing for you to do.

retorquere · 2016-10-26T06:59:46Z

If there's an EDTF pattern I don't yet pick up I'll gladly fix it, but my main worry is not about validating input but about recognizing it and transforming it to something that the Bib(La)TeX processor will understand.

I can't be in the validation business because I have no feedback channel. If I establish that your date is b0rked (if it were to say, e.g., that birthday when Jimmy got hammered and tried to kiss a chicken), I still have to do something with that (but in this case it's overridingly likely I'll just say "stuff it" ad put it in the year field like I would with forthcoming) because there's no way¹ for me to tell you about it as the translators are sandboxed and can only write to the file you selected for export.

And EDTF dates are the easy case, because they're well-structured. I also have to deal with dates like September 20, 2006, 05/01/1992, 15 juin 2009, 8. juli 2011, Januar 1945, März 1, 2008, 四月 1, 1976... all of which are actual samples of dates that have found their way into my test cases.

Anyhoo, I've cut the release because it contained some urgent fixes, so we have time for this. If there's patterns I missed, do tell me about them.

What does the u mean? I currently treat the u dates you gave me as invalid which means I'm stuffing in the year field rather than in the date field, as BibLaTeX seems to treat stuff in the year field more liberally and so allows for things like forthcoming to be in the Zotero date field (although that would be more cleanly achieved with status: forthcoming in the extra field, but if there's something in the date field I don't just want to ignore it).

¹ there actually is a way but the UI would be clunky and everything about the Zotero internals for translators screams "you should most definitely not be doing this"

You are right that there are RFCs that cover email addresses, but a) they're more complex than most webdevs are willing to understand, and b) not every smtp server is strict about enforcing those rules and might accept more than the RFCs would permit -- in which case the user would have an out-of-spec but wholly functional email address. Either you pick a well-established library to do such validation (but many seem to think "simple regex, done"), or you need to be liberal about such matters.

retorquere · 2016-10-26T07:01:53Z

Oh holy crap that u is unspecified... wow dates are gnarly things.

retorquere · 2016-10-26T07:05:20Z

I'm going to find a lib to parse EDTF as a special case. Too many edge cases.

JohnLukeBentley · 2016-10-26T11:05:12Z

Dates are gnarly things and EDTF does well to accommodate the gnarliness, yet be disciplined about it when doing so.

Yes finding a library for EDTF parsing could well ease your implementation burden. There's a few at https://www.loc.gov/standards/datetime/implementations.html , notably there's two javascript libraries. I haven't evaluated these.

If there's an EDTF pattern I don't yet pick up I'll gladly fix it,

Add these to the list ...

% 'unknown' and 'open' are string literals
unknown/2006
2004-06-01/unknown
2004-01-01/open

% Illegal under EDTF but biblatex allows
% it  as a special exception for reasons I
% can't recall (backward compatibility I
% think).
1997/

% Years exceeding four digits
% Not yet implemented by biblatex but it is part of EDTF 1.0 Level 1
% And I'm in the process of persuading the biblatex dev to implement it
% for the sake of completeness.
y170000002 % the year 170000002
y-170000002 % the year -170000002

Looking at the EDTF level 0 and level 1 table of examples and biblatex.pdf I think that exhausts all the types. After I get my hands on your implementation I'll, of course, attempt to break it with edge cases.

Oh holy crap that u is unspecified... [Out of order] ... I currently treat the u dates you gave me as invalid which means I'm stuffing in the year field rather than in the date field,

Yeah we want to just pass these through, unaltered, to date (I'm assuming you mean to flag what you are currently doing as an erroneous artifact of current code).

my main worry is not about validating input but about recognizing it and transforming it to something that the Bib(La)TeX processor will understand.

That helps me see two things.

That we agree that you shouldn't be too worried about (strict) validation of input and the main thing is just ensuring something gets through that biblatex understands; and
I'd failed to recall that Zotero date fields are very permissive and you therefore have to handle exotic date formats, including the colloquial date formats you mention 8. juli 2011, etc. That is, it's not just a matter of you passing through something basically biblatex valid ... but a matter of you sometimes performing transformations (e.g. From 8. juli 2011 to 2011-07-08).

On this second point is there are a Zotero 5.0 issue (I can't yet install it), or should there be one? That is, does Zotero 5.0 treat date fields differently from Zotero 4.0? Should Zotero 5.0 treat date fields differently by, for example, by allowing users to input any format but having Zotero convert the date into EDTF internally (if so this entail us lobbying the Zotero folks) - something which could ease the burden on the BBT side?

I can't be in the validation business because I have no feedback channel. If I establish that your date is b0rked (if it were to say, e.g., that birthday when Jimmy got hammered and tried to kiss a chicken), I still have to do something with that (but in this case it's overridingly likely I'll just say "stuff it" ad put it in the year field like I would with forthcoming) because there's no way1 for me to tell you about it as the translators are sandboxed and can only write to the file you selected for export.

Incidentally why wouldn't you put it in the date field rather than year field? And how about, whatever your target field, you write an error message to the field? ...

% Zotero
Date: that birthday when Jimmy got hammered and tried to kiss a chicken

% ... passing through whatever BBT translator 
% (like Better Biblatex) to end up with something like
date={
      Mate, that's b0rked: I don't even know Jimmy. 
      Source: 'that birthday when Jimmy got hammered and tried to kiss a chicken'
 }

... ?

blip-bloop · 2017-11-07T23:48:50Z

🤖 this is your friendly neighborhood build bot announcing test build 4628 ("fixes #590 for real (forgot to commit files)").

JohnLukeBentley · 2017-11-08T07:21:34Z

Done

rest is the smart thing to do.

Yes indeed. It sounds like we are in opposing hemispheres, timewise, (I'm at +11:00). If so, that neatly means we are tag teaming it.

Those strikethroughs were to make clear where I'd edited my post. Mostly for your benefit in case you read the original, wrong, post before seeing my edits. So, yes, you don't act on the strikethroughs.

Add support for ".." to express an open range as in "../2004-01-01" or "2004-01-01/.." (while also preserving support for "open" and blank in open ranges).

On build 4628 that works at my end!

To do

Another issue/bug. For "unknown" date ranges:
- unknown/2006 wrongly comes through as year = {unknown/2006}. Should be date = {unknown/2006}. Report ID: DI8C2DXH. simpson_cool_book_0092.
- 2004-06-01/unknown wrongly comes through as year = {2004-06-01/unknown}. Should be date = {2004-06-01/unknown}. Report ID: HIEGAV27. simpson_cool_book_0093
I could let the minute-level go through (by testing for EDTF-ness by adding :00 as one of the tests), but that would mean it would just show up in the date field with no mention of the potential problem in the QR.

Yeah, let's do that.

At the moment, with minute-level precision coming through as year = {2017-10-28T04:44}, we'll get a biblatex compile warning like ...

WARN - year field '2017-10-28T04:44' in entry 'greenwald_2017_how' is not an integer - this will probably not sort properly.

... which is a bit cryptic.

By contrast, if you let it go through as date = {2017-10-28T04:44}, then we'll get a better biblatex compile warning ...

WARN - Entry 'greenwald_2017_how' (Biblatex-Tester-EdtfFull-FromZotero.bib): Invalid format '2017-10-28T04:44' of date field 'date' - ignoring.

blip-bloop · 2017-11-08T12:09:33Z

🤖 this is your friendly neighborhood build bot announcing test build 4634 ("minute-level precision for #590").

blip-bloop · 2017-11-08T12:45:48Z

🤖 this is your friendly neighborhood build bot announcing test build 4637 ("value-qr").

retorquere · 2017-11-08T12:53:39Z

Check out the new QR, too.

JohnLukeBentley · 2017-11-09T11:58:57Z

Unknowns now work.
I see the QR. For date = {2017-10-28T14:30}, I get ...

% 'date': not a valid ETDF date

All my EDTF dates now pass (against 4637)!

Maybe the thing to do is merge 590 into the master, and whatever else you think is outstanding for the "Rewrite for Zotero 5" (including #771?), and I'll retest my EDTF against the master. That is, as a test against a release candidate.

That would serve as a final check of my EDTF test dates (noting one's tests are themselves tested). At that point, I'll suggest, I'd post those datetime formats here.

blip-bloop · 2017-11-10T00:17:44Z

🤖 this is your friendly neighborhood build bot announcing test build 4657 ("Merge branch 'master' into 590").

retorquere · 2017-11-10T00:59:20Z

Here you go. This has everything bundled in.

retorquere · 2017-11-10T00:59:51Z

(#771 should be done tomorrow somewhere)

JohnLukeBentley · 2017-11-10T04:49:01Z

Intro

As expected all my tests against zotero-better-bibtex-5.0.32.4657.issue-590.xpi pass. Therefore you can close this issue.

(Repeating earlier information here ...) Biblatex has been supporting the EDTF standard (to "level 1") as hosted at the (US) library of congress (which I'll reference as "EdtfLoc"). Biblatex supports a superset of that standard: a few extra symbols for backward and forward compatibility reasons.

Find below an exhaustive excercise of the datetime formats biblatex supports. At least, that is, as the main format variants go.

Three of the formats will throw up "WARNS" in biblatex:

date = {2017-10-28T14:30}, // Minute level precision. E.g.:

WARN - Entry 'musk_2017_picture' (Biblatex-Tester-EdtfFull-FromZotero.bib): Invalid format '2017-10-28T14:30' of date field 'date' - ignoring
date = {2004-01-01/..}, // EdtfIso201x. simpson_cool_book_0396. E.g.:

WARN - Entry 'simpson_cool_book_0396' (Biblatex-Tester-EdtfFull-FromZotero.bib): Invalid format '2004-01-01/..' of end date field 'date' - ignoring
date = {../2004-01-01}, // EdtfIso201x. simpson_cool_book_0397. E.g.:

WARN - Entry 'simpson_cool_book_0397' (Biblatex-Tester-EdtfFull-FromZotero.bib): Invalid format '../2004-01-01' of date field 'date' - ignoring

In biblatex the minute level precision warning is expected. The double dot ".." warning for open date ranges is a biblatex bug (biblatex.pdf mentions support for this symbol). For BBT purposes those three dateformats are correctly passed through to biblatex.

Let me know if you need further info (e.g. Report IDs against the following formats).

Legend

"EdtfLoc", the draft EDTF specification hosted at the library of congress: http://www.loc.gov/standards/datetime/pre-submission.html. That is, the spec we've been going off.
"EdtfIso201x", the EDTF profile as found in the draft ISO documents: ISO DIS 8601-201x Part 1; and ISO DIS 8601-201x Part 2 (as made available from http://www.loc.gov/standards/datetime/pre-submission.html). The current version of these are dated 2016-10-26. Essentially what matters for EDTF purposes is Part 2 (that is, you could get away with only reading Part 2). Within Part two "Annex C" stipulates the EDTF profile. However, that profile references eariler parts of the Part 2 document.
For my own reference I've included my cite keys (in the format "author_basedate_titleword").

Common

Typical

date = {1995-03-01}, // chalmers_1995_facing

Approximate and Uncertain

date = {1590~}, // Approximate. galilei_1590_motion
date = {1256?},  // Uncertain. aquinas_1256_two
date = {1273?~}, // Approximate and Uncertain. aquinas_1273_compendium

Eras

origdate = {-0379~}, // Before the year zero ("BCE"). plato_2004_republic
origdate = {0125~}, // Not long after the year zero (“CE”). epictetus_1916_enchiridion

There is nothing special about origdate here. You could just as well use date.

Precision finer than a year

Times

date = {2016-07-18T20:26:06}, // Unpsecifed time zone. simpson_cool_book_0074
date = {2016-06-06T18:20:00+10:00}, // Local time zone. barker_2016_swiss
date = {2016-07-18T10:26:06Z}, // Zulu/UTC timezone. simpson_cool_book_0078

Minute level precision - Invalid under EDTF (triggers a ”WARN” under

biblatex)

// Minute level precision. 
// EDTF and Biblatex Invalid. musk_2017_picture
date = {2017-10-28T14:30},

Dates

date = {2008-01}, // Month precision. kennett_2008_true
date = {2010-01-25}, // Day precision. mendelsohn_2010_enough
date = {2016-24}, // Season precision. sep_2016_stanford

Ranges

Common

date = {1487/1490}, // da_vinci_1487_codex

Open

date = {2004-01-01/open},  // Edtfloc. simpson_cool_book_0096
date = {open/2004-01-01}, // Edtfloc. simpson_cool_book_0097
date = {/2004-01-01}, // Biblatex backwards compatibility. simpson_cool_book_0099
date = {2004-01-01/}, // Biblatex backwards compatibility. simpson_cool_book_0098
date = {2004-01-01/..}, // EdtfIso201x. simpson_cool_book_0396
date = {../2004-01-01}, // EdtfIso201x. simpson_cool_book_0397

Unknown

date = {unknown/2006}, // Edtfloc. simpson_cool_book_0092
date = {2004-06-01/unknown}, // Edtfloc. simpson_cool_book_0093
// Biblatex backwards compatibility. Edtfloc invalid. simpson_cool_book_0094
date = {*/2006}, 
// Biblatex backwards compatibility. Edtfloc invalid. simpson_cool_book_0095
date = {2004-06-01/*},

Unspecified

date = {199u}, // Within a decade. simpson_cool_book_0110
date = {19uu}, // Within a century. simpson_cool_book_0120
date = {1999-uu}, // Within a Year (to month level). simpson_cool_book_0130
date = {1999-01-uu}, // Within in month. simpson_cool_book_0140
date = {1999-uu-uu}, // Wtihin a Year (to day level). simpson_cool_book_0150

Years exceeding four digits

date = {y-51234}, // A long, long time ago. rebels_y-51234_rebel
date = {y52000}, // A long, long time ahead. seldon_y52000_encyclopedia

retorquere · 2017-11-10T07:15:50Z

Holy shitsnacks I saw that wall of text coming on on mobile and I just panicked there were 10 more edge cases to deal with 😆. Thanks man, merged, two more to go before release.

JohnLukeBentley · 2017-11-10T09:44:07Z

Thanks for all that EDTF coding. That really opens up the gates for where I want to go.

Incidentally, "wall of text" does not mean "lots of text" ...

https://www.urbandictionary.com/define.php?term=wall%20of%20text

A piece of writing that does not use proper grammar and generally looks like a giant essay with 20 to 400 sentences without using paragraphs or any bit of spacing at all.

... I look forward to the release and hoping that'll afford you a break, or at least allow you to spend far less time on BBT.

retorquere · 2017-11-10T09:54:25Z

I stand corrected on the wall of text. In my defense, it looked a little packed on my mobile mail preview 😟.

A break would be nice, but I'm in the process of putting together a PhD proposal, and that process, as I'll just blindly guess you know, allows for no breaks.

incidentally, @blip-bloop should now just ignore closed issues. Stay, @blip-bloop.

github-actions · 2021-02-17T21:02:23Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

retorquere added the enhancement label Oct 24, 2016

retorquere changed the title ~~Handling exports in virtue of new EDTF support in Biblatex. Exporting the date field when it includes a time value.~~ EDTF dates in BibLaTeX Oct 24, 2016

retorquere added a commit that referenced this issue Oct 24, 2016

pref for #590

bfe4ff0

retorquere added a commit that referenced this issue Oct 25, 2016

more test cases for #590

61e0ea3

retorquere added a commit that referenced this issue Oct 25, 2016

more test cases for #590

e5fe793

retorquere added a commit that referenced this issue Oct 25, 2016

#590 picky about dates

526e419

retorquere mentioned this issue Oct 25, 2016

Empty entries in the collection context menu after installing BBT #592

Closed

retorquere added a commit that referenced this issue Nov 7, 2017

new testcases for #590

a3cdffc

retorquere added a commit that referenced this issue Nov 7, 2017

fixes #590 for real (forgot to commit files)

3d55e3d

blip-bloop removed the awaiting-user-feedback label Nov 8, 2017

retorquere added a commit that referenced this issue Nov 8, 2017

two more testcases for #590

16f1795

retorquere added a commit that referenced this issue Nov 8, 2017

minute-level precision for #590

f295d12

retorquere added a commit that referenced this issue Nov 8, 2017

minute-level precision for #590

288ee0b

blip-bloop added the awaiting-user-feedback label Nov 8, 2017

JohnLukeBentley mentioned this issue Nov 8, 2017

Snapshot urls continue to be exported even after deletion. #771

Closed

blip-bloop removed the awaiting-user-feedback label Nov 9, 2017

blip-bloop added the awaiting-user-feedback label Nov 10, 2017

blip-bloop removed the awaiting-user-feedback label Nov 10, 2017

retorquere closed this as completed in 9878e8c Nov 10, 2017

blip-bloop added the awaiting-user-feedback label Nov 10, 2017

blip-bloop removed the awaiting-user-feedback label Nov 10, 2017

github-actions bot locked as resolved and limited conversation to collaborators Feb 17, 2021

EDTF dates in BibLaTeX #590

EDTF dates in BibLaTeX #590

Comments

JohnLukeBentley commented Oct 24, 2016

New support for Extended Date/Time Format in Biblatex

Support times in the Zotero date field.

2016-07-18T20:26:06+10:00

2016-07-18T20:26:06

2016-07-18T20:26:06Z

retorquere commented Oct 24, 2016

JohnLukeBentley commented Oct 25, 2016

retorquere commented Oct 25, 2016

JohnLukeBentley commented Oct 25, 2016

retorquere commented Oct 25, 2016

retorquere commented Oct 25, 2016

JohnLukeBentley commented Oct 25, 2016

retorquere commented Oct 25, 2016

retorquere commented Oct 25, 2016

JohnLukeBentley commented Oct 25, 2016 • edited Loading

retorquere commented Oct 25, 2016

JohnLukeBentley commented Oct 25, 2016 • edited Loading

retorquere commented Oct 25, 2016

retorquere commented Oct 25, 2016

retorquere commented Oct 25, 2016

retorquere commented Oct 25, 2016

retorquere commented Oct 25, 2016

JohnLukeBentley commented Oct 26, 2016 • edited Loading

retorquere commented Oct 26, 2016

retorquere commented Oct 26, 2016

retorquere commented Oct 26, 2016

JohnLukeBentley commented Oct 26, 2016

blip-bloop commented Nov 7, 2017

JohnLukeBentley commented Nov 8, 2017

Done

To do

blip-bloop commented Nov 8, 2017

blip-bloop commented Nov 8, 2017

retorquere commented Nov 8, 2017

JohnLukeBentley commented Nov 9, 2017

blip-bloop commented Nov 10, 2017

retorquere commented Nov 10, 2017

retorquere commented Nov 10, 2017

JohnLukeBentley commented Nov 10, 2017

Intro

Legend

Common

Typical

Approximate and Uncertain

Eras

Precision finer than a year

Times

Minute level precision - Invalid under EDTF (triggers a ”WARN” under

Dates

Ranges

Common

Open

Unknown

Unspecified

Years exceeding four digits

retorquere commented Nov 10, 2017

JohnLukeBentley commented Nov 10, 2017

retorquere commented Nov 10, 2017

github-actions bot commented Feb 17, 2021

JohnLukeBentley commented Oct 25, 2016 •

edited

Loading

JohnLukeBentley commented Oct 25, 2016 •

edited

Loading

JohnLukeBentley commented Oct 26, 2016 •

edited

Loading