Must navigation order match spine order?

[dauwhe]

There have been several discussions in the EPUBCheck repo about a new validation test, which checks that the order of links in the toc nav element matches the spine order. Existing EPUBs are failing validation because of this new check, although the spec has required this since EPUB 3.0.

EPUB 3.2 spec text::

The references in the toc nav element MUST be ordered such that they reflect both:

• the order of the referenced EPUB Content Documents in the spine;
• the order of the targeted elements within their respective EPUB Content Documents.

EPUB 3.0.1 spec text:

The order of li elements contained within the toc nav element must match the order of the targeted elements within each targeted EPUB Content Document, and must also follow the order of Content Documents in the Rendition's spine.

In w3c/epubcheck#1036 there is a proposal to change this error to a warning. If we are to keep the spec aligned with EPUBCheck, that would mean changing our MUST to a SHOULD.

Should we change the EPUB spec? Is making this a SHOULD good enough?

[GeorgeKerscher]

Hello, For background: Today there are reading systems that do not have a “move to next chapter” Which is really move to next spine item. Edge for example has this. The technique to move to the next chunk of the publication is to go to the navigation area, move down and select the heading, and then go to it. If there are items not in the navdoc, it would be difficult/impossible to get to that unreferenced spine item. Best George

[murata2makoto]

George, What will happen when the last spine item is the first in the nav doc? Takami san described interesting examples, and this is one of them. Regards, Makoto

[damiangza]

Hi

The intention of author/creators order of all the content is described and the spine. On the other hand, ebook reading apps could, for whatever reason, provide the user with a features which might override author's toc, the author might purposefully want to display the chapters/sections/headings in a different order, or multiple toc to provide alternate navigation paths.

Perhaps stipulating SHOULD for toc nav allows for flexibility for authors and reading apps.

[Doktorchen]

About the edge behaviour mentioned by George:
Does this edge (I do not have access to this program) not already fail at some large books without a chapter structure?
Some tools split the content due to file size, but mention only the first part of such a split in the toc at all.

For example I read already a book, created with such a tool.
It has 42 files in the spine, most of them with a file size of about 150 kiB.
But only cover, imprint and the first split document are mentioned in the toc.

If edge has already probiems with such simple books, it may have even much more problems with books with a more complex structure.

[dauwhe]

Would it be acceptable to change this to a SHOULD in EPUB 3, and thus a warning in EPUBCheck?

[dauwhe]

Question: if we adopt SHOULD instead of MUST, do we do it both for links to the same content document, and for different spine items? Is there actually a use case for the primary navigation links to the same content document not being in document order?

[shiestyle]

In a reflowable content, it may be prepared that the NavDoc with chapters first and columns second for the spine item when a long XHTML has some chapters and columns.

For example, it will be the case of integrated books with some volumes.

[avneeshsingh]

The correct implementation in EPUBCheck is the cause of this issue. The EPUBCheck 4.2 was released a month ago and the development will continue through 2019. So, it is OK to relax this rule in EPUBCheck for now i.e. give a warning instead of error and revisit this issue in later part of 2019.

For longer term solution we should work on specifications. Mokoto asked me in DPUB summit, if first spine item is last Nav Doc item how would you feel from accessibility perspective. My reply was that it can drive me crazy. If someone is doing it, I should at least get a warning from reading system that Nav items do not align to spine order. I understand that there may be use cases for this, may the book is in form of a decision tree where path through the book is chosen by the user and it cannot be aligned with spine. Makoto, romain and I talked about different ways to resolve this issue, one way was to have a new property to give warning to the users, and other was to evolve landmarks so that it can be used for the pointers that are not in-sync with spine order.

[mattgarrish]

My reply was that it can drive me crazy. If someone is doing it, I should at least get a warning from reading system that Nav items do not align to spine order.

Not to harp on the distinction here, but this is an accessibility consideration not a technical one. We're making a content issue a normative requirement.

Or can someone explain why ordering is mentioned at all? What is it enabling in the specification?

That's why I feel this belongs in Ace not epubcheck.

[bduga]

Or can someone explain why ordering is mentioned at all? What is it enabling in the specification?

Ordering allows a RS to indicate position in the book. For instance the number of pages in the current chapter/section can be displayed. Additionally, the "current" chapter/section can be indicated in some special way (eg a different color for visually rendered ToCs). In this non-linear table of contents, what is "next"? What is the relationship between the items (before/after, above/below)? It seems like truly non-linear content wouldn't have an extensive ToC (eg a choose your own adventure may just have ["prologue", "start", "afterward"]). For the case mentioned, this doesn't seem to be non-linear, but is rather a legacy from print where dynamic ToCs don't exist. That is, I can't fold chapter 2 open or closed to show or hide the sub columns. But I am not really sure I understand that example - is it really the case where this non-linear toc is better? Can we get a real world example?

[Doktorchen]

bduga: non-linear books from own production (author, coauthor or contributor; language: german), examples:
http://hoffmann.bplaced.net/epub/pug.php#Abstraktiones
(minimalistic global table of content, distributed navigation)
https://www.bookrix.de/_ebook-marie-de-sade-esm4/
(the table of contents is divided in two parts, a minimalistic one as EPUB toc and an independent expanded one to expose the structure of alternatives)
https://www.bookrix.de/_ebook-marie-de-sade-marie-die-gruft/
(list of alternatives and additional SVG as an alternative access)

All examples have either multiple linear reading orders or none, depending on the point of view.

A special case is a literature maze (here a kind of abstract literature, but there could be a similar structure in text adventures like an escape game):
https://www.bookrix.de/_ebook-marie-de-sade-syegih-en/
(English introduction, but abstract content, random order or spine and toc, local navigation)
Because the task is to explore, it would be counterproductive to expose successful ways through the maze with a linear reading order ;o)

[bduga]

I was thinking samples of currently failing content. Do one of those fail due to out of order ToCs? I took a quick look but didn't see it. When it comes to multiple linear orders, that has a long history in epub and was what "tours" were created for. However, there was very little interest in those in practice, so they were dropped years ago. As for essentially no linear order, that would seem to call for a simple ToC with only a "start here" entry, for more complex navigation you would want to use other nav lists (but again, not sure what the support would be for those). Of course, even entirely non-linear content can benefit from a linear ToC (eg a collection of unrelated poetry might still be useful to navigate by poem name).

It seems like the original request from @ShinyaTakami was to flatten and misorder a volume with a single linear order, which is a little different than multiple or no linear orders.

[Doktorchen]

For the third one the check with epubcheck 4.2 fails due to repeated issues or another order.
The second one had in previous versions a similar problem, but with the appearance of epubcheck 4.2.1 we divided into the the minimalistic EPUB toc and an additional list of alternatives to avoid the problem.

As far as I remember, in the fourth example both the spine content and the toc content are both in alphabetical order of the file names (which are generated randomly), therefore just luck, that both have the same order (or simplicity of the generating script).

For the first one, the problem just does not appear, because it provides no deeper access to all files with one navigation list, there is no such global alternative to link deeply into the story. But the story has cyclic orders including repetitions, if the audience chooses the corresponding links. a toc representation would be more complex than the graphics in the third one.
This one is bilingual (german/english)
https://www.bookrix.de/_ebook-marie-de-sade-mein-facettenreich/
Both versions of a poem are in the same document (for comparison for example), but the toc contains one german and one english sublist as alternatives, depending on the preferred language.
Due to this it fails as well in epubcheck 4.2.1 - maybe this will be fixed with the next version. Currently BookRix does not use epubcheck 4.x, therefore up to now no problem for updates with unchanged navigation concept.
One solution could be to use an SVG with a language switch for the toc here, but this is not applicable as EPUB navigation document anyway, therefore just another workaround.

[mattgarrish]

In this non-linear table of contents, what is "next"?

But this is going outside anything the specification has defined. The only requirement of reading systems is that they provide access to the navigation links to users and move the user to the destination when activated: https://www.w3.org/publishing/epub3/epub-packages.html#sec-package-nav-rs-conf

It's a case for recommending order, but I just can't get on board the requirement train when the specification is silent about a required purpose for matching order with the spine.

[bduga]

Thanks @Doktorchen! I just checked the behavior on our reading system, and it kind of works - we just sort all the entries, but it does look odd. For instance, all sub items come under "My Empire of Facets", and the German version is empty, but they still take you to the correct page, etc. I don't think the ToC mechanism is really designed for this use case (two parallel ToCs).

@mattgarrish I am not sure I agree with your reasoning here. We should assume that Reading Systems make use of any and all conformance requirements, and we have many others that have no specific reason given. While it might be nice to list reasons, I don't think we can ignore conformance requirements just because the spec doesn't specifically say why they are there or what capabilities it enables. For instance, we also require 'A span element MUST be followed by an ol ordered list; it cannot be used in "leaf" li elements' but there is no matching required purpose for that. However, we should assume that Reading Systems expect epubs to conform to that requirement and will make use of it in their processing code.

[mattgarrish]

I don't think we can ignore conformance requirements just because the spec doesn't specifically say why they are there or what capabilities it enables.

But there needs to be at least some rationale for a restriction, otherwise what governs anything we include? You don't have to agree with the content model for navs, but the restrictions are to make for simple parsing which gets to the requirement for presentation and activation. Order doesn't relate to what the specification requires of reading systems.

Ordering was never even discussed as far as I'm aware, and I take it as gospel that authors don't familiarize themselves with every requirement of the specification but do whatever epubcheck blesses.

I understand the utility for you, but if it's not being used in these ways in Japan and Japanese publishers find the rule problematic, I think that in itself is a case that the rule is too restrictive.

Is pushing it down to a recommendation really going to be all that problematic, though, if warnings basically have the opposite effect here of keeping such content out? As a compromise, I can live with that.

(And for the record, we should do a better job of documenting restrictive statements that aren't clearly tied to reading system requirements. A good example of the confusion we inflict on people when we do this is the mathml "simplifications" that have been repeatedly returned to.)

[bduga]

But there needs to be at least some rationale for a restriction, otherwise what governs anything we include?

Order seems to have a pretty good rationale - it provides a consistent data model across reading systems. Without it, it is not even clear in what order the ToC should be presented to the user, let alone how to handle edge cases. If I have toc that lists ch2, ch1, ch3, and I am in the middle of ch2, where does next chapter take me? ch1? ch3? If I navigate to the first item in the toc, then read to the end, do I ever get to ch1? There are plenty of possible answers here, but it breaks epub navigation interop since reading systems will just do whatever they think is right. Then we end up with the mess of publishers expecting all reading systems to behave in the same way and complaining about bugs, because system x doesn't work like system y.

Ordering was never even discussed as far as I'm aware, and I take it as gospel that authors don't familiarize themselves with every requirement of the specification but do whatever epubcheck blesses.

I don't recall off hand, but this is a pretty explicit requirement. It's not some weird edge case where we were discussing ordering for one thing and accidentally made it apply somewhere else. I expect this stems from playOrder and the ncx. It sounds like you are arguing that requirements in the spec not already covered by an explicit test in epubcheck should not get a new check and should be moved to a recommendation, since we can't expect publishers to know about requirements that aren't already checked.

I understand the utility for you,

There is no specific utility for us. We just sort them, which gets us to a vaguely reasonable outcome. Hopefully it is what the publisher and readers want, and what everyone else does!

but if it's not being used in these ways in Japan and Japanese publishers find the rule problematic, I think that in itself is a case that the rule is too restrictive.

Hmm... It doesn't seem like they find it problematic in the sense that violating the rule is enabling a behavior they want, but rather that it is just too hard for some publishers to fix it. The properly ordered and hierarchical toc for the Japan cases I have heard about sound better and more interoperable.

[mattgarrish]

Order seems to have a pretty good rationale - it provides a consistent data model across reading systems.

Order of entries, yes, hence the choice of an ol in the content model. One thing always follows the next, whether the destinations seem logical or not. Consistency is only broken if you break from the publisher's intent and try to force a different logic onto the list.

The fact that it doesn't make much sense for the order to differ from the spine in most cases leads to an ordered outcome without any reason to make it a "must". As an authoring practice, accessibility guidelines will also push an ordered outcome. I'm not arguing it as something I would do, but order isn't necessarily the only logic that could govern entries in a toc (e.g., listing supplementary content at the end because reading systems don't show alternative navs).

It sounds like you are arguing that requirements in the spec not already covered by an explicit test in epubcheck should not get a new check and should be moved to a recommendation, since we can't expect publishers to know about requirements that aren't already checked.

No, I'm arguing that when we receive feedback like this that a rule is problematic, and it's not entirely clear why we need it, we shouldn't rush to enforce it just because the spec says so. It should get a proper hearing from all interested parties.

Since we ruled discussion of this too late for 3.2, leaving the status quo in epubcheck would be the fairer thing to do given that we've opted to quietly ignore other problematic things that by rights should receive exactly the same treatment. I can live with a warning in the interim, too,

But I'm not sure if we're just deciding here how to handle this in epubcheck, or whether we're opening a new revision to tackle the spec?

[Doktorchen]

bduga:

• we just sort all the entries, but it does look odd.

Of course, if you rearrange content, you will get a surprise ;o)
Presumably most authors will consider a viewer to be completely borked, if it starts to sort/reorder any content of their books. If this really exists, such behaviour can result in a revival of hints like 'best viewed with' or 'do not use program x for the reception of this book, the program has too odd bugs'. (of course, in this case, due to the current recommendation the navigation document has a bug as well).
What is the reason to sort any content? - I cannot imagine.
Traditionally it is considered the task of authors to confuse the audience with odd content ;o)

Well, I have seen already a browser extension presenting the navigation document properly, but the links in it do not work at all - I cannot imagine as well, why to publish such programs ;o) I tested EPUB viewers, which did not manage to present any EPUB, maybe I'm expecting too much straight forward or traditional behaviour from programs? ;o)
And what, if if the navigation document does not contain links to all content documents at all (as already mentioned related to the 'edge' case?)

If the requirement to align with the spine has any (not only historical) purpose, it would be a good idea to mention this in the specification and to report bugs by epubcheck to give authors a better chance to reconsider, what to mention at all in the navigation document, what to mention in other documents with expanded navigation structures.
If not, it could be replaced by a hint, that the requirement is removed, but older programs may have odd behaviour, if the navigation listing is not aligned with the spine.
Maybe useful for authors in both cases a hint, that other or more complex navigation structures than that of linear content in the spine might be better arranged in a different, normal content document, which does not have such requirements or restriction that the navigation document.
Typically due to backwards compatibility there is still the old nav.ncx pointing only to the content document with the navigation listing, this can be the fate of the EPUB 3 navigation document as well for books with a more complex structure.

[dauwhe]

I'm starting to think we should remove the restriction entirely. What business is it of ours to prevent people from using the tools afforded by EPUB to do what they want? EPUB should ensure that reading systems know what to do with the information they are given. The spine determines the order of primary resources. Reading systems must provide access to nav, and if you click on a particular nav entry the RS must take you there. Beyond that we are trying to legislate morality, but will succeed only in tying the hands of authors.

[clapierre]

I for one like the new check to ensure the Nav order matches the Spine order, this has caught some publisher mistakes which can cause navigation issues for assistive technology and we use this as part of our Global Certified Accessible (GCA) program.

I do understand that this may not always be the case as was stated from our Japanese members, and in some special edge cases.

I would suggest that since it is both helpful for some and harmful for others that we place an option in EPUBCheck to include this check or not. I am fine with this test being either opt-in or opt-out, but it sounds like opting in would be the safer choice and those who want the Nav/Spine to match would add an additional command line option to enable to check.

[laudrain]

@clapierre so THERE IS an accessibility issue.

I support Dave's above comment, and Matt's pragmatic vision on not checking what was not checked.
And there is a huge business case on the market when spine and nav have not the same order.

So is there any other mean for accessibility to work fine even when removing the restriction entirely?

[mattgarrish]

So is there any other mean for accessibility to work fine even when removing the restriction entirely?

Put the check in Ace.

[laudrain]

@bduga I am speaking for the Japanese market, referring to @ShinyaTakami comments in w3c/epubcheck#1036

[clapierre]

@laudrain

@mattgarrish I fully support to put the check in Ace, but still the spec have to be less restrictive to let EPUBChek be coherent with them, as @rdeltour remind us.

I am also fine with moving the check from EPUBCheck into Ace, as long as both gets updated at the same time, otherwise we will not upgrade EPUBCheck until Ace supports this level of checking.
Thanks!

[murata2makoto]

I created three EPUB publications having spine-nav mismatches.

https://1drv.ms/f/s!An5Z79wj5AZBgpFAxdv8z4Fo_z3ijw

moonstone: a very lengthy novel. The nav contains a top-level TOC followed by second-level TOCs.

first_book: a collection of poems. The nav contains a TOC as well as an alphabetical list.

HolmesMixture: a novel and intervening short novels

[dauwhe]

Test file available at https://github.com/w3c/publ-epub-revision/blob/master/tests/Moby-Dick-backwards-nav.epub

[GeorgeKerscher]

Hello, In Edge, the Backward Moby Dick was not useable at all. There is no go to next chapter function, so even though the spine order is linear , I could not get to the next chapter. In VitalSource Bookshelf for Windows version 7.6.3, I could read the book, because there is a go to next chapter function. I can envision this approach when there are different ways to explore the title, city guides, cookbooks, etc. Best George From: Dave Cramer <[email protected]> Sent: Thursday, July 25, 2019 12:16 PM To: w3c/publ-epub-revision <[email protected]> Cc: George <[email protected]>; Comment <[email protected]> Subject: Re: [w3c/publ-epub-revision] Must navigation order match spine order? (#1283) Test file available at https://github.com/w3c/publ-epub-revision/blob/master/tests/Moby-Dick-backwards-nav.epub — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1283> , or mute the thread <https://github.com/notifications/unsubscribe-auth/ABW4OSFBNT53OQHKKN65OC3QBHUUPANCNFSM4H27MOAA> . <https://github.com/notifications/beacon/ABW4OSEUMVGTALOCV2VN6U3QBHUUPA5CNFSM4H27MOAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD22KVDA.gif>

[llemeurfr]

On the Readium side, there is not problem with Backward Moby Dick on R2 Reader iOS, R2 Reader Android nor Thorium Reader (MacOS, Windows, Linux). The spine is processed for the reading order (next/previous) and the ToC links to the proper locations in the publication.

[murata2makoto]

@llemeurfr @GeorgeKerscher Could you try my three examples?

[GeorgeKerscher]

Sure, where are they? Best George From: MURATA Makoto <[email protected]> Sent: Thursday, July 25, 2019 6:42 PM To: w3c/publ-epub-revision <[email protected]> Cc: George <[email protected]>; Mention <[email protected]> Subject: Re: [w3c/publ-epub-revision] Must navigation order match spine order? (#1283) @llemeurfr <https://github.com/llemeurfr> @GeorgeKerscher <https://github.com/GeorgeKerscher> Could you try my three examples? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1283> , or mute the thread <https://github.com/notifications/unsubscribe-auth/ABW4OSG22QSGCUGV6SMA2TTQBJB5ZANCNFSM4H27MOAA> . <https://github.com/notifications/beacon/ABW4OSCFT7G65EFKKMYDLPLQBJB5ZA5CNFSM4H27MOAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23FJBA.gif>

[murata2makoto]

@GeorgeKerscher
Here you go.

[danielweck]

moonstone: a very lengthy novel. The nav contains a top-level TOC followed by second-level TOCs.

@murata2makoto in two of the Readium desktop readers I am working on, this is how the TOC of moonstone.epub looks like (is that what you would expect?):

[murata2makoto]

@danielweck Yes, this is the intended rendering.

Does your desktop application provide navigation within the TOC? For example, go to the next/previous subsection, go to the beginning of the parent section, etc. If it does, is my nav doc tractable?

[danielweck]

navigation within the TOC

Nothing sophisticated, just the standard (shift)keyboard-tabbing "up and down" the hierarchical list of headings, as per the linear document order of the source navigation document or NCX.
And each heading is basically a hyperlink into any arbitrary document location within the publication. At this point in time the TOC implementation does not guide the user with additional contextual heuristics / helpers. As you can see from the screenshots, the TOC is presented much like the HTML rendering (also shown in the screenshots), with a combination of inert headings (anchor with no href => not clickable, not linked to anything), and active interactive hyperlinks.

[VeryGoodErotica]

I'm brand new to ePub - previously I've done all my publishing via LaTeX where a considerable amount of freedom exists. However PDF is not easy to read on phones etc.

So I apologize if this concern has been brought up, but it is very important to me.

The order should NOT be required to match and there is historical publishing precedence where they do NOT match - such as in the pulp magazines and presumably others.

Very often stories will be listed together in the ToC, followed by things like essays/poems, followed by glamour photo shoots, etc.

Their actual location within the magazine itself is mixed - but having them grouped in the ToC makes it easier for someone looking for a specific type of content within the magazine to find it.

In LaTeX just use the tocloft package and you can do the ToC any way you want. Why does ePub want to be more restrictive on creative typographical freedom?

[danielweck]

Welcome to GitHub @VeryGoodErotica, and thanks for chiming in! (here and in the other issue)

Would EPUB "landmarks" be useful in your use-case?
https://www.w3.org/publishing/epub/epub-packages.html#sec-nav-landmarks

Note that you can design any "table(s) of contents" (I use the term loosely here) and display it/them to your readers using HTML document(s) judiciously placed in the reading order (aka publication "spine"), or linked from landmarks but declared as "non-linear" (although I would personally refrain from using this latter technique, as it is not supported consistently across reading systems).
You would of course still need to design a machine-readable TOC (i.e. nav with epub:type="toc") that would only provide a high-level and linear-order-matching view of your publication structure, lacking the detail of your rendered (HTML + CSS) table(s) of contents ... thus why I suggest defining machine-readable landmarks as well, so that reading systems can at least point to the more "fancy" TOCs available in the publication.

I hope this makes sense! (it is getting late here ;)

[carltongibson]

Is making this a SHOULD good enough?

I think a SHOULD is fine.

Working on the EPUB for the docs for the Django Web Framework, we have done everything we can to produce a valid EPUB. This one rule is blocking us. Django Issue

We have a section index page with a TOC at the top, linking to various subpages, and to sections lower on the main page. Given the content, it makes no sense to reorder, and in the end we had to decide to ignore this warning. (Maybe with a big restructure we could satisfy it but that feels like putting the cart before the horse.)

Hopefully the example helps you to decide.

Thanks for all the effort!

[wareid]

F2F Meeting October 22, the group has resolved to change the MUST to a SHOULD.

This will align the specification to the current state of EPUBCheck. We also recommend that Ace add a check for linear order as an accessibility requirement.