UI object for epubReadingSystem

[mattgarrish]

A more extensive proposal for discovering the presence of UI chrome (and other features) was proposed on the working group mailing list than the simple boolean property that was proposed for the first draft.

For more information, see the thread at:
https://groups.google.com/d/msg/epub-working-group/L3M5K8vrU4U/cMxF7S5tCAAJ

[pettarin]

In case it helps the discussion:

http://www.readbeyond.it/minstrel/developers.html#js
http://www.readbeyond.it/menestrello/menestrello_test_js.epub

[danielweck]

Shared Google doc: https://docs.google.com/document/d/1YlnxlyO960hIFiJmXm6uke_cT9JhEI3cTipyY7m1R5g

[danielweck]

Comments I made in the aforementioned doc:

uiVisible = "Boolean value indicating whether the Reading System chrome is currently visible"

https://rawgit.com/IDPF/epub-revision/master/build/31/spec/epub-contentdocs.html#app-ers-properties
or:
https://raw.githack.com/IDPF/epub-revision/master/build/31/spec/epub-contentdocs.html#app-ers-properties

Some concerns:

As a reading system implementor, what value should I give navigator.epubReadingSystem.uiVisible during the application's lifecycle?
As a content creator, can I rely on the uiVisible property to help me deliver predictable user experiences? (for example, a typical nuisance is when authored scripted interactions are interfered with, because of overlapping application chrome, user event hijacking, etc.).

A typical user interface would consist in several parts, some of which may indeed partially overlay the surface area used to render the EPUB HTML/SVG documents, thereby resulting in visually-obscured EPUB content, and in trapped user events (e.g. touch gestures and mouse interaction). However, other UI controls would not necessarily overlap (i.e. application chrome laid out around the EPUB document frames), but may indeed be made temporarily "invisible" (for example, to eliminate distractions during the reading experience ; after a preset inactivity timeout ; only to be reactivated when user activity resumes, e.g. mouse move, finger tap). In fact ; as is typical in most reading systems ; the previous / next "page turn" UI affordances are usually hidden / discrete (visually speaking), whilst still being reactive to user interaction (which may cause problems if the user intent was originally targeted at the EPUB content itself, during a time when the UI controls were effectively "invisible"). Obviously, swipe gesture handlers are totally invisible, so I am wondering if the uiVisible property (or such like) would be addressing this too?

Bottom line: I think that the concept of "UI visibility" proposed by the uiVisible property is ambiguous at best, and at worst: cause for major inconsistencies in reading systems, potentially making the property useless for content authors.

Note that I am talking about "persistent" UI, not transient popup menus, dialogs, etc. which are usually modal (mutually exclusive with a normal reading experience), such as for selections / highlights / annotations, etc..

In order to add a meaningful / useful new property to navigator.epubReadingSystem (e.g. uiOverlap instead of uiVisible), perhaps we should articulate the user needs first (i.e. use cases where content authors would benefit from using such information in their scripts). For example, what granularity of information should be conveyed in this property? A simple true / false indicator probably has limited usefulness, compared to knowing exactly what graphical areas of EPUB content are being interfered with. Unfortunately, the level of complexity implied by anything other than a simple true/false concept is likely to result in poor adoption in reading systems. So, how exactly is uiOverlap true/false going to help content authors build better user experiences?

[danielweck]

@mattgarrish can you please assign me to this? (as agreed on today's concall)

[bduga]

@danielweck I agree with your assessment. This sounds very problematic to get right in any detailed way, and the naive single toggle is so broad and up to so much interpretation as to be effectively useless. The UI of an app and the device on which the app runs is not only varied, but under constant change. For instance, on Android, older versions always showed the status bar at the top. Newer versions added a fullscreen mode. Even newer versions added notifications that can appear on top of content when in fullscreen. Who knows what future versions will bring? If we define exactly what we mean for the UI being visible for all current versions of Android, iOS, MacOS, Windows, Linux, etc. we will have an enormously complex definition that is extremely fragile (OS vendors like updating things). If we don't, then it is up to the RS to decide what counts as UI, which makes it hard for a script to really react in a predictable way.

[danielweck]

Thanks @bduga I like your short summary better than my full comment :)

[danielweck]

Shared Google doc:
https://docs.google.com/document/d/1aLPwp6bvIki5LHz2yWhypqzajr9jR8CyKqTlpzqUwgs

[chrmitchell]

I agree that a simple boolean would probably not be useful at all, and that the more complex approach of returning an object indicating the visibility of various pieces of the UI could quickly become overly complex. The initial motivation for the feature request was to a) minimize the amount of challenges developers face when developing more complex / full-screen interactivity in a document, and b) avoid splintered implementations as more reading systems begin to support interactivity, each with their own way of handling user input to call/dismiss UI elements, trigger page turns, etc.

It sounds like a UI visibility property/object is not the best direction to go in, but I'm wondering if there are any other solutions we could consider at this point that would be beneficial to developing complex interactive content inside epubs, or is this just not something that should be tackled right now?

For example, one potential use case for a UI visibility API would have been to know when to block or allow touch events that would cause some UI to be shown/hidden:

Say a developer wants to create an HTML document in a fixed layout EPUB that has a button that brings up a full-screen widget of some sort, maybe an interactive map. He would like to ensure that when the map is displayed, none of the reading system UI is visible. He could try to achieve this using preventDefault on various pointer events, but since there is no definitive way of knowing whether any UI is currently visible-- since it may have been visible or hidden when the page was initially loaded -- then i'm not sure there's a way to ensure the UI does not cover/interfere with the map.

It's an admittedly narrow use case, so maybe it's not something to solve for right now, but it does seem like it will come up more and more going forward.

[danielweck]

@chrmitchell thank you for your feedback!

The initial motivation for the feature request was to a) minimize the amount of challenges developers face when developing more complex / full-screen interactivity in a document, and b) avoid splintered implementations as more reading systems begin to support interactivity, each with their own way of handling user input to call/dismiss UI elements, trigger page turns, etc.

I can totally empathize with content authors / creators, as I too faced the challenges of highly-interactive EPUB3 content due to reading system inconsistencies. I also contributed to the implementation of e-reading software (not just Readium), so I know what the problems are from this side of the fence as well.

I agree with you that a uiVisibility-like property on navigator.epubReadingSystem is unlikely to achieve the desired goals.
Yes, event.preventDefault() and event.stopPropagation() sometimes help, but not with every reading system.

I fear that this is another case for documenting "good practice", at least at this stage. I am thinking about the "minimal CSS" discussion ( #672 ), which really boils down (in my opinion) to encouraging RS developers to be transparent about the impact of their rendering processes on authored content (well, there's also the proposal to add a new metadata hint for reading systems https://docs.google.com/document/d/1R8bHqR0zV79B9io5JGSD21mYOa8sQIyzrZn6WkT_H_U but there's a strong case for user needs to always override authors wishes in reflowable content, with fixed-layout publications in a special category of publications that prioritize visual fidelity).
PS: there's a writeup about Readium's handling of keyboard shortcuts, touch gestures, etc.:
https://github.com/readium/readium-shared-js/wiki/ContentModifications