Supplemental content crossrefs findings and recommendations I
- We won't move forward with pop-ups for cross-references, as neither approach resonated with participants, and nearly all participants ultimately expressed a need for having a cross-references in a new tabs/windows
- Instead of using pop-ups for cross-references, have these links open in a new tab
- In the live code, we can experiment with a simpler hover option for cross-references, given that we already have a hover style already mocked up
- Study the problem of "where I am / where have I been" to figure out what the problem is - are people afraid of losing their place? If so, would breadcrumbs help?
- For now, strike the "back" functionality and use the live code to further identify navigation/information architecture issues (including but not limited to further experimentation with the "back" functionality)
- Figure out how to work best with what a browser is good at (tabs as a todo list, windows for side-by-side work)
- Where possible, make eRegs act as a normal website (less like an iPad app)
- Explore possible lightweight ways to help facilitate research practices: learning to manage tabs well, using browser extensions, having a bigger monitor
- Try to record at least one short video during our pilot about using eRegs for research
- For the supplemental content sidebar, look more at the slide-out vs dropdown behaviors in the live code to learn about preferences and help decide between them
- All tasks were completed with 100% success for both of the limited prototypes with the exception of one participant using the Subpart Resources menu to find the ACA prior to using the in-text link (which was considered a partial success).
- Most participants indicated that browser tabs would be useful and acceptable for cross reference links.
- Participants preferred having pop-ups that were wider and showed the content more fully.
- Many participants asked if the center pop-up was movable and indicated that it would be strongly preferred if it were.
- Participants would be able to more easily find links in pop-up titles if they were underlined.
- Participants preferred the information design of the collapsed menus for the Subpart Resources in the slide-bar, but further research is needed to determine how the slide-out animation feels to users.
- Move forward with “Subpart Resources” as the title for the supplemental guidance section.
We wanted to know if people preferred the center pop-up or the side pop-up for cross referencing citations in regulation text. When asked directly, most of the participants expressed a preference for the center pop-up though there were many caveats. It should also be noted that both pop-ups yielded successful test results in all cases, and preference for the center pop-up was mostly with regard to the width.
- P4 said “I really did like the actual pop-up in the page, even though I said that I would like to compare the two, there's still a way to do that. Having that front and center seems feels better to me”
- P4 went on to say “I would find myself using the pop-up windows because I can see that the information I was looking at was still there. What frustrates me is if I click a link and it takes me completely away from the page that I was working on”
Multiple participants expected or requested the ability to move the pop-up away from the middle of the page. While they preferred the width of the mid-window pop-up, they didn’t like that it blocked the main text as they wanted to be able to compare the information displayed in both.
- P2 said “It would be nice if there were some way to scroll up or scroll down to see the surrounding language”
- P1 said “I would want that pop-up not to be in front of the language that it's already open. I want to move it out of the way so that I understand what I'm reading over here.”
While some participants found the pop-ups useful, all participants expressed the expectation or desire that clicking on a link would open in a new browser tab. While previous research has indicated that users may often end up with a plethora of open tabs, the ability to have the full information side by side is valued.
- P3 said “I would prefer that it either open in another tab so I could toggle back and forth if I needed to and keep it open or that I would just go right to that page because here it's going to be a bit smaller. And if it's a lengthy thing it's going to be, it's going to be a little bit harder to read”
- P1 said “I prefer separate tabs because if i have to put it side by side to compare what the regulation say I can just go ahead and don’t have to minimize a window or anything”
- P2 said “I would copy this. And then, you know, open up a new window and Google it, just to pull up the original ACA to see.”
Recommendation
- If it is possible for the center pop up to be movable, then this would be the preferred option between center and side popup.
- Consider just having links open in a new browser tab.
All participants were successful in finding source links in both the center pop up and the side pop-up, though links without an underline were harder for participants to identify.
- P2 said “I would expect [the pop-up] to link out to that specific Reg, on the other hand, [the title is] not underlined. So I wouldn't necessarily say it screams to me ‘Oh, I'm a link.’”
For the center pop-up cross-reference, with both the title and the ‘visit hhs.gov’ working as links, some participants were unsure which link would take them to their goal location. One participant said “If they're going to the same place, unless you're doing two different formats, it's versions of the same format, why would you have two links?” Some participants expected the ‘visit hhs.gov’ link to go to the HHS homepage, instead of to the source material.
Recommendation
- Consider consistently indicating links with an underline
- If the center pop-up is used, consider removing one of the links
While all participants were able to find the in-text link to the ACA reference, one participant’s first inclination was to utilize the statutes menu in Subpart Resources. With prompting to find another way to access the information the participant was able to find the appropriate pop-up and link respectively.
Recommendation
- Move forward with link to the ACA in a way that aligns with pop-up decision
While all participants were successful in finding and following the “back” link to return to the regulation where they had started, participants mentioned being unsure where the link would take them.
- P2 said “Is ‘back’ going to take me to where I was or was it going to take me to 435.602, because then I could see both being useful”
Recommendation
- Add supplemental text to indicate to where the back button will take you
In both the slide-out menu and the dropdown supplemental sidebar all participants were successfully able to locate CIBs.
The majority of the participants preferred the slide-out menu over the dropdown menu though few commented on the slide-out or dropdown action specifically. Rather, the participants spoke favorably about the rolled-up information as displayed in the slide out menu as it allowed them to focus on the specific content they wanted to see rather than having to scroll through the content.
- P3 said “It's cleaner because I can open up right what I want to open up and then go from there”
- P4 said “I liked the [slide-out menu a] whole lot better because it's focused by the different guidance… [the dropdown has] them all listed out and it seems like I'll have to go through a lot to get to the CIB that I wanted.”
Recommendation
- Participants indicated a preference for collapsed sub-menus but further research should be done with regards to the slide-out vs dropdown menu.
While not all participants were asked, those who were, preferred the section
labeled “Subpart Resources” over “Subpart Help”, citing that it seemed to be a more appropriate description given the audience.
- P1 said “I'm not really looking for it or help, I'm looking for information, I'm looking for resources.”
- Similarly, P3 stated “I would say resources just because I don't, I don't know that it's necessarily ‘help’, but it's more, it's more like related, related documents and related sites and stuff like that. So I think resources is actually a better term for it.”
Recommendation
- Move forward with “Subpart Resources” as the title.
Study goals
- To observe and learn from users in order to determine which design approach performs best and identify possible usability improvements.
Research questions
- How do users want to see cross referenced information?
- Does a center pop-up, or side bar pop-up work more effectively?
- Would users prefer a different way to cross reference information?
- Can users find the link to the source material in the pop-ups?
- Can users find their way back from cross reference jumps to other regulations on the site? C
- Can users find specific information in the Subpart Resources section?
- Do users prefer a slide out menu or drop down menu?
- Is “Subpart Resources” or “Subpart Help” a more appropriate title for the section?
Participants P1, P2, P3, & P4 all had some experience using regulations as part of their job but were not necessarily policy owners/SMEs. They rely on policy owners in the CMS Central Office.
Methodology We conducted a moderated formative evaluation using the think-aloud protocol. Since two prototypes were tested, the order will be flipped for each participant so as to not bias users towards one over the other.