[VoiceOver] Object responses not heard with hand slider

[Nancy-Salpepi]

Test device
MacBook Air (m1 chip)

Operating System
macOS 13.0.1

Browser
safari 16.1

Problem description
For phetsims/qa#852:
With VoiceOver, it seems like object responses are being overwritten by context responses when moving one hand at a time. I am not hearing any changes to Challenge Ratio proximity. You can see at the start of the video that first time I move a hand, the response appears in the VO box, but it is never uttered and I never see the response again.

When both hands have focus and I move a hand, I do hear the object responses.

Met with @terracoda over Zoom to discuss. TS is not seeing this issue with macOS 12.1.1.

Steps to reproduce

1. Turn on Voiceover
2. Tab to a hand and move it up/down one press at a time.

Visuals

VOchallengeratio.mov

Troubleshooting information: !!!!! DO NOT EDIT !!!!! Name: ‪Ratio and Proportion‬ URL: https://phet-dev.colorado.edu/html/ratio-and-proportion/1.2.0-rc.1/phet/ratio-and-proportion_all_phet.html Version: 1.2.0-rc.1 2022-11-11 22:40:36 UTC Features missing: applicationcache, applicationcache, touch Flags: pixelRatioScaling User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Language: en-US Window: 1544x712 Pixel Ratio: 1.7999999523162842/1 WebGL: WebGL 1.0 (OpenGL ES 2.0 Chromium) GLSL: WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium) Vendor: WebKit (WebKit WebGL) Vertex: attribs: 16 varying: 31 uniform: 1024 Texture: size: 16384 imageUnits: 16 (vertex: 16, combined: 32) Max viewport: 16384x16384 OES_texture_float: true Dependencies JSON: {}

[zepumph]

Next steps are to try to reproduce this, and then make sure that it wasn't an issue with VO state as opposed to sim state.

[zepumph]

@marlitas and I discussed and investigated this a lot.

Marla is on mac OS13.0 and was easily able to reproduce this bug.

• It occurred in master, on the published RAP, and also is GFL. So it seems like quite a breaking in how VO works.
• It doesn't not occur in iOS+VO+safari, it also doesn't occur with VO+chrome on a mac.
• It did not fix it to alter the timing of the context response that comes in after interaction, so it is clear that it isn't about interruption.
• It wasn't totally clear that @marlitas and I were using VO the way that it expected to we never "entered" the slider with the key commands it recommended. Instead we just used arrow keys after tabbing to a slider hand. Perhaps the bug is that we aren't "in" the slider.

No matter, I wonder how much more effort this issue is worth when I can't help but think that this is a safari bug introduced in OS13.

@emily-phet can you please recommend if we should spend more time on this? Thanks!

[zepumph]

My recommendation is to close as a wont fix.

[emily-phet]

I think we should confirm if this issue is present when the slider is accessed using the recommended key commands. If no issue is found then, let's close this as "won't fix". Responses are always sketchy when not using the recommended keyboard interactions.

[zepumph]

After doing some testing with @samreid, it looks like this bug is present in chrome on Mac OS13 also. We didn't experience any different/better behavior when "entering" the slider with "ctrl+option+shift+down arrow".

@emily-phet from here we could either assume it will be fixed by apple at some point and do nothing, or try to investigate workarounds. What would you like to do?

[zepumph]

@emily-phet and I think the next steps here are to reach out to @jessegreenberg.

[jessegreenberg]

I tested on macOS 11.7.2 and I was not able to produce the problem. Ill see if I can update and will try again.

[Nancy-Salpepi]

@jessegreenberg I think you need to be on macOS 13.

[jessegreenberg]

Thanks @Nancy-Salpepi, unfortunately my Mac is from 2013 and cannot install macOS 13 according to https://support.apple.com/en-us/HT213264.

Id like to see how these alerts work (aria-valuetext vs aria-live?) then try to reproduce in macOS 13 to isolate the problem. Then we can find a workaround or submit a bug report.

[zepumph]

From @jessegreenberg and my brainstorm:

• Is it aria-valuetext interrupting itself? No, the aria-valuetext is only heard once.
• Is it performance? Likely no
• We want to create a dummy case outside of phet to demonstrate that this is all apple's fault.

[zepumph]

@jessegreenberg and @Nancy-Salpepi and I made progress on this today. With the snippet below we were able to see that if the aria-live has a delay of 1000ms after the value changes, then we hear the aria-valuetext completely, but with no delay, or 250ms we didn't. Furthermore @Nancy-Salpepi and I saw the same behavior where runniong RAP with ?aFunTest will delay context responses for 5000ms and we will hear the aria-valuetext in most cases. It still wasn't perfect in a way that we couldn't quite pin down. From here we can work on setting a better default for the context responses, or see if there is another workaround.

https://phet-dev.colorado.edu/html/jg-tests/tests/test.html

Now that we understand that it is about timing, I feel like it is less likely that mac will think of it as a bug to fix, and we probably do want a workaround for it.

<html>
<body>
<!--StartFragment-->

  | <!DOCTYPE html>
-- | --
  | <html>
  | <head>
  | <meta charset="UTF-8">
  | <title>TEST PAGE</title>
  | </head>
  | <body>
  |  
  | <input id="sim-slider" type="range" aria-orientation="vertical" aria-valuetext="hands, at challenge ratio" min="0"
  | max="1" step="0.01" aria-valuenow="0.2" style="top: 0px; left: 0px; width: 1px; height: 1px;">
  |  
  | <p aria-live="polite" id="my-alerter"></p>
  | <input type="range" id="my-slider" min="1" max="10" step="1" aria-valuetext="My value: 5" value="8">
  |  
  | </body>
  |  
  | <script>
  | // things to test
  | // basic aria-valuetext
  | // Add an interrupting aria-live alert
  |  
  | const simSlider = document.getElementById( 'sim-slider' );
  |  
  | const mySlider = document.getElementById( 'my-slider' );
  | const myAlerter = document.getElementById( 'my-alerter' );
  |  
  | mySlider.addEventListener( 'change', event => {
  | mySlider.setAttribute( 'aria-valuetext', `My value: ${mySlider.value}` );
  |  
  | window.setTimeout( () => {
  | myAlerter.textContent = `You just change value to ${mySlider.value}`;
  | }, 250 );
  | } );
  |  
  | simSlider.addEventListener( 'change', event => {
  | simSlider.setAttribute( 'aria-valuetext', `My value: ${simSlider.value}` );
  |  
  | window.setTimeout( () => {
  | myAlerter.textContent = `You just change value to ${simSlider.value}`;
  | }, 250 );
  | } );
  |  
  | </script>
  | </html>

<!--EndFragment-->
</body>
</html>

[jessegreenberg]

I don't think this issue is caused by the timing. More fundamentally, if aria-valuetext is ever interrupted by an alert from aria-live, VoiceOver will stop reading all aria-valuetext for that control. @Nancy-Salpepi and I verified this by changing the test to use aria-live=polite every three seconds instead of on input value change. After the first time aria-valuetext gets interrupted by an alert, VoiceOver stops reading any more aria-valuetext at all.

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>
</head>
<body>

<p aria-live="polite" id="my-alerter"></p>
<input type="range" id="my-slider" min="1" max="10" step="1" aria-valuetext="My value: 5" value="8">

</body>

<script>
  const mySlider = document.getElementById( 'my-slider' );
  const myAlerter = document.getElementById( 'my-alerter' );

  mySlider.addEventListener( 'change', event => {
    mySlider.setAttribute( 'aria-valuetext', `My value: ${mySlider.value}` );
  } );

  let alertValue = 0;
  window.setInterval( () => {
      myAlerter.textContent = `Alert value is ${alertValue++}`;
  }, 3000 );

</script>
</html>

Even more fundamentally, it is buggy that VoiceOver in macOS 13 interrupts the aria-valuetext from an aria-live=polite alert. This goes against the W3C spec (https://www.w3.org/TR/wai-aria-1.1/#aria-live) for aria-live and polite.

If there is a workaround, I think it will be in the attributes/markup of the aria-live/input element to prevent the interruption from happening in the first place. But I don't think we should look for one. We should submit a bug report to Apple about this. Then, I think we we should consider seeing if a different browser works better with VoiceOver and switching to that for our supported VoiceOver platform. Or, we should consider no longer using aria-valuetext and going all in on aria-live.

[jessegreenberg]

I submitted the bug report in phetsims/a11y-research#180.

[zepumph]

That makes a lot of sense! Thank you so much for your hard work investigating.

[Nancy-Salpepi]

Actually @jessegreenberg and @zepumph, the more I play with Friction, the more I think it is a slightly different issue. I hear the "more atoms break away" statement sometimes, but lots of times it gets completely skipped over.
Sorry. I hope I'm not confusing things too much.
I'm going to move my original comment to an issue in Friction Repo.

[zepumph]

At this point I think we should close this issue since we have phetsims/a11y-research#180. Thanks all