Option to play audio coordinate when using object navigation

[nvaccessAuto]

Reported by jteh on 2012-07-25 07:05
Sometimes, it is useful to get a quick idea of where an object is located on the screen. While mouse audio coordinates can be used to do this, this could be extended so that an audio coordinate beep is heard when using object navigation indicating the location of the current navigator object. This would be particularly useful for touch screen users when flicking between objects.
Blocking #4072

[nvaccessAuto]

Comment 2 by nvdakor on 2014-04-13 10:51
Hi Jamie,
Thanks for locating the correct ticket for this request.

[nvaccessAuto]

Comment 3 by nvdakor on 2014-04-14 23:51
Hi,
Initial work is ready for review:

• Branch: t2559
• Commit ID: d994dc3cb
  I decided to implement this on next nav object command for now to gather feedback. Borrowed play audio coordinates method from mouse handler unless if there is a better version.
  Thanks.

[nvaccessAuto]

Comment 4 by nvdakor on 2014-04-15 00:45
Further to the commit, I have a design suggestion:
If audio coordinate beep is to be heard while moving the mouse, flicking on the touchscreen and via keyboard-based object navigation, then I propose that a new function be created in the API module. The specs would be as follows:

• api.playObjectCoordinates(obj)
• Purpose: given an object, play its XY coordinate.
• Implementation: essentially move mouseHandler.playAudioCoordinate to API module so it can be invoked in touchHandler, globalCommands, mouseHandler and other modules.
  Algorithm:
• The new function retrieves the coordinates for the obj via obj.location.
• Retrieve the "screen resolution" i.e. desktop object's coordinate.
• After this, it'll be same as the current mouseHandler.playAudioCoordinates function.
  Usage from other modules:
• When the user sets play audio coordinates to yes, the routine which calls this function should pass the desired obj to the API function if the object has location.
  Example:

  if config.conf.PlayAudioCoordiante: api.playObjectCoordinates(curObject)
  
  

Thanks.

[nvaccessAuto]

Comment 5 by nvdakor on 2014-04-15 07:12
Hi,
A preliminary implementation is ready for review:

• URL: https://bitbucket.org/nvdaaddonteam/nvda
• Branch: t2559APIExperiment
  All I did was copied mouseHandler.playAudioCoordinates to api.py for now and added a new function called api.playObjectCoordinates. The global commands commands for next/previous object now calls api.playObjectCoordinates(curObject) to play the object coordinates.
  I'm sure a better solution exisets. Thanks.

[nvaccessAuto]

Comment 6 by jteh on 2014-04-15 09:00
In general, new functions should not be added to the api module. It's poorly named, and in an ideal world, everything would be in more specific modules and the api module wouldn't exist. :) Perhaps the screenExplorer module is a better place for this.

[nvaccessAuto]

Comment 7 by nvdakor on 2014-04-15 09:19
Hi,
I think that's reasonable to move the audio playback function to screen explorer.
as for the original question: I think playObjectCoordinates should be moved into screen explorer so other modules may use them instead of adding object coordinate beep code in the global command routines. What's your take on that?
Thanks.

[nvaccessAuto]

Comment 8 by nvdakor on 2014-04-15 10:06
Hi,
Okay, a major refactor has been made thanks to the suggestion:

• URL: https://bitbucket.org/nvdaaddonteam/nvda
• Branch: t2559APIExperiment
• Commit ID: f342f74f
  In this refactor, we have:
• Coordinate playback functions were removed from API and were transfered to screen explorer which also now has copyright header.
• playAudioCoordinates has been renamed to playScreenCoordinates to indicate the fact that we're broadening the scope from mouse to touch exploration as well (sorry if I should have kept it as playAudioCoordinates).
• mouseHandler.playAudioCoordinates is kept back in case we need to revert this change.
• Global commands: object nav commands now calls screenExplorer.playObjectCoordinates(curObject).
• Screen explorer now imports some needed modules (mouseHandler, screen bitmap, config) used in audio coordinates method. Also added mouseHandler.scrBmpObj instead of scrBmpObj as Python complained about not finding global variable.
  I'm going to bed. Thanks.

[nvaccessAuto]

Comment 9 by nvdakor on 2014-04-18 02:43
Hi,
A major flaw with object location sound was found when navigating large websites via object navigation:

• If you move to the middle of the page and if your navigaor object is located towards the top of the page i.e. offscreen but on top, NVDA will emit higher pitch than expectedd. in other words, if the top location value is negative, this means the navigator object is located offscreen.
• If the navigator object moves to the end of the page i.e. offscreen towards the bottom, Python throws overflow error. That is, if the top location of the object returns more than 100 percent, nVDA will play negative pitch, which is impossible.
  This would be problematic for those using touchscreens to navigate a webpage.

[nvaccessAuto]

Comment 10 by jteh (in reply to comment 9) on 2014-12-01 06:37
Replying to nvdakor:

• If you move to the middle of the page and if your navigaor object is located towards the top of the page i.e. offscreen but on top, NVDA will emit higher pitch than expectedd. in other words, if the top location value is negative, this means the navigator object is located offscreen.
• If the navigator object moves to the end of the page i.e. offscreen towards the bottom, Python throws overflow error. That is, if the top location of the object returns more than 100 percent, nVDA will play negative pitch, which is impossible.

The coordinates should be restricted to the screen size, which you can get with api.getDesktopObject().location.

[nvaccessAuto]

Comment 11 by nvdakor (in reply to comment 10) on 2014-12-01 06:52
Replying to jteh:

Replying to nvdakor:

The coordinates should be restricted to the screen size, which you can get with api.getDesktopObject().location.

Right. Should the user be alerted that the object is offscreen? I'm thinking no.
Thanks for the review and pointing out something I haven't thought about.

[nvaccessAuto]

Comment 12 by nvdakor on 2014-12-02 03:38
Hi,
Hopefully this commit should be enough - I think I've got everything except for the GUI option to configure this behavior:

• RURL: http://bitbucket.org/nvdaaddonteam/nvda
• Branch: t2559APIExperiment
• Commit ID: 5f5e042
  I'm willing to create a t2559 branch based on current work.
  For mouse coordinates, would you like to retain the coordinates function in mouse handler or move it to screen explorer? If moving to screen explorer, I propose creating a master coordinate player method that is really the coordinates function from mouse handler and which the new object coordinate can call. Thus when moving to next/previous object, screenExplorer.PlayObjectCoordinates will call screenExplorer.playLocationCoordinates (or perhaps call it play screen coordinates), with mouse handler calling playScreenCoordinates in screen explorer.
  For the config section, I'm thinking it should live in object presentation settings with the setting set to off.
  Thanks.

[nvaccessAuto]

Comment 13 by jteh (in reply to comment 12) on 2014-12-02 04:02
Replying to nvdakor:

For mouse coordinates, would you like to retain the coordinates function in mouse handler or move it to screen explorer?

Move it to screenExplorer.

If moving to screen explorer, I propose creating a master coordinate player method that is really the coordinates function from mouse handler and which the new object coordinate can call. Thus when moving to next/previous object, screenExplorer.PlayObjectCoordinates will call screenExplorer.playLocationCoordinates (or perhaps call it play screen coordinates), with mouse handler calling playScreenCoordinates in screen explorer.

This makes sense. I think playLocationCoordinates and playObjectCoordinates.

For the config section, I'm thinking it should live in object presentation settings with the setting set to off.

Agreed.

[nvaccessAuto]

Comment 14 by Joseph Lee <joseph.lee22590@... on 2014-12-02 05:45
In [9e5eb65]:

It is now possible to hear object coordinates when using object navigation commands via keyboard or mouse and moving via touch gestures. re #2559
Audio coordinate announcement function now lives in screen explorer. Essentially the new play location coordinates function is same as old mouse handler version.
In screen explorer, added a new function to play coordinates for a given NVDA object. This function in turn calls location coordinate function in screen explorer.
In object presentation settings, there is now a new option to toggle announcement of object location coordinates when using object navigation commands. By default this checkbox is not checked.

[josephsl]

Hi,

Coming back to this issue...

about to rebase it based on the current master branch and moving the branch to my own forked repo so people can start testing this.

A possible follow-up issue might be asking NVDA to play coordinates when one uses focus command such as tab, arrows and so on (for this, an even simpler solution is to expand what becomeNVDAObject event can do by playing coordinate beeps).

Thanks.

[josephsl]

Hi,

Blocked by #5906 due to the following:

If indent announcement is set to beep, moving to an edit field that indicates indents (such as Notepad++) will mean one of the new beeps will not play. Will wait until a decision on what to give priority is made.

Thanks.

[derekriemer]

@josephsl @michaelDCurran I'm curious if simply playing indent beeps 80 MS or so in the future would help alleviate this bug

[josephsl]

Hi,

Coming back to this...

Speech refactor may open up possibilities for this work, although lately I've been thinking about withdrawing this issue in favor of Audio Themes or similar mechanism.

Thanks.

[Brian1Gaff]

Yes I find the 3D nature of that add on very handy. For example when a little alert from dropbox comes along you can hear its that from the sound style and the fact that it is on one channel. It might make sence for those wanting to get some idea of where something is on a screen from left to right and up and down if some form of theme or pitch could be introduced for the screen location of a control. Brian [email protected] Sent via blueyonder. Please address personal E-mail to:- [email protected], putting 'Brian Gaff' in the display name field. Newsgroup monitored: alt.comp.blind-users ----- Original Message ----- From: "Joseph Lee" <[email protected]> To: "nvaccess/nvda" <[email protected]> Cc: "Subscribed" <[email protected]> Sent: Sunday, January 05, 2020 4:23 AM Subject: Re: [nvaccess/nvda] Option to play audio coordinate when using object navigation (#2559)

…

Hi, Coming back to this... Speech refactor may open up possibilities for this work, although lately I've been thinking about withdrawing this issue in favor of Audio Themes or similar mechanism. Thanks. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #2559 (comment)

[josephsl]

Hi,

2024 update: with the introduction of sound split, I don't think we can proceed with this issue unless it is okay to play audio coordinates for navigator objects across channels. But then it may cause issues for folks using mono audio. Therefore, I'm closing the issue for now.

Thanks.

[Adriani90]

The doubt related to sound split is not valid, because the same principle applies for already implemented mouse coordinates. By the way, sound split can be disabled completely in NVDA if needed.
I am reopening.