Pointer Events not Invoked within ScrollViewer

[robloo]

Current behavior

I have a ScrollViewer hosting a canvas control as the content. The Canvas seems to be rendered just fine inside the ScrollViewer. However, I cannot zoom or pan within the ScrollViewer. Zoom is handled by Buttons overtop of the ScrollViewer and pan is handled by UIElement Pointer events that should be coming from the ScrollViewer itself. However, none of the events work -- not even the zoom Button Click events.

Expected behavior

• Button.Click is invoked for a button displayed on top of a ScrollViewer.
• UIElement.PointerPressed|PointerReleased|PointerMoved|PointerExited is invoked for the ScrollViewer.

How to reproduce it (as minimally and precisely as possible)

1. Add a ScrollViewer with content larger than the view area
2. Add event handlers for the UIElement pointer events and watch that they are invoked
3. No event handlers are invoked

Workaround

None found so far.

Environment

Nuget Package:

• Uno.UI / Uno.UI.WebAssembly / Uno.UI.Skia
• Uno.WinUI / Uno.WinUI.WebAssembly / Uno.WinUI.Skia
• Uno.SourceGenerationTasks
• Uno.UI.RemoteControl / Uno.WinUI.RemoteControl
• Other:

Nuget Package Version(s): 3.2.0

Affected platform(s):

• iOS
• Android
• WebAssembly
• WebAssembly renderers for Xamarin.Forms
• macOS
• Skia
  • WPF
  • GTK (Linux)
  • Tizen
• Windows
• Build tasks
• Solution Templates

IDE:

• Visual Studio 2017 (version: )
• Visual Studio 2019 (version: 16.7.3)
• Visual Studio for Mac (version: )
• Rider Windows (version: )
• Rider macOS (version: )
• Visual Studio Code (version: )

Relevant plugins:

• Resharper (version: )

Anything else we need to know?

[jeromelaban]

@dr1rrb Does it look familiar ?

[dr1rrb]

@robloo just to better understand, here you are trying to listen for pointer event directly on the ScrollViewer itself? If you listen for events on a UIElement that is embedded in the ScrollViewer, you are able to get them, right?

[robloo]

@dr1rrb There have been some updates in understanding since this was first written. There are a few different cases where this issue is appearing (outlined below).

The original issue is as described: A canvas was inside a ScrollViewer and the ScrollViewer was managing pan/zoom (which didn't seem to work). Then there were additional zoom buttons directly on top of the ScrollViewer + Canvas and the Click event also didn't work there. Since this issue was filed a ScrollViewer is no longer used in this portion of the UI. Instead, SKSwapChainPanel is used for rendering and overlaid directly on top is a Canvas to capture input and overtop yet again are buttons. Zoom and pan is then all calculated internally based on inputs. However, input events don't work at all (case 1).

1. Case 1 : An SKSwapChainPanel is used to render a diagram. Directly on top of it is a Canvas to capture input. However, no input events seem to be received (PointerExited, PointerPressed, PointerReleased, PointerMoved, PointerWheelChanged). It's noted that these events likely need to be adapted to a touch screen but PointerPressed et al. should still fire.

XAML

<Grid>
      <!-- The skia canvas must have a non-zero size to trigger drawing events -->
      <skia:SKSwapChainPanel
          x:Name="DiagramSkiaCanvas"
          HorizontalAlignment="Stretch"
          VerticalAlignment="Stretch"
          MinWidth="1" 
          MinHeight="1" />
      <Canvas
          x:Name="DiagramCanvas"
          Background="Transparent"
          HorizontalAlignment="Stretch"
          VerticalAlignment="Stretch" />
      <Border
          Background="White"
          CornerRadius="0,0,12,0"
          HorizontalAlignment="Left"
          VerticalAlignment="Top">
          <StackPanel
              Orientation="Horizontal"
              Margin="5">
              <RepeatButton
                  x:Name="ZoomInButton"
                  Width="32"
                  Height="32"
                  CornerRadius="8,0,0,8"
                  Padding="0">
                  <RepeatButton.Content>
                      <FontIcon
                          FontFamily="{ThemeResource SymbolThemeFontFamily}"
                          FontSize="20"
                          Glyph="&#xE8A3;" />
                  </RepeatButton.Content>
              </RepeatButton>
              <RepeatButton
                  x:Name="ZoomOutButton"
                  Width="32"
                  Height="32"
                  CornerRadius="0,8,8,0"
                  Padding="0">
                  <RepeatButton.Content>
                      <FontIcon
                          FontFamily="{ThemeResource SymbolThemeFontFamily}"
                          FontSize="20"
                          Glyph="&#xE71F;" />
                  </RepeatButton.Content>
              </RepeatButton>
          </StackPanel>
      </Border>
  </Grid>

1. Case 2 : A continuation of Case 1 (also see included XAML above), zoom buttons are then added on top of the Canvas. The button click events are however never fired.

2. Case 3 : In some custom charts clickable elements are added. These may be parts of a chart legend, or things such as a bar chart rectangular section. These elements are commonly a Border added in code-behind with a PointerPressed event handler also connected in code. In most cases the Border elements are then added to a Grid and that Grid itself is within a ScrollViewer. For some reason the PointerPressed events of the border elements never gets called. Scrolling within the scroll viewer functions and clicking buttons also added to the Grid seems to work just fine. I expect you will need more details on this one so just let me know.

[dr1rrb]

@robloo I've tried to repro in a sample app (PtInScroll.zip), but I was unable to repro your issue

When tapping directly on the PointerHandlingLayer (i.e. the Canvas), I get all expected events

[ENTER] PointerHandlingLayer-3E31876A @691.95,422.48 (691.95,446.48)
[DOWN] PointerHandlingLayer-3E31876A @691.95,422.48 (691.95,446.48)
[ENTER] MainPage-047B7EE2 @691.95,422.48 (691.95,446.48)
[DOWN] MainPage-047B7EE2 @691.95,422.48 (691.95,446.48)
[UP] PointerHandlingLayer-3E31876A @691.95,422.48 (691.95,446.48)
[EXIT] PointerHandlingLayer-3E31876A @691.95,422.48 (691.95,446.48)
[UP] MainPage-047B7EE2 @691.95,422.48 (691.95,446.48)
[EXIT] MainPage-047B7EE2 @691.95,422.48 (691.95,446.48)

And when I tap on the Zoom<In|Out>Button, the click event is fired and I receive relevant pointer events:

[ENTER] ZoomInButton-0A224E65 @14.49,17.47 (19.49,46.47)
ZoomInButton-0A224E65 clicked!
[CAPTURE_LOST] ZoomInButton-0A224E65 @14.49,17.47 (19.49,46.47)
[EXIT] ZoomInButton-0A224E65 @14.49,17.47 (19.49,46.47)
[EXIT] MainPage-047B7EE2 @19.49,22.47 (19.49,46.47)

(The issue in that log is that I should not have receive the "exit" on the MainPage)

BTW if I remove the PointerHandlingLayer and set the background of RenderingCanvas to Transparent, I'm able to get the pointer events directly on that RenderingCanvas (i.e. the SKSwapChainPanel):

[ENTER] RenderingCanvas-20F54B10 @634.45,372.47 (634.45,396.47)
[DOWN] RenderingCanvas-20F54B10 @634.45,372.47 (634.45,396.47)
[ENTER] MainPage-15881CE2 @634.45,372.47 (634.45,396.47)
[DOWN] MainPage-15881CE2 @634.45,372.47 (634.45,396.47)
[UP] RenderingCanvas-20F54B10 @634.45,372.47 (634.45,396.47)
[EXIT] RenderingCanvas-20F54B10 @634.45,372.47 (634.45,396.47)
[UP] MainPage-15881CE2 @634.45,372.47 (634.45,396.47)
[EXIT] MainPage-15881CE2 @634.45,372.47 (634.45,396.47)

And even if I add a ScrollViewer at the root of the page and set the size of the root Grid to 3192x3192, I'm still able to get all pointer events.

[robloo]

@dr1rrb Interesting, I'll get more details ASAP and hopefully a repro. My guess is now that this has something to do with UserControls. Each case above occurs within a UserControl (the XAML example was heavily simplified).

BTW if I remove the PointerHandlingLayer and set the background of RenderingCanvas to Transparent, I'm able to get the pointer events directly on that RenderingCanvas (i.e. the SKSwapChainPanel):

The reason a Canvas is used in the App itself is not only for pointer input (although that is all that is relevant here). It also shows indicator guides for some of the charts. It is more efficient to render guides that move with the cursor on a canvas as a different layer than re-rendering the entire chart when the cursor moves.

[robloo]

@dr1rrb

I found the root-cause enough to stop investigating. It isn't that this Xaml is within UserControls; it's that each of these controls was using a Loading control from the UWP community toolkit on top of everything else. That loading control doesn't work correctly with Uno and it was causing some yet-unknown issue with input.

Apologies for not noticing earlier that the toolkit loading indicator was the issue. As it is entirely invisible in this error-state and should have been collapsed the thought didn't occur that it might be stealing input. The work around is simply to stop using the toolkit Loading control (that was planned anyway). As you don't support most toolkit controls this aspect is likely not worth further investigation.

There were still more minor issues found with the latest testing though:

1. A ScrollViewer can take over input and scroll even when the pointer is captured by another control within it. At least on Android I would expect scrolling not to function when the pointer is captured by a child element.

2. PointerPressed fires correctly over an element within a ScrollViewer. However, then moving the pointer up or down will start a scroll and:

   1. PointerExited is never called if the pointer moves outside of the element while scrolling in the scroll viewer
   2. PointerReleased is never called after the scroll viewer takes over input even if the pointer is released again over an element that previously fired PointerPressed

   ScrollViewer takes priority and then doesn't handle managed events correctly. If it is impossible to handle this transition (to/from native ScrollViewer) I would still expect PointerExited to fire immediately after scrolling is stopped. PointerReleased may be impossible but I'm not sure the internals here at all.

Separately, I'm getting crashes on PointerPressed now that input is working better over some controls. I'll have to investigate that separately but zoom in/out buttons over canvas (case 2 also works). SKSwapChainPanel just seems to have issues.

[dr1rrb]

@robloo You are facing what I usually refer as "the ScrollViewer is stealing the pointers". It's the behavior I've observed on UWP.

1. The pointer is captured by a Button when you press it, but if you begin the "scroll gesture", the button do lost its capture and the scolling kicks-in on UWP. If it was not the case, it would mean that user cannot initiate the scrolling from certain parts of the screen, which would be really annoying.
2. In both cases you should get a PointerCancelled, like on UWP (even if not expressly documented as it, it's what I saw). Actually Microsoft recommends to not assume to get a release for each pressed: "To function properly, your app must listen for and handle all events that represent the possible conclusions to a Press action, and that includes PointerCanceled." (Don't remember if UWP raises the PointerExited, but I think it won't ... pointer are really "stolen"!).

If you want to take priority over the ScrollViewer for a nested control, you have to define the ManipulationMode. For instance if you set the ManipulationMode=None on an element, you wont be able to initiate scrolling when pressing on that element. (And BTW the fact that the ScrollViewer is stealing the pointers is due to the "Direct manipulation". This will most probably change with the new "Scroller" in WinUI, cf. microsoft/microsoft-ui-xaml#108)

[robloo]

@dr1rrb

The pointer is captured by a Button when you press it, but if you begin the "scroll gesture", the button do lost its capture and the scolling kicks-in on UWP. If it was not the case, it would mean that user cannot initiate the scrolling from certain parts of the screen, which would be really annoying.

I have tested this again and Uno behaves differently than UWP. Although, I'm not sure if UWP is changing behavior for a touch screen in this case (I'm not testing with touch on desktop and have no experience with that). In UWP, if I have captured the pointer on a custom control within PointerPressed, moving the pointer up/down while continuing to press does not start a scroll. Releasing the pointer will fire events as expected. In Uno, as you said, capturing the pointer within a PointerPressed and then moving the pointer up/down will start scrolling and ignore the capture. PointerReleased/Exited are not invoked.

Now, there is a good reason for this difference as you explained. The UWP app is running on a desktop and not being interacted with directly. It is much more intuitive on a desktop to scroll the mouse wheel or use the scroll bar. On a touch-first device I agree it is more intuitive for touch interactions up/down to start a scroll. Whether a button or a custom control, you can't have scrolling disabled as soon as PointerPressed is invoked as the user would just think things are broken. In my case it would actually disable scrolling from most places as most elements within the scrollviewer are interactive to some degree - so Uno is actually helping.

That said, unless UWP modifies it's behavior in this case for touch, there is a difference and it probably should be documented some place in Uno if it's not already. I'll do more testing to see if I get the PointerCancelled event as you recommend and that should solve this particular issue.

In both cases you should get a PointerCancelled, like on UWP (even if not expressly documented as it, it's what I saw). Actually Microsoft recommends to not assume to get a release for each pressed: "To function properly, your app must listen for and handle all events that represent the possible conclusions to a Press action, and that includes PointerCanceled." (Don't remember if UWP raises the PointerExited, but I think it won't ... pointer are really "stolen"!).

I don't assume I get a PointerReleased for every PointerPressed; however, I do assume I get a PointerExited for every PointerEntered. In Microsoft's docs here it explicitly states "PointerExited is fired for any case where the pointer had at one time fired a PointerEntered event, but some UI state change happens where the pointer is no longer within that element." You are correct the events are not always paired and we discussed that a while ago in microsoft/microsoft-ui-xaml#1959. Microsoft agrees it's a bug though if PointerExited isn't invoked.

Again, however, in UWP this code works correctly. It is only within the Uno Platform that the ScrollViewer is stealing the pointer and not firing the PointerExited or PointerReleased events. So the question is, did you test this specifically with touch interaction on UWP? Otherwise, I think there is a difference and Uno really should fire PointerExited as soon as the ScrollViewer steals focus.

Thanks a lot for continuing to follow-up with this!

[robloo]

@dr1rrb In both cases we were loosely discussing above, Uno invokes the PointerCaptureLost when the ScrollViewer steals focus. This allows me to clean things up correctly. Note that PointerCancelled is never invoked though.

I still had to change the code from UWP so am not entirely sure behavior is the same. However, as mentioned above, I have never tested the base UWP on a touch device so it's also entirely possible it behaves the same there. I defer to your judgement on this. If it looks good to you (and I'm assuming it does based on no further comments) then it's good enough for me. You have a lot more experience with this and I trust your testing more than mine :)

For all the issues discussed here I now have solutions so I'm closing this. Thanks a lot for taking a look!

[dr1rrb]

Hi @robloo, yes pointers with a touch device are signifcantly different than a mouse or a touchpad (what I assume you are testing with). A touchpad is really handled like a mouse, and if you "scroll" using 2 fingers for instance, it's actually raising some PointerWheel events.

I've validated on my surface, and I confirm that, as arguable as it could be, we only get a PointerCaptureLost on UWP when I touch and start the scroll:

[ENTER] PointerHandlingLayer-02543DEC @583.57,533.65 (583.57,533.65)
[ENTER] MainPage-03981CBC @583.57,533.65 (583.57,533.65)
[DOWN] PointerHandlingLayer-02543DEC @583.57,533.65 (583.57,533.65)
[DOWN] MainPage-03981CBC @583.57,533.65 (583.57,533.65)
[CAPTURE_LOST] PointerHandlingLayer-02543DEC @583.57,546.31 (583.57,533.65)
[CAPTURE_LOST] MainPage-03981CBC @583.57,533.65 (583.57,533.65)