Data virtualization is no longer supported after preview4

[jim-jiang-github]

Describe the bug
It seems that data virtualization is no longer supported after preview4? Similar to using the IList interface to implement data virtualization in Wpf. I wrote a demo to implement data virtualization and display very, very, very large amounts of data.
Is there a new way to implement the function of data virtualization after preview4?
To Reproduce
Mini repo
data_virtualization.zip

Expected behavior
Can support data virtualization in preview7
Screenshots

IList Can support in preview4 like this:

bandicam.2023-06-04.13-40-28-606.mp4

Desktop (please complete the following information):

• OS: [e.g. Windows, Mac, Linux (State distribution)]
• Version [e.g. 0.10.0-rc1 or 0.9.12]

Additional context
Add any other context about the problem here.

[maxkatz6]

I am not sure if hack with int.MaxValue length is was intended to be supported.
More typical data virtualization can be implemented with handling scrolling events and adding more items when scroll is reaching specific point.
Also, you have not mentioned what exactly is happening on rc1, but I suppose avalonia just can't handle this int.MaxValue.
cc @grokys

[jim-jiang-github]

I am not sure if hack with int.MaxValue length is was intended to be supported. More typical data virtualization can be implemented with handling scrolling events and adding more items when scroll is reaching specific point. Also, you have not mentioned what exactly is happening on rc1, but I suppose avalonia just can't handle this int.MaxValue. cc @grokys

We don't need int. MaxValue so much, we only need to give a larger data to find that there is a big difference between preview4 and later versions. In preview4, Listbox will only get the data displayed on the current page from IList, and then in the preview4 version No matter how much data Listbox currently displays, it will get all the data of IList. You can see the demo I submitted, which has examples of two versions.
data_virtualization.zip

[jim-jiang-github]

I am not sure if hack with int.MaxValue length is was intended to be supported. More typical data virtualization can be implemented with handling scrolling events and adding more items when scroll is reaching specific point. Also, you have not mentioned what exactly is happening on rc1, but I suppose avalonia just can't handle this int.MaxValue. cc @grokys

To add, there is already a problem with the preview7 version

[jim-jiang-github]

Listbox virtualization seems to fail after preview4???

[rabbitism]

From what I observed, Listbox is obviously virtualized, no matter how many items in the collection, the ListBoxItem in logic tree remains the same. But virtualizing stackpanel is trying to realize all containers in memory. But virtualizing stack panel is trying to calculate all snap points. Memory is consumed by these snap points.

[maxkatz6]

@emmauss I think knowing all items count shouldn't be required for uniform snap points.

[emmauss]

VirtualizingStackPanel doesn't use the items count to compute both regular snap points, only irregular snappoints require items. By default, snappoints are irregular. If they are certain all items are uniform, they would have to set either AreVerticalSnapPointsRegular or AreHorizontalSnapPointsRegular to true

[rabbitism]

@emmauss I tried to do so but then items are misplaced. Offset errors are accumulated.

[emmauss]

@emmauss I tried to do so but then items are misplaced. Offset errors are accumulated.

That's most likely an error in realization. @grokys may have some insight into this.

[grokys]

This is actually due to the new renderer which uses float precision. If you look at the position of the items in DevTools, they're laid out at the correct position because the layout system uses double precision. However because our compositing renderer is based on Windows.UI.Composition which uses float precision, our renderer also uses float and for good technical reasons I believe. If you try something similar on UWP/WinUI you will also see a similar problem.

I think we may need a separate virtualizing panel for large lists which uses logical non-smooth scrolling so that we don't hit this issue, or maybe we can switch to double precision in the renderer - @kekekeks would have thoughts on that.

(As a clarification - this is a response to the "items are misplaced" part of this issue, not the data virtualization part)

[grokys]

Ok, after discussing this with @kekekeks we're going to see if we can make the renderer work with double precision before 11.0 final.

[jim-jiang-github]

Is there any workground that can solve this problem in rc1 version? I hope to display a large amount of data in the listbox. The current estimated number is 1000w pieces of data.

[timunie]

@jim-jiang-github I think the "easiest" workaround for now is pagination in the VM. May not be most convinient, but the fix we have in mind is not that straight forward afaik.

for example: https://github.com/reactivemarbles/DynamicData#virtualisation

[hez2010]

Windows.UI.Composition which uses float precision

Some GPUs don't support double precision at all and I believe this is an important reason why W.UI.C chose float precision.

[grokys]

@emmauss regarding the data virtualization, it's not great default behavior that the default value for AreVerticalSnapPointsRegular causes all items to be enumerated when the ItemsControl is shown. IIRC this setting is taken from UWP, does the same thing happen there?

[emmauss]

@emmauss regarding the data virtualization, it's not great default behavior that the default value for AreVerticalSnapPointsRegular causes all items to be enumerated when the ItemsControl is shown. IIRC this setting is taken from UWP, does the same thing happen there?

WinUI have both AreHorizontalSnapPointsRegular and AreVerticalSnapPointsRegular return false for StackPanel and VirtualizingStackPanel. The implementation for both properties and the snappoints methods are not available as the source isn't public.

[Developer-Alexander]

This is actually due to the new renderer which uses float precision. If you look at the position of the items in DevTools, they're laid out at the correct position because the layout system uses double precision. However because our compositing renderer is based on Windows.UI.Composition which uses float precision, our renderer also uses float and for good technical reasons I believe. If you try something similar on UWP/WinUI you will also see a similar problem.

I think we may need a separate virtualizing panel for large lists which uses logical non-smooth scrolling so that we don't hit this issue, or maybe we can switch to double precision in the renderer - @kekekeks would have thoughts on that.

(As a clarification - this is a response to the "items are misplaced" part of this issue, not the data virtualization part)

With how many items the issue becomes visible?

[kekekeks]

Some GPUs don't support double precision at all and I believe this is an important reason why W.UI.C chose float precision.

It's not about precision supported by GPU, it's about calculations where values cancel each other out. The most simple case would be:

  <ScrollViewer>
    <StackPanel>
      <Button Margin="0 1000000000 0 0">0</Button>
      <Button>1</Button>
    </StackPanel>
  </ScrollViewer>

So when the ScrollViewer is completely scrolled down, the StackPanel would have a negative Y offset of mostly the same value as the button margin, so the final offset from the window origin would be quite small. However due to the loss of precision during the final offset calculation the buttons would appear overlayed.
The same would happen with double, however it's way more precise (2⁵³ vs 2²³).

With how many items the issue becomes visible?

Assuming the offsets are integers and 96DPI, the loss of precision starts at 2²³ (8388608) pixels. It takes more for it to become noticeable.

[kekekeks]

You can see a similar issue with web browsers, e. g.

<div>
  <div style="margin-top: 57000000px; width: 100px; height: 10px; background:red" ></div>
  <div style="margin-top: 10px; width: 100px; height: 10px; background:blue" ></div>
</div>