Is it possible to get the dragging performance of the image (rectangle) following the mouse as close as WPF?

[tictac1234]

Hello.
Is it possible to get the performance of the image (rectangle) following the mouse as close as WPF?

I am trying to create a multi-platform game tool in AvaloniaUI.
But when dragging the mouse to move an image or rectangle,
I feel a significant delay with AvaloniaUI.

Performance Test of WPF and AvaloniaUI.
( Both on Windows PC )

bandicam.2023-12-30.22-58-01-777.mp4

The code is quite simple, as shown below.
(WPF also uses DrawingContext for drawing as well.)

MyCanvas.axaml.cs

public partial class MyCanvas : Control
{
    Rect _rect;
    Point _rectPoint;
    Point _clickPt;

    bool _isHit = false;


    public MyCanvas()
    {
        InitializeComponent();
        InitializeRect();

        this.PointerPressed += OnPointerPressed;
        this.PointerReleased += OnPointerReleased;
        this.PointerMoved += OnPointerMoved;
        this.ClipToBounds = true;
    }

    private void InitializeComponent()
    {
        AvaloniaXamlLoader.Load(this);
    }

    private void OnPointerPressed(object? sender, PointerPressedEventArgs e)
    {
        _clickPt = e.GetPosition(this);
        _rectPoint = new Point(_rect.X, _rect.Y);

        if (_rect.Contains(_clickPt))
        {
            _isHit = true;
        }
    }

    // After moving the rectangle with the mouse, Redraw this control.
    private void OnPointerMoved(object? sender, PointerEventArgs e)
    {
        if (_isHit)
        {
            var pos = e.GetPosition(this);
            var add = pos - _clickPt;
            _rect = new Rect(_rectPoint.X + add.X, _rectPoint.Y + add.Y, _rect.Width, _rect.Height);

            this.InvalidateVisual();
        }
    }

    private void OnPointerReleased(object? sender, PointerReleasedEventArgs e)
    {
        _isHit = false;
    }


    void InitializeRect()
    {
        _rect = new Rect(0, 0, 64, 64);
    }


    public override void Render(DrawingContext context)
    {
        base.Render(context);

        var brush = new SolidColorBrush(Colors.Blue);
        context.DrawRectangle(brush, null, _rect);
    }


}

Is there any possibility of improvement?
If you can make it a little faster, I would like to know how to do that...

Thank you.

[rabbitism]

Can you try to reuse the brush instance?

[tictac1234]

Thank you for your response.

Yes, I did it.
But the performance hasn't changed.
To Create the brush instance every frame, in itself, would not lead to any performance degradation.
So I think this problem runs much deeper than that...

The code I tried.

public partial class MyCanvas : Control
{

    Rect _rect;
    Point _rectPoint;
    Point _clickPt;
    Brush _brush;

    bool _isHit = false;

    public MyCanvas()
    {
        InitializeComponent();
        InitializeRect();

        ...
    }

    // After moving the rectangle with the mouse, Redraw this control.
    private void OnPointerMoved(object? sender, PointerEventArgs e)
    {
        if (_isHit)
        {
            var pos = e.GetPosition(this);
            var add = pos - _clickPt;
            _rect = new Rect(_rectPoint.X + add.X, _rectPoint.Y + add.Y, _rect.Width, _rect.Height);

            this.InvalidateVisual();
        }
    }

    void InitializeRect()
    {
        _rect = new Rect(0, 0, 64, 64);
        _brush = new SolidColorBrush(Colors.Blue);
    }

    public override void Render(DrawingContext context)
    {
        base.Render(context);
        context.DrawRectangle(_brush, null, _rect);
    }

}

[stevemonaco]

Seems like the input is a few frames behind the render. This happens in the WPF video, too, but it's not as pronounced, especially with slow movements. I have a similar result in my own rendering (which uses SkiaSharp and TopLevel.RequestAnimationFrame), but I need to aggregate change events into a single pre-render pass to a buffer, then InvalidateVisual to actually render it. Which happens at some point further in the future when the cursor has already moved some distance. Regardless, it still renders at 60fps, so it's not a performance issue, but some latency issue. Probably some architectural issue in the timeline between the OS updating/rendering the cursor, Avalonia receiving/forwarding input, and actually presenting the render to the screen.

You can't query mouse state in Avalonia, so you can't do the update portion in the render method itself.

[tictac1234]

Thank you for response.

What you said is simply that Avalonia is slower than WPF to detect mouse input, right?
In other words, we can't improve the recognition speed any further?

[damian-666]

im not looking at the code, just from the symptomology ( and your OnPoiterMoved) , since it might be up to the application to speed this up .. but you are showing a one rect case..

its a holiday so im just throwing out some general ideas might help.. been makiing ides adn game engines for far too long...with wpf adn monogame in it..

so finally got to multtiplatfrom but some of this is jsut for windows.. i spin a background thread to do physics and spatial indexing, between visual frames, then that worker thread, i poll the mouse and touch collectin using monogame in each of these updates. i think it uses SDL.. at 800 fps if needed.. that could help.. i avoidf using asyncs because i dont understand it, and stay off the ui thread it troublesome. there are locks here adn there, many regretable ,all dpeend on that timer resolution. the touchcollection is not concurrent. i had some excpetoins reading touchcollection.. buit i could ingore them. mabye just for mouse you dont need it.

you might have to try this is you poll like that...
[System.Runtime.InteropServices.DllImport("winmm.dll", EntryPoint = "timeBeginPeriod")]
public static extern uint timeBeginPeriod(uint uMilliseconds);

this is whaver ist probably lagging:
private void OnPointerMoved(object? sender, PointerEventArgs e)

if (_isHit)
{
var pos = e.GetPosition(this);
var add = pos - _clickPt;
_rect = new Rect(_rectPoint.X + add.X, _rectPoint.Y + add.Y, _rect.Width, _rect.Height);<**--- REUSE a rect, dont incur garbage this i will not fix it though. ** collecton in render or update..60 fps loops.
this.InvalidateVisual(); <-- super expensive i guies trigggers a whole layout messauge and updater..might affect parents. needs a guru to opiion maybe
}

his being a one rectangle case.. something is wrong that might be a quick fix.. cann you just do a break all.. and build avalania and see where it hangs...? its prbably at the bottleneck.. after a few break alls...

wpf/ silverglith.. has dirty clean reactangles monitored and some gpu acceleration and XNA feature brought in, thats super difficult also and might affect this case.
im guessing threadingt / blockng related...

the code probaby is direct X related. not portable.

he the bigger picture is the functinal overlap. there the font enginers, embedded game render views, 3d views . get redundant .. for for a 100 items on a canvas avalonia shouldnt lag.

a hit test requires a tree structure spatial index tree.. like a quadtree. my guess is Avalonia doesnt have it and maybe shouldnt.

a quick/ dirty is to sort with a hash table by the intervals on the X values  of each testable.     sort the map, by just the X left values.   in the case of say  an audio signal  or columns ing a height map, that is how you would do that. 


so   tree is fastest lookup.      he silverlight code and the wpf is open sourced but when its a lot of objects its takes long to update adding an item.  slow to  move stuff.  they balance the tree periodicailly.

or, a hash table or sorted disctionary, , lots of memory, fast to move stuff around... both can't be generalized without a FFT pass., slow, periodic.. . for sometimes its varies more on X, or sometimes Y. ( list) . its videogame engine stuff and i put a spactial index from Velcocrophyics on top for collision detection.

a gpu hit hit test can be done also, by mapping the ids from gpu objects, to the pixels, into the frame buffer, coded in a margin. and doing a texture lookup.

problem is avalonia is general . and for annotational content...a big tree   overkill. 
   so i dont know how to  even advise.    for me..  make a game in silvlight light and realized it had two hit test trees  doing the same thing and it was dog slow.   so i started doing immediate render and using the hittester in the physiics engine.    
   
   to copy wpf is hard becasue all the platfroms ui thread is so nuanced.. so i try not to block it.. mess with it.. or use any locks, just copy  and swap stuff its cheap and fast. usually.     with android securty about the frame buffer tho.. its tough.. they dont let you see it.   but on desktops you can.  its because an app can peek where it shouldnt be...that "allow apps  to write over other permission" youi dont want to git iti to some videogame..   cause it can see you log into other apps..  sometimes  i wish windows phone succeeded.. dindt have mutliwindows.

[timunie]

@tictac1234 make sure to do the performance measures only in release mode. Debug-mode adds a lot of overhead.

[timunie]

I did some tests using your code and for me publishing the App as Ready2Run feels snappier.

[tictac1234]

@timunie OK, I tested in release mode with the same results.

Then I uploaded a project comparing WPF and Avalonia.
https://github.com/tictac1234/WPF_vs_Avalonia

You can try it if you want.

Avalonia : WPF_vs_Avalonia/Avalonia/AvaloniaApp_Win.sln
Wpf : WPF_vs_Avalonia/Wpf/Wpf.sln

[timunie]

thx, will give it a try. Even if it is not for you anymore, it's always good to know areas which could be improved. Thx.

[tictac1234]

@timunie I'm sorry late replying, thank you too.
I think AvaloniaUI is really great.
I tried many samples and they are very detailed and I felt the developer's strong desire!

I will not hesitate to use Avalonia in the future for any application where I don't care about response, such as game tools.
Thank you.

[damian-666]

"(https://github.com/timunie) is probably right, this may be due to gestures." sounds possible,

i catch thes by hitting break all.. instead of profiling... when doing a laggy thing. itgs almost alwyas stops there ..so i with monogame have to process my own gestores...

Also Nez and Monogame have touchcollection code you can look at ... mabye that can be sped up.. on iphones touch is up to 800 fps..

https://joltfly.com/mouse-latency-test/ on mine if it put dynamic 60 /120 ists alwsys 16 ms, or 60fps.

i set to 120.. i get super fast respeose from touch., mouse and Surface stylus.

but wiht games, some users people complain they have to touch screen all the time to get 60 fps updates on phones .. my surface laptop is supposd to be is 120 hz touched...60 when not.. its will affect the refresh rate also.. thats not great. yo can barely see 120 hz unless your a navy pilot. 30 is too slow for eyes.. and 60 is fine but mouse adn touch must be at least 60 hz or ts laggy..120 is overkill.. you dont need to draw every frame to get a nice response.

i do spin a bk tread as i said.. settig timerselution doesnt affect all apps anymore you dont have to set it back..
but it will drain battery so if not helpoing id leave it as defaut.. but you cannt even measure a sleep (1) wih rust from 0 to 16 ms. if you need to..

. so that why i spun a bk thread to poll the touch collectoins... i dint want to draw any faster tghan the monitor request, 60 hz is fine. in monogame its called vsync.. you get a draw call when the monitor wants it, so i used a prpoducer consumper pattern.. copy buffer, draw last good one.. again i dont know it approaprate but i got no locks working on adroid ever. so i used immumatbles. and copy n net8 is fast..

mousing might have a sort of event or interupt... not sure.. im polling.. state changes

[tictac1234]

sorry I missed your reply.
I tried on the site you gave me.
But why does the mouse detection in the browser seem so slow to begin with?

For example,The Electron web app is also quite slow…

[damian-666]

mine is the same as my refresh on Chrome using Windows 11 bloated with insider junk. Beta level.

. It's about 16 ms.. then 8ms if I move to the 120 Hz refresh. all I need is 60 fps, gtood enough
one thing, I try pen, optical mouse, then turned on trackpad and it went really slow. drawing the line seemed to bog it touch is laggy but its i have llos of junk running.. I closed all my Chromes now I can use it... I use GameMode on my Projector I get it 60 Hz I'm ok.. 30 Hz is not ok. .. I need a high quality HDMI cable. playing with some of the settings in the mouse latency game gave me some different info.. some effects. try turning off HDR also. try a different user, not admin maybe..

I suspect some driver issue that isn't the same with Avalonia as WPF. . the main different is maybe DirectX? is Avalonia doing that for you? I though you might need to set it.. to DirectX.. in your code.. maybe your OpenGL driver is bad, or set wrong.. or some other issue because the web thing uses angle or OpenGL or WebGL.. I wouldn't give up on

if you install vulkan on your PC will it help?.. I would try some more stuff before giving up on Avalonia, not dragging one block and being very laggy seems like an issue that would not be accepted to anyone..

I haven't tried avaloniaUI lately ..looking at the big picture and in a holding pattern, hovering

sorry #Offtopic, if can get moved to discussion if i should.
i'm just putting it here for now.. another 20 hour day, tired. i don't want to raise an official issue(s) its more of a question to the community if about ui design / direction and if someone has influence at msft .. the windows lead product designer. i doubt there is one. The best one resigned after win 11 move start to the middle, . and tweeted out why , i agreed w all his points, noone listened to him there.

its just looking forward. its very general .. but relevant to planning your projects.. and about Automation.

offline audio assistants are getting good...since this team is about generalizing the best UI layout system, but its i now to hard to use any UI layout system. driver issues, screen layouts, .. styluses, drag and drop / no code/ no code for windows scripting, customizability , ( i don't want to) its all pukeware.. its too hard to maintain. grown too complex.. The code /feature search, gives command line random access o+ UI discoverability is in devstudio.. but even that is too hard to use.. its in two panels. should be on a ribbon like start..
the scripting / batching they are fixing in a no code drag drop experience that is terrible.

the stuff is making a windows user have to be a programmer. spend 12 hours a day on pc and or phone. they don't save us time they kill our bodies. AI could make that not the case, mice and keyboard are 1970s tech. the billions were spend and MSFT paid most of the bills and they don't use AI to make a killer os and killer dev tools.. i mean talk over all pcs.. the last lead product designer tweeted why he left and its that UI is all different on everything. changes too much. etc. at least the basics should be the same.. gas on right , breaks left, flaps down is this way. etc.

I'm learning AI stuff and started in NLP and now its could be the best way to task to your pc. so trying to influence on voice input I never want to use a mouse or keyboard again. but i fear Microsoft is going to blow this opportunity so it's not gonna happen soon and it should already be done. given how complex it is .now it's build over the mouse and keyboard not going to the apps object model and DOM and documents and scraping all the test APIs that automation can use. . to do it right would have been easier.. the voice access is great but.. is not replacing mice keyboard our touch .. soon and I'm telling them how. I don't know if Avalonia is spyable UI with tooltips alot , and/ or have automation or testing apis, or has settings exposed or if that every application in C# . a public object model with all the commands and settings public, and the document context this would be for an external process..or some general plugin.

they are pushing for "AI first " and I pushing them so add context, ( if i'm focused on File menu it should not guess i'm saying something's that's not on there) scrape api's and param types and output , ORC and spy UIs options and tooltips., do the command to Automation and object model of the apps, via the com wrappers in Semantic Kernel. or silk or, pre-pass and index UIs, like Visual Studios code /feature search , using NATO commands Alpha Bravo Charlie for when it presents UI, as you say something like it.. Context aware to limit the choices, 2 way hits.. fast hotwords. mix with completion, and I won't need to use a mouse or keyboard . it's not accessibility it's to prevent disability, clutter , and UI tabs well , I don't have room anymore.. even on 4k running dev studio with all the features.. I want command query on my OS and all apps in a UI to me needs to be speak and command / query. NLP and LLMs, and general external process, implementation. not a plugin

unified across all the desktops. just mentioning this so that if you try voice on window11, talk to Chatgpt about it, see it correct your typos, take voice and take a app context snap shot and it will fix wrong guessers, , you might see that it's possible to do that. and with a dynamic mic and preamp it's amazing. if anyone has influence with Microsoft and agrees.. I will make tooling but I want to end up moving no touch , not even looking... then they can skip fixing the UI all the meetings .. it can stay a mess! you use it once. to discover the features. or for fine drawing and or when voice integration passes cant do some aspect..

[ds1709]

I have the same issue just rendering pointer (the crosshair is lagging behind the cursor).
Here's the minimal case to reproduce.

internal sealed class TestControl : Control
{
    private readonly Pen pen = new Pen(Brushes.Blue);

    private Point pointerPosition;

    protected override void OnPointerMoved(PointerEventArgs e)
    {
        base.OnPointerMoved(e);
        pointerPosition = e.GetPosition(this);
        InvalidateVisual();
    }

    public override void Render(DrawingContext context)
    {
        base.Render(context);
        context.FillRectangle(Brushes.Transparent, new Rect(Bounds.Size));
        context.DrawLine(pen, new Point(pointerPosition.X + 0.5, -0.5), new Point(pointerPosition.X + 0.5, Bounds.Height + 0.5));
        context.DrawLine(pen, new Point(-0.5, pointerPosition.Y + 0.5), new Point(Bounds.Width + 0.5, pointerPosition.Y + 0.5));
    }
}

[stevemonaco]

RequestAnimationFrame should be faster here. AFAIK, the best you can get with Avalonia is 1 frame delay between input and presenting the rendered visual. If you aren't careful, you'll have several frames, upwards of 5ish. Especially if you have an update cycle that takes input + invalidates during one frame, starts updating the next frame (or two), and some cycle of dispatches from state changes carry updates across a few frames. Since you can't directly query input devices in Avalonia, your last input is necessarily at least a frame behind. Higher frame rate monitors should have a smaller positional error delta than 60Hz.

[tictac1234]

Hello, I’m the person who originally posted this article.
It’s been almost a year since I started this thread, but the lag issue still hasn’t been resolved, has it?

Hmm... If this problem were fixed, I’d love to start using AvaloniaUI right away.
This lag issue is especially problematic when developing tools for games, as it severely affects usability. It’s truly disappointing...

With native AppKit and Metal on macOS, or DirectX on Windows, it’s possible to achieve very responsive mouse tracking when VSync is turned off. I hope AvaloniaUI can aim for something similar...

By the way, could the mouse lag issue be related to the use of Skia internally...?

Thank you.

[stevemonaco]

Short answer: no.

Long answer: Your sample doesn't use the compositor and invalidating the visual will add extra delay. And yes, those APIs will be faster because they're native APIs made for immediate mode rendering. Avalonia, like most GUI frameworks, is retained mode. You will see similar latency elsewhere, especially when using the naive approach.

Frameworks like UWP can have low latency via manual use of swapchains and even dedicated input threads: See the low latency input sample. Avalonia doesn't expose swapchains (yet?) so the best you can do is tie into the compositor. Unfortunately, the documentation there is effectively non-existent still.

I get the same mouse location in OnPointerMoved regardless whether I use Avalonia's position, SharpHook, or PInvoke GetCursorPos, so I suspect the latency is almost entirely on the rendering side.

[atticus-lv]

I use canvas to make this color picker, it seems fast enough for me (video in debug mode)

2024-11-25.06-10-17.mp4

private void PointerMoved(object? sender, PointerEventArgs e)
{
    var currentPoint = e.GetPosition(this);
    Canvas.SetLeft(ColoPickerPanel, currentPoint.X);
    Canvas.SetTop(ColoPickerPanel, currentPoint.Y);
}

<!-- color picker -->
<Canvas IsVisible="{Binding ShowColorPicker}">
    <Border Name="ColoPickerPanel" Canvas.Left="300" Canvas.Top="200"
            Background="#111111" Margin="5" CornerRadius="4" IsHitTestVisible="False">
    <!-- some ui here -->
    </Border>
</Canvas>

[ds1709]

This issue is noticeable when you move cursor slowly. Just set speed of the video to 0.25, and you'll see the lag.

[atticus-lv]

Unable to view the video,, Can I upload it again?

I can see the video clearly. Maybe switch to edge to check it

[atticus-lv]

This issue is noticeable when you move cursor slowly. Just set speed of the video to 0.25, and you'll see the lag.

yes, but closer to the wpf one. notice that in release mode is a little bit better

[tictac1234]

Thanks, I could see it in Edge.
I could not see it at all on Apple devices Mac,iOS.

However, the background is black, the popups are not colored, and the Mouse moves too fast, so it's hard to see the lag...

Can you upload that sample project?
Or if possible, like my very first video I posted,
If you can see the Mouse move slowly at first, you can see the lag.

If it's too much trouble, it's ok.

[tictac1234]

Also, it's hard to tell if you're moving the Mouse to the side only,,
Mouse movement to the diagonal direction is the easiest to understand.
I'm sorry for the excess...

[tictac1234]

Sorry, but on Windows it is much faster with the following settings!
It is an order of magnitude faster than what I had before!
However, you need to set VSync off for your app in the NVidia Control Panel.

public static AppBuilder BuildAvaloniaApp() => AppBuilder.Configure<App>() .UsePlatformDetect() .WithInterFont() .With(new Win32PlatformOptions { CompositionMode = new[] { Win32CompositionMode.LowLatencyDxgiSwapChain } }) .LogToTrace();.

But this kind of setup is impossible on MacOS, right?

[ds1709]

This options significantly reduces latency.