[BUG] Drawing Tools Stroke Latency (Lag) & Interpolation Issue

[Jose-Moreno]

--Issue Overview--

There are two related problems that happen in both vector & bitmap layers.

First the stroke latency: basically every drawn stroke lags behind the cursor or pointer.
Second, the Stroke interpolation: Speed is a factor that modifies how the lines are interpolated, thus giving it enough precision to mimic the stroke that was input from a graphic tablet.

Both problems become altered when the Pencil2D window layout, and thus the canvas size are reduced as shown in the video demo.

It's important to say that in previous version 0.5.4b both Stroke Latency & Stroke Interpolation were present but it was to such a minor degree that they were barely noticeable for the regular user.

--Video Demo--
https://www.youtube.com/watch?v=9pTepQSgLCQ&feature=youtu.be

--Steps to reproduce--

Stroke Latency & Stroke Interpolation

1. Open Pencil2D
2. Select a drawing tool (i.e Pencil)
3. Select either a Vector or a Bitmap Layer
4. Try to draw as "fast" as a normal stroke with pencil and paper would feel like
5. Notice both the stroke lagging behind the cursor and the poor interpolation
6. Reduce the Pencil2D Window Layout to Half it's size (make it to occupy half the screen)
7. Try the same kind of strokes as in step 5.
8. Notice the reduced latency and interpolation problems.

--Expected Results--
Stroke Accuracy ( Latency & Interpolation ) should be independent of:

• The Window layout size
• The Canvas Size
• The Document / Camera resolution

Purely from a "User Experience" standpoint the stroke drawing interpolation method should not be apparent to the user, or it can throw off their drawing perception with the program and limit their freedom while creating artwork.

--System Information--
Pencil2D Nightly Build 15 Jan 2016 / Nightly Build 16 Feb 2016 (Run & built with Qt Creator 3.4.2 based on Qt 5.5.0)
Windows 7 64bits SP1
ATI Radeon HD 6770 1GB
AMD Phenom II X4 965 BE 3.4 GHZ
4 GB RAM

[feeef]

Running Pencil2D from QTCreator, you may have latency problems comming from debug data being processed all the time. Especially debug messages. You should try to run the same test with a release binary, away from the IDE. Wait for the next nightly build.

[Jose-Moreno]

@feeef Thanks for the suggestion, when I recorded the video I was coincidentally testing the latest source code using QtCreator, however this is NOT a "new" problem. I've already tested and confirmed this problem in previous Nightly Builds from later November 2015, so it's not a coincidence :)

The "interpolation" issue began to become more apparent around the Nov 29th 2015 Build onwards, although the Latency was still similar to previous versions. From that point up to JAN 14 2016 Nightly Build Binary, It has become increasingly apparent.

I know it's not something "critical" or a life or death matter either haha, that's why I was actually waiting to properly report this ever since last year just to make sure it wasn't just my computer :P

Besides although I was comparing the "latest" Nightly build to version 0.5.4b, since it somehow works "better", it suffers from the same problems, just to a minor degree.

To be clear on this, I've run Pencil2D January binary without any other program or contraption and Latency is still noticeable, sometimes the stroke doesn't even "catch-up" with the cursor. But truth be told the latency issue doesn't worry me that much, but I'm much more preoccupied with the interpolation issue. That is, how the drawing engine chooses to handle drawing from one point to the next when creating continuous lines.

I hope other users can maybe contribute with answers to the following questions:

1. ¿Have any of you noticed this same problem?
2. hich OS and which version are you using then if you don't experience these issue(s)?.

If Linux and Mac versions are not suffering this at all, it might be a windows related problem (I ruled out being wacom drivers because it happens with the mouse as well), but if other windows users say they don't see the same issue, or if the issue happens for some, then I would like to ask our users to attentively record their current pencil builds and submit their observations to this thread for the sake of comparison.

I'll wait for the next binary to test it out, of course, but I know this is one of those problems that are extremely difficult to fix, so I just wanted to bring awareness early on to this subtle but important issue so we can work on fixing it for a future version :)

[feeef]

I will have a look at this bug as soon as I have some time because it is an important issue and we have to make sure that this software is fast and works well for everyone.

It may not be difficult to fix. We have to check how much calculation is done between the mouse move event and the screen output.

If you say that when you resize the window to a lower resolution the problem is going away, it means that the refresh event is doing something on the entire surface of the screen, which is pointless while drawing a stroke. Only the area where the stroke is made should be refreshed.

I know the paint event of the ScribleArea (canvas area) widget has a lot of code that is being called contantly. It should be cleaned up.

[chchwy]

This is not a new issue, but i don't have time to solve it before, sorry ;p

The last point of stroke sometimes delay about 2 or 3 mouse event point, I guess the stroke smooth Interpolation case the latency.

[feeef]

I have added the "bug" label as this problem makes Pencil2D useless for many users.

The problem mainly comes from the way the scribble area is rendered while drawing to to it. It uses the QWidget::paintEvent method to redraw its content which constantly redraws the full surface of the widget.

I propose to optimize this by using a tiled area that renders only what is being drawn. MyPaint works this way so I plan to implement libmypaint within Pencil2D. A couple of months ago, I had started a libmypaint to QT interface for that matter : https://github.com/feeef/QTMyPaint.

I will use that interface.

[feeef]

I have started to work on the mypaint implementation so I thought I would give you some update.
I haven't done much yet but I have finished integrating the QTMyPaint library into the project and started playing with the ScribbleArea to render the stroke with mypaint.

It works and it is indeed much faster than it was previously. I tried it on my old powerbook G4 and it had no latency at all. Even full screen.

However, I plan to do this integration the right way and will have a lot to refactor. I will completely get rid of the previous bitmap brush engine and remove everything that creates stroke output from the ScribbleArea. Every stroke visual output will be handled by libmypaint. It will simplify the code a lot!

It represents quite some work to be done on my free time so I changed the milestone from 0.6 to 0.7.

Anyway, I am very excited! I think that this will be one huge improvement for Pencil2D. First, we won't have to worry about the brush engine anymore plus we will be able to easily integrate all brushes that work with mypaint. We will have to think about how that will work in term of UX.

[Jose-Moreno]

@feeef That sounds amazing François! As always thank you for your dedication and hard work. I'm glad it is possible to integrate it with Pencil. We all knew it was a time consuming task but for now I think it is the best course of action to be honest. Just as other drawing programs got an animation module, now Pencil2D, and animation program, will get a proper drawing module 😃

As long as this can be solved, as well as stabilizing the program so it won't suffer crashes I think the program will be fine in the long run. Keep up the great work, if you need anything please let us know!

[feeef]

I have been a bit silent lately because I have been busy with other projects but I managed to find some time to work on the mypaint integration on my free time so here are some news.

I have been mainly working on the small mypaint to QT interface that I will use within Pencil2D. I have completed the features that I planned to have on this lib to be required by pencil. You can have a look at the lib here : https://github.com/feeef/QTMyPaint
You can build it with QTCreator and the demo app should launch, showing you an overview of what the new pencil brush engine should be like.

BTW, I plan to include all the brushes from this demo in pencil to have them all available to the user. I plan to organize them in sections corresponding to the tools already existing in pencil (sketch, ink, paint, erase and smudge) so that when you select the pencil tool, you get a list of all available pencils and sketching tools. The same goes for the eraser, smudge, pen and brush tools. All these tools are going to work out of the box now. It is going to be quite amazing actually! :)

I still have some work in order to implement all this in pencil but I know exactly how I am going to do it. Only time is missing at the moment but as soon as I get some I'll spend it on this.

This update is going to be really amazing! A big jump for Pencil2D. The users are going to love it! :)

I can't wait to have something usable for you to test!

I'll keep you updated!

[Jose-Moreno]

@chchwy @feeef Fantastic news! Truth be told I've been quite busy as well, plus I had to format my computer, so I've had issues resuming work on the github site again. However I'm trying my best to at least keep track of what is happening and assisting over at the forum normally to see if it breaks or what.

The categorization effort sounds like a marvelous idea! Thanks a lot for your hard work as always 😄 I wish I could help with anything about the code. If only there was a way to understand the Qt code and use my primitive python skills to do something ... sigh.

Anyway take your time, we are indirectly rivaling OpenToonz now. Many new users have found themselves having to migrate back to Pencil2D due to the complexity of OT. So as advertised I still think we have a positive chance to affect the open-source animation market for good.

On an important Side-Note:

There are other problems I have found through (heart breaking) user reports about Pencil2D apparently terrible memory management and (possibly) segmentation faults. I've seen corrupted animation files, both PCL and PNG files because Pencil2D crashed which I'll share in due time.

So this is a worrysome problem. I have not had any of these problems because I've never done any animation longer than 3 seconds (72frames), however some users did sequences that went beyond the 200 frames, then crashed and lost all their work.

In the past I have investigated lightly how Pencil used it's memory making 300+ frame animations use about 80% of a 4G RAM setup, and with the newest Nightly builds how still the memory "garbage collection" was improved, but is still not ideal. So I think we have to review the work on that after you crown the MyPaintLib milestone, because stability should remain as our #1 priority if these problems continue to persist.

Have a happy week everyone!

[feeef]

Just to give you some news about my progress in the MyPaint integration. I am currently very busy with work and didn't have much time to spend on this. I will keep being very busy until the end of the month. Then, I should be able to find some time for Pencil2D in June.

I will keep you updated anyway! :)

[puetsua]

I was bored and decide to trace this bug. xD

I have investigated this issue and found a related commit 09d96fe

• Jagged strokes fix: added a subtle anti-shaking

Low speed brush strokes are now smoothed. I can't believe I a simple
interpolation division / 2.0 was required. How I like small changes like
this one :)

In strokemanager.cpp:

void StrokeManager::mouseMoveEvent(QMouseEvent* event)
{
    QPointF pos = getEventPosition(event);
    QPointF smoothPos = QPointF( ( pos.x() + mLastPixel.x() ) / 2.0, ( pos.y() + mLastPixel.y() ) / 2.0 );

    mLastPixel = mCurrentPixel;
    mCurrentPixel = smoothPos;

    if ( !mStrokeStarted )
...

It seems that smoothPos delayed the position during mouseMoveEvent. Replacing smoothPos with pos should be able to fix this bug. But we might lose an interpolation enhancement. .3.

[feeef]

To give you some more updates about the mypaint integration. Things haven't progressed much for the last few month but don't worry, I didn't abandon the project and still plan to complete it.

I have a new position as director of creative at Purism since January and have much less time to work on Pencil. With time, I should be able to organize myself and complete this mypaint integration.

Actually, before being too busy I did a lot of work and managed to have the bitmap layer to use mypaint to draw. It worked very well and I was stuck with the zooming feature that I couldn't manage to have working correctly. As soon as I find some time, I will work on this issue and once fixed, I should be able to have a first working, still pre alpha version of Pencil using mypaint.

For some reason, I am unable to connect to the forum so I'll keep posting updates here.

[Jose-Moreno]

@feeef Hey man! congratulations on the job. I'm glad that you're still thinking about Pencil2D haha. Thanks for letting us know. To be honest there's very few people working on the project, including yourself. We're all busy for all I know, and the website is kind of locked down. I'm trying to manage the social media pages but there's so much I can do.

For now just try to get yourself comfortable and when you have some time keep up with the great work you're doing so far! In the end hopefully everything will work alright. Take care!

[feeef]

It may take some time but as soon as the mypaint version will be out, I am sure that it will get much more interest from the artists.
Currently, I moved to Krita, which is pretty amazing but I still plan to make Pencil my main animation tool because I love this software.

[MrStevns]

I've done exactly what @hankofficer mentioned, however even after removing the smoothing, the stroke interpolation is unchanged when using a tablet. When you use a mouse, it's gone.

[MrStevns]

https://bugreports.qt.io/browse/QTBUG-44964
it seems that qt automatically compresses the amount of input it gets from the mouse and tablet, this is not good since we need as many inputs as possible to get a smooth stroke. On average the mouse skips 13 pixels. The tablet (mine at least) sends more inputs but it's still not enough, here it skips on average 16 pixels, but it is still smoother because there are in total more values to interpolate between.

The solution however seems to be rather non cross-platform, as you have to work with native event types.

[chchwy]

This issue has 2 parts, one is Qt mouse event compression, that's true.

But another more important aspect is performance, Pencil2d is just not fast enough to handle so many events. We did disable the event compression on Mac in previous version, but ended up we got a extremely slow pen stroke.

We have to find out what pencil do between 2 mouse event, and minimize it.

[feeef]

@chchwy, the answer to that is what I had previously investigated. Between 2 mouse events Pencil2D redraws the entire canvas surface through the paintEvent method in the ScribbleArea. It is why, when reducing the window size, the application is faster.

The solution here is to implement tiled surface. Where only a small area of the canvas is redrawn. We can implement it ourselves or complete the integration of MyPaint.

The second option is, to me the safer and the faster as I have already completed most of the work. I only struggle with implementing zooming and panning, which seems to be very tricky. The rest should be straight forward. All in all, this integration removes a lot of code from our current version of Pencil2D. It should make things cleaner and easier to maintain.

Here is my current progress : https://github.com/feeef/pencil/tree/mypaint

[chchwy]

@feeef Pencil doesn't redraw whole canvas anymore, I fixed that. Pencil2d only redraw a small rectangle that covers 2 mouse positions. But even do so, Pencil is still not fast enough.

[feeef]

ok @chchwy sorry, I didn't know you changed this behavior.

[MrStevns]

@chchwy Ahh I see, so it's related to SetMouseCoalescingEnabled in macosx.cpp. Enabling coalescing not a good solution though, it must be possible to use all the events at once, store them in a queue or something. This developer guide from wacom talks about possible solutions: http://www.wacomeng.com/mac/Developers%20Guide.htm
search for "Performance Issues" to get to the interesting part.

[chchwy]

@feeef no need to say sorry, I just want to point out that there is something else slow down the stroke.
As performance issue, we probably have to do a profiling to get the root cause.

[feeef]

ok @chchwy, @candyface, unfortunately, I can't reproduce this bug on my side as both my tablets are fast with Pencil2D. I let you guys handle that issue.

[Jose-Moreno]

[2018 Review] On the latest version this has improved considerably but you need a decent computer to let the CPU handle those drawing events.

For context, I have a way to underclock my quad core rig "on the fly". So using stock voltage values to get my regular CPU speed (3.2 GHZ) allows for the drawing to work fine and the stroke lag is gone. If I underclock my CPU to simulate a lower GHZ rate "rapid drawing" in Pencil2D becomes impossible (the lines can't quickly calculate the position and everything becomes straight; cool effect though!).

Thank you once again for taking your time to review these issues in the past. I believe the discussion you guys held before is still valid, so let me know if you want me to keep this open or close the issue as it's been "solved".

Edit: On a closer inspection I don't think this issue is really solved completely. According to some discussion in the DISCORD server the line interpolation problem is still happening too often with normal drawing.

[Kaikogf]

I don't recall where I posted a similar issue, lagging when drawing diagonal lines exclusively.

Just to let you know, today I tested the last code from Github and also the last code using Mypaint' engine,, and I experienced no lag when testing the latest.
Maybe it can bring a new light here

[Jose-Moreno]

@Kaikogf Hey, sorry for the delay. Yes it has gotten a bit better, but this is a problem with the original Pencil2D brush system and it depends a lot in the processing power from the CPU.

However there is a baseline for this problem to occur and it requires very deliberate motions and timing that happens too often when drawing. That said MyPaint does not have this issue when I tested last year with the original hardware I i used for this report. As such I'm afraid this will only be solved properly when MyPaint brush engine is implemented, not before 😓

[scribblemaniac]

There is some caching added since v0.6.4 which should improve the speed of brush strokes rendering, particularly when you have many layers. Issues will remain when drawing very large strokes and probably under some other circumstances.

[Kaikogf]

Ohhh I see...
Thanks for the reply

Indeed, pencil2D can better handle long strokes... 👍

[Jose-Moreno]

@Kaikogf @scribblemaniac Haha ... I had an inkling that this wasn't going to be so easy 😅 I tested the latest build from June 17th using:

• Windows 10 Pro Version 1909, Build 18363.900
• Wacom Intuos 5 Touch PTH-450 with Driver 6.3.38-2 from 12/17/2019
• Hexa-core AMD FX-6300 w/ 16GB RAM and Nvidia Geforce GTX 1050 4GB

The current problem for me is as follows:
When using the compatibility tab option with Windows 7, to enable Pressure, drawing normally is severely crippled by lag, as described in this bug. All the lines are VERY slow. Even drawing small strokes makes the drawing slow. As if the hardware is struggling to calculate the lines.

Mind you I have a better PC than I had before when i first submitted this issue, so it shouldn't be slower.

When the compatibility flag is removed, this improves the issue a bit but it's still comparatively worse than how drawing behaves in the stable version 0.6.4. This worries me quite a bit.

Since Kaiko has obtained Windows 10 recently I'd like to ask him if it's possible to test this and provide a report both with and without windows legacy compatibility enabled.

Also if you both of you can use different tablets to test, it would also help establish a baseline as to which hardware is more prone to fail with the latest nightlies in different OS so we can at least have enough information to provide support for future users that have similar issues when the software releases.

On my end i'll retest after I update the tablet drivers to the latest version and compile Pencil2D myself to test on either build. That said I'm starting to doubt doing any of this is going to bring the issue back to the level it was before (where drawing was still usable), I'll look into it over the weekend and report back when I can 😔