Roadmap of 0.6.x and follow-up tasks

[chchwy]

Hey guys,
@Jose-Moreno @candyface @scribblemaniac @J5lx @feeef @Kaikogf @gordielachance

In the days of waiting for 0.6 official release, it's time to discuss what we can do next in 0.6.x patches.

This roadmap will more focus on stability, robustness and performance issues. I would like to put new features into 0.7 rather than a 0.6.x patch. Still, any suggestion/discussion are welcomed.

0.6.1 Reduce memory usage

Currently, Pencil2d keeps all frames in RAM while loading a project. The upside is obvious, it's simple, efficient and intuitive to program. But on the other hand, the downside is obvious, too. It consumes a lot of memory. For example, a 200-frame animation may take up to 1.6GB ram if every frame is in full size of 1920x1080 (8MB ram each). The total memory consumption is considerable.

We've heard some cases said Pencil2D became very unstable after creating more than 700 frames. I suspect that's because Pencil2D has used up all the available RAM. Especially on a 32bit system, the limit of RAM is more restricted (on a 32bit Windows it's only 1.5GB per process as I know).

So the plan is to use the disk as much as possible. Pencil2D will keep maybe only 10 or 20 active frames beside the current frame and save all other frames to disk. I might use methods like LRU to decide which frames Pencil2D should keep, but the main idea is to reduce the number of active frames in RAM. It may cause a sudden short pause in a long frame jump (e.g. jump from frame 1 to frame 100) due to the disk loading time of images, but we'll see.

Related issues: #253 #520 #574

0.6.2 Crash recovery

Another advantage of writing frames to disk is, it will be easier to implement a decent crash recovery since almost all frames will not get lost when pencil2d accidentally crashes.

At the moment Pencil2D creates a temporary workspace in system's temp folder while loading a project. (The temporary workspace can be thought of as an uncompressed pclx) The temporary workspace will be wipe out when Pencil2D exits normally. So if there is something left, it means Pencil2D was not properly closed last time. And then we can try to restore the project in the next start.

Related issues: #782

0.6.3 Stroke performance

The issue has been long recognized but hasn't been resolved yet. There are some inefficient operations when Pencil2d handles pen strokes.

Ideally, we can try our best to reach the standard of some basic level WACOM tablet, which is 133 points per second. That means we have only 7 milliseconds to handle every mouse/tablet event.

As I mentioned in the previous link, there are 2 parts we can improve: the first is minimize the repaint area when users are drawing. (This might have been done but I'll double check it)

The second is preventing image copying during a stroke. According to a comment from @candyface, image copying happens more than 20 times in a single stroke, which is very inefficient.

While I'm working on this, I will refactor the code structure to be better suited to the mypaint brush lib integration which is in the plan of 0.7.

Translation

The online translation platform will move to Transifex as @J5lx suggested in #505. Please tell your friends or anyone who is still on OneSky, time to move to Transifex. The old OneSky site will be deprecated soon. (@Kaikogf I saw you there sometimes?) I will do a sync in a few days and close OneSky project.

Forum

Probably needs to think about whether to move to the new forum built by @J5lx. I'll create another thread to discuss it later.

Internal code changes

1. Replace the current compression library with a more lightweight zip library called miniz. It will give us a more reasonable project structure, also resolve the request of please allow using system zlib #530. Plus a simpler software license (maybe!) after getting rid of zlib & Quazip. But a compatibility and performance test is required.

2. Migrate our Unit Test Framework from Qt Test lib to Catch2. As I have been aware of some limitations of Qt Test lib, I will give Catch2 a go. It is also very lightweight (only 1 .hpp file) so that I can easily include it without a pain of library integration on all supported platforms. And hope the tests can reduce the workload of our beta testers. You can check the progress in this branch.

[Jose-Moreno]

@chchwy Thanks a lot for this write up. First of all let me state that I think what you have proposed serve as great targets, particularly considering the amount of grief users have had in the past, memory optimization it's a must at this stage. Crash Recovery is also a good one, because let's face it, no application is crash free, and this will go in hand with memory usage optimization.

However, I must ask everyone to please make sure and discuss the other proposals that were made to prioritize important issues. Just in case I color coded them to assist visualization in the following document:
https://docs.google.com/spreadsheets/d/1c2GOMLolfCu6qpwxZ-ZLcb_VIZUeeJd0P-PU8WzJZc4/edit?usp=sharing

I have enabled anonymous editing if you want to change something, but at least leave a comment to know who edited the document.

[MrStevns]

These are some good milestones you've set @chchwy my only addition to this would be documentation, and that should be on-going. The codebase can be really confusing sometimes, and this can make it really unattractive to new potential developers hoping to join the project. So my rule of thumb would be, whenever you work on a method, document it.

We don't have to document each and every method when we're not even sure whether it's used or not but those we do use should be documented.

[chchwy]

Totally agreed with you @candyface, documentation is very important.

I'm planning to write an introduction to pencil2d's project structure and conceptual overviews. Hope it could be a good starting point for new developers.

And I would like to say that reading unit tests is another good way to understand the APIs.

Please take a look at Layer's tests:
https://github.com/pencil2d/pencil/blob/master/tests/test_layer.cpp
It contains a lot of short examples to show how to use a single API like Layer::keyExists(). Writing tests is a good starting point for new developers to get involved, too. At the moment I'm the only one who wrote tests, we need more people to put efforts in it.

In my current company's project, the code is fully doxygened but most are just useless, for example:

/**
 * @fn   std::string Client::CreateCommonRequestString( Client* client )
 * @brief   Creates common request string.
 * @author  Someone who has left the company
 * @date 29/07/2015
 * @param [in,out]   client name   If non-null, the client.
 * @return  The new common request string.
 */
std::string Client::CreateCommonRequestString( Client* client ) { ...

It's a complete waste of lines because it says nothing. (and I have to scroll down one more time to see what it really does) So I would say, add the doxygen comments only when the function cannot explain itself.

Currently, my personal priorities would be:

1. Clear, self-explained code (sometimes it means refactoring, or simply renaming)
2. Documents of conceptual overview
3. Unit tests
4. Doxygen comments

But yeh, all contribute to making the code more understandable are welcomed (even it contains only comments)

[chchwy]

Just an update.

I've synced all the translations from OneSky to transifex. The old OneSky project has been marked as private and cannot be accessed anymore.

[Kaikogf]

Hi @chchwy
About the Translations on OneSky, you were right, I was there helping Luis (I don't recall his name exactly) with the Portuguese translation.
I would like you to check the languages on the new Transifex site, the Portuguese(Portugal) language is missing and was the one we were working on. 👍

[chchwy]

@Kaikogf Portuguese added. Sorry I didn't know you are working on this.

[Kaikogf]

Hi @Jose-Moreno
I have checked the link you provided and the "Colour Priority" 's table doesn't match the colours assigned to the Features (there is a green colour on line 5 "Restore Camera Layer..." which is not present on the list)
Just to let you know 👍

[Jose-Moreno]

@Kaikogf Hi, thanks. Hmm I made a task priority chart for the task themselves, but you're right the completion state color chart is missing. I'll update accordingly. 😄 , although basically is similar to a traffic light /semaphore red: not done, yellow: in process, green: done

[chchwy]

https://github.com/pencil2d/pencil/milestone/4

[chchwy]

Keep you all updated, I'm going to work on the memory issue this week.
Hopefully I can get it done before the end of January. And then we can start a QA session and release the 6.1 in the mid of Feb.

Just want to know you guys have any plan in these days?
@scribblemaniac Any news of the new timeline? or the progress of flood fill algorithm?
@J5lx How is the FFMpeg exporter going?
@candyface Are you going to do anything recently?

Nothing is in a hurry, just want to have a rough idea of what will happen in the 6.x.
Cheers.

[scribblemaniac]

No news on the timeline, because I have a few smaller projects I'm going trying to finish at the moment. My primary focus right now is getting the GSOC application ready. After that, you can expect a proposal for a new and improved flood fill algorithm, with an implementation following if we can solve the issues with it. If it comes to it, I can rebase my work to remove the new algorithm so the other good stuff can be included in 0.6.1. I am almost finished some refactoring which will remove QDomElement and its deprecated xml qt module entirely (there will be no changes to the file format). I also still have to finish that autocrop fix. I'm done most of that too, but there is some major bugs with it that I still have to fix. The color swatch dragging is also partly done, but that probably not something I will work on soon. None of this is critical, so if it doesn't make it into 0.6.1, it won't be a huge deal. There might also be some other small bug fixes or features from me, we'll see.

[MrStevns]

I'm working on #748 currently. So far I can undo a bitmap layer but the current implementation only backups the latest frame, so I've added a Qlist to backupElement which should hopefully make it possible to restore multiple frames. This should make it possible to undo a layer with lots of frames and bitmap information, though i'm not sure how well that's going to work yet.

[chchwy]

@candyface I was thinking to introduce QUndoStack to have better undo/redo architecture, but haven't got chances to do that. This could also be a preparation step for the undo history window in the future.

[chchwy]

0.6.1 is out.

[Kaikogf]

I will give it a go when I got more free time.
Fantastic

[chchwy]

I am closing this as v0.6.2 is out. Most tasks in this issue have been done. I will create another issue for the v0.6.3 later on if necessary.