Update QEverCloud and use system browser for OAuth (with ability to also use QtWebEngine or QtWebKit)

[d1vanov]

See my comment here for more details.

This PR updates the copy of QEverCloud within nixnote2's source tree to latest 6.1.0 release. The update includes adaptation of the entire codebase in places where QEverCloud API is called. The adaptation was quite straightforward so I shoudn't have broken anything. I was able to login to my account and sync it using nixnote2 built from this branch so I guess it counts for some testing.

One more thing that I did put into this PR as a separate commit is a change of default authentication token for public linked notebooks - I have a public notebook linked into my account and I had an error during sync until I replaced the token. It's kinda strange actually, the last time I checked Evernote's sync documentation it said public linked notebooks should not require auth token at all but my experience shows it actually does require it.

Why this PR is incomplete - because in order to fix OAuth I had to add a new dependency - on webengine and webenginewidgets Qt modules. I updated the nixnote2.pro file so it can be built but there are multiple build and package scripts in this repo - these I haven't updated and it's unlikely that I will. Moreover, there are downstream maintainers for Nixnote2 for various distros, they all would have to update their packaging scripts as well if this gets merged.

So I encourage anyone interested to continue my work and update various build and package scripts to include the new dependency. The completed version could then be merged.

[d1vanov]

Oh and I forgot to mention, to the best of my knowledge QtWebEngine cannot be used along with MinGW on Windows, it supports only MSVC. So I guess if the switch to QtWebEngine would be unconditional, it would break the way to build Nixnote2 with MinGW on Windows. Perhaps the dependency on QtWebEngine could thus be made optional so that on Windows there would be a way to continue using QtWebKit only, perhaps using a maintained fork of it instead of the official version which is known to not quite work for OAuth anymore.

[d1vanov]

And one other notable thing, I switched the C++ standard from C++11 to C++14 along with updating QEverCloud because QEverCloud requires C++14 standard. C++14 is 9 years old by now so I guess it's ok to update it.

[d1vanov]

Did some more tinkering here:

1. I was able to make QtWebEngine usage conditional and not enabled by default. In order to use QtWebEngine for OAuth, one needs to launch qmake with CONFIG+=oauth_webengine command. When QtWebEngine is not used, a warning is issued by qmake about OAuth with QtWebKit possibly not working (except in MinGW on Windows case). Downstream maintainers and packagers of Nixnote2 will hopefully see this warning and enable the use of QtWebEngine in their builds.
2. The problem of building on Windows with MinGW is also kinda solved in that the conditional use of QtWebEngine doesn't block building the old way with QtWebKit and MinGW.
3. Switched debian packaging files to use QtWebEngine OAuth.

I also tried to switch docker build to QtWebEngine but I get some problems there which I have already encountered with Quentier: there are some libraries which, when included into AppImage, lead to crashes, in particular libnss3.so and libnssutil3.so. They seem to depend on some other system specific stuff which is not bundled into the AppImage. In case of Quentier excluding them from the AppImage worked but in case of Nixnote2 I see some SSL related errors in the log and OAuth still doesn't work. Will see if I can sort it out.

[robert7]

if you can solve the AppImage problem, this would be great

[d1vanov]

Welp, I discovered that Quentier AppImage doesn't work on my machine as well.
It's all because of probonopd/linuxdeployqt#568. In short, Ubuntu 22.04 stopped shipping OpenSSL 1.1. All of AppImage tools blacklisted these libs from bundling because they believed these libs are too low level and should be updated independently of AppImages, particularly because it's important to deliver patches to security vulnerabilities for all apps on the system at once. But now apps using QtWebEngine stopped working due to these missing libs.

I tried to install OpenSSL 1.1 manually on my system, it didn't help. I also tried to bundle these libs manually into the AppImage, it didn't help as well.

Probably the only viable option is to provide two AppImages, one for systems older than Ubuntu 22.04 and one for newer systems...

[vitaly-zdanevich]

If you are talking about AppImage, can you please include dark QT theme inside?

[d1vanov]

I don't know the details of what needs to be bundled into the AppImage to support themes. And there are more important problems with the AppImage as I described above than some missing themes - it won't work on Ubuntu 22.04 and beyond if this PR is merged.

[vitaly-zdanevich]

Probably the only viable option is to provide two AppImages, one for systems older than Ubuntu 22.04 and one for newer systems...

My vote is to only support newer systems, I respect your time to support this stuff.

[boo-yee]

@d1vanov @robert7

Hi, d1vanov,

What you want to achieve is to login successfully, right?

If so, that will not be a big problem, so will this issue(#171). When you type in username and password in the login page, click login button, blank page appears as issue 171 said. At the moment, you could just wait for seconds and then close it, nixnote will pop up a message saying that login fails. You could just ignore that, and then click the sync button again, the authorization page will appear normally this time, and then you click authorize button to finish it.

I found this issue months ago, but I didn't think it serious. If we want to fix this some confusing issue, we could just help QtWebKit finish the jump-off manually, which will be more economical than using QtWebEngine, I think.

And also, replacing QtWebKit with QtWebEngine might bring problems such as:
the html code of the order/unorder list, todo list and other html entities may differ between these two web engines;
more usage of resources.

So, in summary, I think, using QtWebEngine is not a good bargain.

[d1vanov]

Interesting, I haven't tried to try authentication twice. And in my case the page wasn't blank at the start, it showed me the field to enter my login but when I entered "continue" a new field for password should have appeared but it didn't. With QtWebEngine it worked properly.

I still think it would be useful to offer at least an option to use QtWebEngine for OAuth because I'm not surprised that unmaintained QtWebKit has problems with it - modern web pages use modern JS frameworks and whatnot while QtWebKit has been stale for years.

And you misunderstood, this PR won't replace QtWebKit with QtWebEngine for the note editor but only for OAuth page. So the only problem would be just having a dependency on it.

[boo-yee]

but when I entered "continue" a new field for password should have appeared but it didn't

I met this sometmes, I fixed it with another two methods:

1. force the password form to show with the help of QtWebKit api. (I wrote the code, but I didn't add it the to this repo.)
2. use a proxy.

[d1vanov]

force the password form to show with the help of QtWebKit api. (I wrote the code, but I didn't add it the this repo.)

It would be interesting to see this code, maybe it would be a simpler solution to this problem.

[boo-yee]

force the password form to show with the help of QtWebKit api. (I wrote the code, but I didn't add it the this repo.)

It would be interesting to see this code, maybe it would be a simpler solution to this problem.

I remember that it was just serveral lines. I'll add it to this repo later.

[vitaly-zdanevich]

So, in summary, I think, using QtWebEngine is not a good bargain.

Most Linux distributions does not have QtWebKit anymore in their repos, so we cannot use it as a dependency.

[boo-yee]

So, in summary, I think, using QtWebEngine is not a good bargain.

Most Linux distributions does not have QtWebKit anymore in their repos, so we cannot use it as a dependency.

Thanks for your info. I didn't focus on QtWebKit dev too much ever before. This is an imporant point I neglected.

[boo-yee]

force the password form to show with the help of QtWebKit api. (I wrote the code, but I didn't add it the this repo.)

It would be interesting to see this code, maybe it would be a simpler solution to this problem.

I remember that it was just serveral lines. I'll add it to this repo later.

I looked into the code again, and realized that if we want to show the password form programatically, we have to write some application level code within QEverCloud api, which is not a reasonable thing. Concerning to the fact that QtWebKit is dropped by most distros as vitaly-zdanevich said, using QtWebEngine to replace QtWebKit is probably more reasonable in the long term.

[robert7]

I also think, for long term, replacing of QtWebKit by QtWebEngine is sure preferable, just it was not in my time budget to do it

other option which I tried (back in 2018) was port of WebKit - see e.g. https://github.com/movableink/webkit

[d1vanov]

if we want to show the password form programatically, we have to write some application level code within QEverCloud api, which is not a reasonable thing

Could you please demonstrate how this could be done nevertheless? As the maintainer of QEverCloud I'm interested in getting it's OAuth to work with QtWebKit as well as QtWebEngine even if that would involve some hacky code. Maybe it won't be too hacky I could hide it inside QEverCloud internals.

[d1vanov]

I also had a thought of another possible way to do Evernote OAuth - delegate the display of Evernote's login page to the system browser and thus not depend on either QtWebKit or QtWebEngine for OAuth. I'm not yet sure but it should be possible - if we start a HTTP server on some port on localhost, we could pass the address of this localhost server as the "callback url" which is one of the parameters with which Evernote's OAuth page is opened. After the user authorizes the app, the system browser should make a request to this callback url with the token for the app as one of URL parameters. With QtWebKit or QtWebEngine backends we just intercept this callback request and extract the token from it's url parameters. In case of local server we would also be able to extract the token from URL parameters. However, we'd also need to respond with some HTML which would say something like "Authentication is complete, you can now close this page and return to the app".

[d1vanov]

I'll try to do an experiment and see whether this approach is feasible.

[boo-yee]

if we want to show the password form programatically, we have to write some application level code within QEverCloud api, which is not a reasonable thing

Could you please demonstrate how this could be done nevertheless? As the maintainer of QEverCloud I'm interested in getting it's OAuth to work with QtWebKit as well as QtWebEngine even if that would involve some hacky code. Maybe it won't be too hacky I could hide it inside QEverCloud internals.

OK. I just share the idea, but I don't recommned anyone to implement it. As I cannot reproduce this issue now, I suspect that it was caused by network condition in my case. And QWebEngine is almost the choice in the future, so I think when we finish the migration from QWebKit to QWebEngine, QEverCloud can use it together with the editor.

The idea is simple, just find the element of password when the login page finishes loading, delete its 'invisible' attribute. A web frontend operation. It can be done with QWebView api.

[boo-yee]

I also think, for long term, replacing of QtWebKit by QtWebEngine is sure preferable, just it was not in my time budget to do it

other option which I tried (back in 2018) was port of WebKit - see e.g. https://github.com/movableink/webkit

If we choose only QWebKit, we may have troubles in distribution, maybe we have to include the webkit code in the nixnote repo, and tolerate security issues, though its possibility is low.

[d1vanov]

Good news - I was finally able to successfully authenticate to Evernote via QEverCloud and system browser instead of either QtWebKit or QtWebEngine. However, I ended up with quite messy code for the prototype and would like to refactor it somewhat before I update this PR. But I intend to do it some day.

[paoletto]

@d1vanov have you succeeded in creating an appimage for 22.04? When i tried doing that, the resulting appimage (created on 22.04 for 22.04) did not run..

[d1vanov]

I was able to create AppImage on 22.04 which did run on 22.04. However, I wasn't able to create an AppImage on 20.04 which would run on 22.04 due to problems with QtWebEngine and OpenSSL. When I'm finished with this PR, there shouldn't be a dependency on QtWebEngine and thus the AppImage problem should also be solved.

[paoletto]

thanks for the feedback! if you did use linuxdeployqt, may i ask what options got it working for you? if not, may i ask how you generated the 22.04 on 22.04 appimage? were you still including QtWebEngine at that point? thanks

[d1vanov]

linuxdeployqt only recently started to work on 20.04 and doesn't work on 22.04 so I didn't use it. Instead I used linuxdeploy, linuxdeploy-plugin-qt and appimagetool. Roughly as follows:

wget -nv "https://github.com/linuxdeploy/linuxdeploy/releases/download/continuous/linuxdeploy-x86_64.AppImage" && \
        chmod a+x ./linuxdeploy-x86_64.AppImage && \
        ./linuxdeploy-x86_64.AppImage --appimage-extract && \
        mv squashfs-root linuxdeploy && \
        mv linuxdeploy/AppRun linuxdeploy/linuxdeploy
wget -nv "https://github.com/linuxdeploy/linuxdeploy-plugin-qt/releases/download/continuous/linuxdeploy-plugin-qt-x86_64.AppImage" && \
        chmod a+x ./linuxdeploy-plugin-qt-x86_64.AppImage && \
        ./linuxdeploy-plugin-qt-x86_64.AppImage --appimage-extract && \
        mv squashfs-root linuxdeploy-plugin-qt && \
        mv linuxdeploy-plugin-qt/AppRun linuxdeploy-plugin-qt/linuxdeploy-plugin-qt
wget -nv "https://github.com/AppImage/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage" && \
        chmod a+x appimagetool-x86_64.AppImage && \
        ./appimagetool-x86_64.AppImage --appimage-extract && \
        mv squashfs-root appimagetool && \
        mv appimagetool/AppRun appimagetool/appimagetool
export PATH=$(pwd)/linuxdeploy:$(pwd)/linuxdeploy-plugin-qt:$(pwd)/appimagetool:$PATH
cd nixnote
qmake PREFIX=appdir/usr
make -j
make install
linuxdeploy --desktop-file=appdir/usr/share/applications/nixnote2.desktop --appdir=appdir --plugin qt
rm appdir/usr/lib/libnss3.so
rm appdir/usr/lib/libnssutil3.so
appimagetool -n appdir

The first linuxdeploy call collects the dependencies and puts them into the appdir + adjusts their link paths. Then I chose to remove the collected nss libraries as they are a known pain point of AppImages, see probonopd/linuxdeployqt#35. And then I call appimagetool to create the AppImage from the prepared appdir.

I did include my changes with switching OAuth to QtWebEngine but only because I coudn't get the OAuth via QtWebKit to work properly. It seems that for some people it strangely works though so you might not need the QWebEngine related changes to authenticate the app with Evernote.

[d1vanov]

Updated the PR - now it brings in QEverCloud 6.2.0 which can be built with different "backends" for OAuth - one using system browser, another one using QtWebEngine and another one using QtWebKit. I brought these configuration options to Nixnote2 as well and made OAuth via system browser the default option as it seems to work surprisingly well. Checked that with all three available options I can compile Nixnote2. QtWebKit backend doesn't work quite well but we already know that.

I've also updated the script to build things with docker to use Ubuntu 20.04 image and checked that the AppImage generated by it works on Ubuntu 22.04.

[robert7]

hi @d1vanov, the PR is from your side now ready to merge?
Sorry, I did not yet have time for review, but I'll do it, when you say its "ready" 😀

[d1vanov]

Well, yes, I believe it is ready, if not for immediate merge then for review at least )

[robert7]

Hi @d1vanov,
I looked at the changes... It looks really great!

I found a minor problem. After initial sync with authorization via system browser, an error message is displayed, saying that the sync failed (but actually it worked fine and authorization is successful). After clicking OK on the error message, all seems to be OK and further sync, work. Can you maybe check that?

A further really minor problem is: if the updated "docker build" is called with a branch name which is like "abc", then the git checkout fails. (If the branch name is like "feature/abc", then it works).
e.g. ./development/build-with-docker.sh d171 fails... as it does: "git checkout d171" and, I guess, it should be "git checkout origin/d171".

Further, we need to update the ".travis.yml" to get the Travis build in sync with build-with-docker.sh i.e. to change the build to run with Ubuntu "focal" instead of "xenial".
Would you be so kind to also update that?

Anyway, it is a big improvement and I will merge the changes soon.
Big thanks!!!

sync error message:

[d1vanov]

Hmm, I think I caught this error during my tests but just once. I'll see if I can reproduce and debug it.
Ok, it should be easy enough to update the Ubuntu image on Travis CI.
I'll also try to take a look at the branch problem with docker build script.

[d1vanov]

Found a couple of things about this sync error:

1. It triggers on attempt to fetch user-related information from Evernote. I presume it is only attempted during the first sync after the authentication so that's why this error doesn't show up later.
2. One problem was forgotten "https://" before the Evernote host. I solved it but it lead to further problem.
3. The further problem was the fact that Evernote for some reason decided to answer with HTTP code 302 - temporary redirect - which QEverCloud was not ready to handle. Never saw a redirect from Evernote API before.
4. Fine, I set req.setAttribute(QNetworkRequest::FollowRedirectsAttribute, true); attribute to let Qt handle redirects automatically for me. So now it fails on attempt to parse user information from what is supposed to be a thrift encoded binary message with the data. Instead what I receive from the network reply is some HTML weirdly resembling the OAuth page. I've attached the archived HTML page just in case.
   user.zip

[d1vanov]

Ok, I figured it out: wrong URL was used to initialize UserStore: it was just Evernote host (i.e. www.evernote.com or sandbox.evernote.com) but it should also have had suffix /edam/user.

[d1vanov]

Pushed fix for that sync issue + attempt to update Travis CI configuration. It rarely goes smooth from the first attempt so I might need to do some back and forth steps to make it right...

[d1vanov]

Ok, I got Travis CI build fixed.
Besides updating the Ubuntu image I also removed the usage of custom built tidy-html5 in favour of the package provided by distro. I believe that this custom package is a kind of legacy which was necessary in the old days of Ubuntu Trusty which provided much older version of libtidy which didn't quite work well. But nowadays distro provided libtidy is actual enough so the custom package shouldn't be needed anymore.
I also decided to use linuxdeploy and linuxdeploy-plugin-qt instead of linuxdeployqt as this tool appears to be more modern and more user friendly, at least it doesn't have that pesky problem with complaining about too new glibc if you try to use it on a recent enough distro. Plus these tools are used in my version of build-with-docker.sh as well.
Speaking of which, I tried to fix the checkout in build-with-docker.sh too, it seems to work for me now.

[robert7]

OK great! I'll look at it later. Thanks.

[robert7]

thanks!