Investigate the feasibility of supporting Flatpak packages

[x80486]

The title says it all!

Is there any plan to distribute Standard Notes as a Flatpak?

[moughxyz]

Not familiar with Flatpak, but don't believe we'd do this ourselves as it creates more points of maintenance.

[x80486]

I guess I get that, but the adoption of Flatpak is going up and up – also for Snaps. Those two has some clear (out-of-the-box) differences compared to AppImages.

Snaps has some disadvantages since Ubuntu has total control over it — basically the non-independence of the provider; having to agree with the Ubuntu CLA to contribute, a centralized store (it is possible to create independent repositories, but it is complex); they also still depends on an LSM like AppArmor (Solus OS had to integrate it into the system to offer all the features of the format, something I doubt all distros would do).

With Flatpaks everything is open sourced and is not as centralized as Snaps. Moreover, they are intended on delivering software to desktops — Snap packages are basically just server technology that Canonical has adopted for the desktop...but works for both.

I think an AppImage is fine until the project has some level of maturity. I also think this one falls into that bucket already...but it's really up to the project maintainer(s).

It's just a suggestion that I think people in the Linux world will appreciate, although this have been here for a week now...so I guess (1) there aren't so many Linux users, or (2) they are super-fine with the AppImage, or (3) they just don't care about Flatpaks 😆 ...or (4) they don't read the issues 😱

[moughxyz]

We use electron-builder to distribute the app. So if they offer Flatpak at some point, I'd see no reason we wouldn't offer it as well :) electron-userland/electron-builder#512

[x80486]

I may be wrong on this one.

My assumption (without even dig it) is that all Electron-based app are built pretty much the same way...if so, I think this can be a start.

Basically, what I'm trying to say is: if Atom, Slack, and Standard Notes are all Electron-based apps, and the first two are in Flathub already, then Standard Notes can be too 😇 ...of course, provided I know nothing about Electron 😉

[moughxyz]

It's the autoupdate functionality that would be tricky. That's given to us by electron-builder. We'd have no solution when it comes to Flatpak, as far as I'm aware..

[x80486]

@mobitar, do you think you can make use of the electron-installer-flatpak?

I'm not entirely sure if it's comparable with the method you are using to build Standard Notes, but maybe you can take a look at it whenever you have some free (what's that?!) time.

[moughxyz]

Honestly I'm looking to make things even simpler going forward, in terms of what I have to manage and maintain. I've removed .deb support in the last release. Going to stick with AppImage for now because it supports autoupdates out of the box. If I do decide to switch package format in the future, I'd probably replace AppImage and not add more options.

[x80486]

OK, fair enough...since this is not a thing it makes sense to close it.

[x80486]

So, the AppImage stopped working again (I know why, but I wouldn't like to have more dangling dependencies because of that):

uplink org.gnome.Shell.desktop[1028]: #10 0x7ffefcc215c0 I   resource:///org/gnome/shell/ui/tweener.js:182 (0x7fd68c0cf560 @ 15)
uplink org.gnome.Nautilus[995]: /tmp/.mount_standahE6yRx/app/standard-notes: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory
uplink gnome-shell[1028]: Object Meta.WindowActor (0x55678a9bb730), has been already finalized. Impossible to get any property from it.

Anyway, I went ahead and (try to) build a Flatpak 😉 ...but one of your dependencies ("spellchecker": "github:mobitar/node-spellchecker") is failing in a strange way on the build phase:

  • electron-builder version=20.13.3
  • loaded configuration file=package.json ("build" field)
  • installing production dependencies platform=linux arch=x64 appDir=/run/build/standardnotes/main/app
Error: /app/bin/node exited with code 1
Error output:
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~preinstall: [email protected]
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://[email protected]/mobitar/node-spellchecker.git
npm ERR! 
npm ERR! ssh: Could not resolve hostname github.com: Temporary failure in name resolution
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128
...
Error: module standardnotes: Child process exited with code 1

Is it possible for you to reference this one in a different way? Like a standard npm package?

---

I can access GitHub via SSH. Right after that failure, I can execute this successfully:

$ ssh -vT [email protected]
OpenSSH_7.7p1, OpenSSL 1.1.0i  14 Aug 2018
debug1: Reading configuration data /home/x80486/.ssh/config
debug1: /home/x80486/.ssh/config line 2: Applying options for github.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to github.com [192.30.253.113] port 22.
debug1: Connection established.
debug1: identity file /home/x80486/.ssh/github_rsa type 0
...
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
...
debug1: Host 'github.com' is known and matches the RSA host key.
debug1: Found key in /home/x80486/.ssh/known_hosts:1
...
debug1: Offering public key: RSA SHA256:e...Q /home/x80486/.ssh/github_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug1: Authentication succeeded (publickey).
Authenticated to github.com ([192.30.253.113]:22).
...
Hi x80486! You've successfully authenticated, but GitHub does not provide shell access.
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3360, received 2024 bytes, in 0.1 seconds
Bytes per second: sent 57386.0, received 34568.2
debug1: Exit status 1

[moughxyz]

Hmm..it may be that the submodule is indeed added through ssh. Can you see if this command works for you:

git clone [email protected]:mobitar/node-spellchecker.git

I probably should have added it through https for simplicity. Can try to make this update in the next release.

[x80486]

Yes, I can actually do anything, but so far, that's the only thing I can tell that it's failing:

[x80486@uplink no_backup]$ git clone [email protected]:mobitar/node-spellchecker.git
Cloning into 'node-spellchecker'...
remote: Counting objects: 1142, done.
remote: Total 1142 (delta 0), reused 0 (delta 0), pack-reused 1142
Receiving objects: 100% (1142/1142), 11.37 MiB | 29.40 MiB/s, done.
Resolving deltas: 100% (593/593), done.

Maybe it's a limitation of the build tool (on the Flatpak side) – I know there is a step to bring over all dependencies offline for repeateable builds and what not.

If you find a way to change it, I'll give it a try again, otherwise I'll wait for the whole build process to get better I guess.

[moughxyz]

Ok, have made a note to get around to this. You can also edit this yourself in the meantime by editing the .gitmodules file and the .git/config file to change from ssh to https.

[x80486]

I'll reopen this one and also leave this work in case a more knowledgeable soul in npm, Electron (and alike) wants to give it a whirl. The build process fails while building mobitar/node-spellchecker, but my guess is that this is due to a bad marriage between npm / node-gyp / Flatpak. I can't tell so far.

[moughxyz]

Hmm..do you have console output for the error?

[moughxyz]

Oh sorry, just scrolled up. So this is that same ssh issue? I'll update this to use https hopefully this week.

[x80486]

No that's fine the way it is. I just get passed that. I'm able to prefetch the source code from that repository and build it in --offline mode with npm.

It's now failing trying to build mobitar/node-spellchecker on this operation: prebuild-install || node-gyp rebuild. This is the error message:

npm info lifecycle [email protected]~install: [email protected]
sh: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
sh: prebuild-install: command not found
sh: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/app/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/run/build/standardnotes/main/app/node_modules/spellchecker/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/app/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/app/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/app',
gyp info spawn args   '-Dnode_gyp_dir=/app/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/app/$(Configuration)/iojs.lib',
gyp info spawn args   '-Dmodule_root_dir=/run/build/standardnotes/main/app/node_modules/spellchecker',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
gyp: Call to 'node -e "require('nan')"' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/app/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:336:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 4.14.68-1-lts
gyp ERR! command "/app/bin/node" "/app/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /run/build/standardnotes/main/app/node_modules/spellchecker
gyp ERR! node -v v8.11.4
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok 
npm info lifecycle [email protected]~install: Failed to exec install script
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `prebuild-install || node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /run/build/standardnotes/npm-cache/_logs/2018-09-08T17_45_00_180Z-debug.log

    at ChildProcess.childProcess.once.code (/run/build/standardnotes/main/node_modules/builder-util/src/util.ts:251:14)
    at Object.onceWrapper (events.js:317:30)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Socket.stream.socket.on (internal/child_process.js:346:11)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at Pipe._handle.close [as _onclose] (net.js:557:12)
From previous event:
    at installOrRebuild (/run/build/standardnotes/main/node_modules/electron-builder-lib/out/util/yarn.js:68:17)
    at /run/build/standardnotes/main/node_modules/electron-builder/src/cli/install-app-deps.ts:57:5
    at Generator.next (<anonymous>)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at installAppDeps (/run/build/standardnotes/main/node_modules/electron-builder/out/cli/install-app-deps.js:174:17)
    at then (/run/build/standardnotes/main/node_modules/electron-builder/src/cli/cli.ts:43:10)
    at <anonymous>

I wonder if I can use any of the prebuilds that you have here. Would pulling one of those be the same as doing prebuild-install || node-gyp rebuild in the spellchecker itself?

[moughxyz]

I've just updated the spellchecker package to use https instead of ssh. Give it another run and see if that helps.

f7d3eee

[moughxyz]

Sorry, didn't see your previous message when I posted the update. Ok, keep me posted.

[x80486]

If you can release (again) a .deb file, there is a way to do this. I can try that way.

[moughxyz]

The app can be built to use .deb. You'd simply have to modify the package.json file to add that option in the linux section. We use electron-builder.

[ghost]

@mobitar would you consider merging a PR that adds support for building a flatpak?

[x80486]

All right! I was able to build this from sources! 🥳

There are some issues that I think, if you are interested in moving forward with this format, should be fixed first before publishing it, because Flatpak is sandboxed by default, hence, it wouldn't make sense to expose permissions that shouldn't be set.

Here is the list of issues:

• The application should follow the Freedesktop XDG base directory standard, even for backups (should be extremely easily to change)

• Disable the update manager because nothing can be written inside the container. There is a file: /.flatpak-info, if it exists, don't do the update check; I see this error all the time I start it:

updateManager: Checking for updates.
14:54:16.548 › Checking for update
14:54:16.571 › Error: Error: ENOENT: no such file or directory, open '/app/standardnotes/resources/app-update.yml'
updateManager: Exception caught while checking for autoupdates: [Error: ENOENT: no such file or directory, open '/app/standardnotes/resources/app-update.yml'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/app/standardnotes/resources/app-update.yml'
}
updateManager: Finished checking for updates. Latest version:
 3.3.5 Current version: 3.3.5

Nice to have:

• Modify or provide another build step for dist-linux
• Use Yarn instead of npm (this is actually recommended for Electron applications that use electron-builder)
• Provide an SVG icon

[arielsvg]

That's really cool, great job! I have two followup questions:

• Where should backups be if we were to follow the Freedesktop XDG base directory standard? something like ~/.standardnotes?
• If we disable the auto-updater, I assume we'd have to publish the flatpak somewhere. Is there a popular flatpak store/repo?

[x80486]

My understanding is that, for Linux, it should be ${XDG_DATA_HOME} (which translates to something like: "data files" go into ${HOME}/.local/share/), hence: ${XDG_DATA_HOME}/standardnotes/. I've read something about Electron and it looks like app.getPath(name) is the way to go, providing the appropriate name identifier, because it will give you the equivalent for all platforms (but you probably know that already). Again, I think that should be done regardless of the Flatpak, but the reason I proposed it in the Flatpak context is because you wouldn't need to widen the permissions due to a design flaw; of course, you can release it like that, and tweak it in the future (IntelliJ IDEA just fixed that).

Let me be clear, the auto-updater should only be disabled for the Flatpak, any other format(s) can make use of it. Being said that, Flathub is the store that I know everyone uses today – think of it like the Play Store (for Android) or the Apple Store. They take care about the entire infrastructure for building, publishing, etc.

[arielsvg]

Gotcha, we don't have much bandwidth to add another publishing format at the moment, but it's good to know for future reference!
We're already using app.getPath(...) for backups. But we give it the home identifier, because we want backups to be easily discoverable by the user. They're here in case of a major problem occurring after all, so burying them alongside technical application files is not desirable

[x80486]

OK, it's really your decision as the owner of Standard Notes.

Now coming back to the bandwidth point. The reason I did it was because this is much better than the AppImage (which is broken in many ways some times, at least for me). So I'm willing to maintain the format for some time (which I would say, it's a matter of updating a couple of lines in JSON file), but it would be better if you (and/or some others from Standard Notes) are part of the (GitHub) repo in Flathub, at some point, if it turns out that Flatpak works great for this application to promote it officially, and things like that...my point is: this could be driven by us (the community) but it would be better if it's supported and not an orphan thing from your side.

Let me know.

[ghost]

I think Flatpak the way forward. Flatpak is already present on many Linux distributions by default and is very well integrated into the system. Flathub is a very popular Flatpak repository and is also included in many distributions (either enabled, or a few clicks away from being enabled). Applications can be easily discovered and (automatically) updated from the software centre (GNOME Software for example) or from the command line.

[arielsvg]

To be clear, we will absolutely look at supporting Flatpak in the future, including how much we'd involve the community in that process 🙂 in the meantime we'll keep supporting Snaps and AppImage.

[A6GibKm]

@x80486 could you please share the flatpak manifest to try building it myself?

The application should follow the Freedesktop XDG base directory standard

Note that, while following xdg standard is a good practice, it is not required for this app as one could simply build with the flags --filesystem=~/Standard\ Notes\ Backups if we want it to access the folder in the user home, or --persist=Standard\ Notes\ Backups for it to be stored inside the sandbox (at ~/.var/app/org.standardnotes.standardnotes/Standard\ Notes\ Backups). User could give it --filesystem=home if they don't want to overthink it when moving the location of the backups.

Disable the update manager because nothing can be written inside the container
Could this crash the app?

I would be happy to keep an updated build at flathub, (bundle icons, desktop file, provide an appdata.xml if needed, and the publish process), so that the devs can take over it when/if they feel comfortable with this distribution method. But for that, having an already working manifest would help a lot.

[x80486]

Hey @A6GibKm, I don't have it anymore. If they decide to move forward, I'll put it together again (it's super-easy now that I know how to do it 🤣 ) and handle it to you.

Pay attention to the thread ☝️, even if you have the manifest right now, they won't agreed on publishing it in Flathub, and as soon as you create a pull request over there they are going to ask you if the developers are OK with the app on their hub.

[justyn]

How about a temporary, unofficial Standard Notes Flatpak called something else? Perhaps "Ordinary Notes".

A community maintained "friendly fork" with no other changes, that would enable those who want to use Flatpak to do so, coming with no guarantees of working whatsoever? It could be folded into an official Standard Notes Flatpak in the future if/when the SN devs so wish. In the meantime it could be a useful experiment for them to keep an eye on, without taking up any of their time.

This is a well-intentioned suggestion so apologies if it inadvertently causes any offence! I don't have the time to dedicate to such a project myself anyway, but I note that a few people here seem to be keen to put the effort in!

[rstrube]

As a Standard Notes subscriber, I'd also love to see official Flatpak support. It's great that x80486 was able to get something up and running!. Lots of folks aren't very pleased with Snaps for a variety of reasons including a proprietary backend, and forced updates. Here's an example of some of the frustrations a fellow developer has delt with in regards to snaps: https://raymii.org/s/blog/Ubuntu_Snap_auto_updates_broke_my_development_setup.html. Also distros like Mint and Pop_OS have officially removed snapd and embraced Flatpak.

Please reconsider adding official flatpak support, many of your Linux customers would really appreciate it!

[arielsvg]

To be clear, we will absolutely look at supporting Flatpak in the future, including how much we'd involve the community in that process 🙂 in the meantime we'll keep supporting Snaps and AppImage.

This comment is still valid, we're slammed with other work that can't be delayed so we're just not in that future yet! Hope you'll bear with us and thanks for subscribing. :)

[baarkerlounger]

@baptiste-grob worth re-opening this if it's something the StandardNotes team are still planning to look at in the future?

[ghost]

Adding another vote for flatpak, as due to my own fault (hardening firefox beyond what is reasonable for this type of app) I get fatal wasm errors (e.g. "Both wasm and asm failed to load", "The Same Origin Policy disallows reading the remote resource at notify.bugsnag ...", ...). A flatpak distribution would be a great.

[arielsvg]

Good point @baarkerlounger :)

[rendhalver]

I would also like a flatpak version of Standard-Notes.
I would be interested in testing things and maybe helping out with the development and release automation if that helps.

[jadmadi]

I would also support the notion of having Standard Notes on Flatpaks hub

[icfmp]

Yes please, provide it on flathub 👍

[dnet890]

There is an unofficial flatpak (https://flathub.org/apps/details/org.standardnotes.standardnotes). But it also asks for keyring every time I reboot my PC. that means I need to enable password reading on the startup-like snap app which is quite annoying

[vchernin]

There is now experimental support for building flatpaks in electron-builder. There is no native support to upload directly to flathub yet. See electron-userland/electron-builder#5711