Investigate AppStream metadata for building app directory

[probonopd]

The AppStream specification describes the AppStream upstream XML format which is used by upstream software authors to describe the software component (application, firmware, font, ...) they provide, as well as the AppStream distro XML format that is used by distributions to ship the metadata in bundled and processed form to their users.

Ideally, apps would be packaged by the upstream projects, and a repository of apps including version and update information, screenshots, as well as stars (or AppStream kudos) could be built. When I tried to do this a decade ago for klik and later for PortableAppImages, cross-distribution metadata infrastructure was not available, but now it begins to fall into place.

AppStream distribution XML files could be parsed.
Debian collects icons and app metadata here:
https://appstream.debian.org/data/unstable/main/

It looks like AppStream is built with Xapian in mind. Xapian is an Open Source Search Engine Library, released under the GPL. It's written in C++, with bindings to allow use from Perl, Python, PHP, Java, Tcl, C#, Ruby, Lua, Erlang and Node.js (so far!)

[probonopd]

Talked to @aleixpol and he suggests that we should be providing a list of available AppImages in the format of archlinux-appstream-data (use this format but serve the icons on demand under a URL). This would make it easiest for software centers like App Center and Discover to pick the information up. It is especially important to use the same <id> tag for each app as in archlinux-appstream-data so that the same app, when provided via different channels, can be shown to the user as the same app in different variants.

I can imagine (at least) three ways where software centers could get the information about AppImages from:

1. Download it from a database via an URL (the user could have the option to add such URLs in order for the system not to be centralized)
2. Extract it from AppImages already present on the system
3. From the AppStream data that comes with the distribution (it might take quite some time to trickle down)

[probonopd]

For presenting the app directory in the web (in addition to software centers coming with the desktop), possibly @ximion has some code that could serve as a starting ground.

[ximion]

I would need more information on what you are planning ;-)
In general, code to read AppStream metadata no matter in which form and also to do basic writing of the data can be found at https://github.com/ximion/appstream - it is used in Limba as primary data source for pretty much everything (except for some details of the bundle itself).
If you have a repository of AppImage bundles, you could also write a backend for appstream-generator ( https://github.com/ximion/appstream-generator ) which is the tool used at Arch and Debian to build the final metadata that is shipped to users. At time, asgen is built with distro repositories in mind as a data source, but its backends concept is abstract enough to allow consuming not only distro packages. What asgen does compared to "raw" usage of libappstream is rendering icons to the appropriate sizes, caching screenshots and doing a couple of other things (font rendering, searching of icons across multiple packages, ...).

For an app-directory on the web, I am currently experimenting with the vibe.d ( http://vibed.org/ ) framework for developing an online repo service. Vibe has some very nice properties, mainly being similar to NodeJS, but using a language which doesn't drive you insane with crazy levels of callback-nesting. But I am not yet 100% sure whether Vibe will be the final solution, although it's looking quite good so far.
The Xapian cache is only a client-thing, while it can also be used for websites (it's certainly fast enough), some properties, like its focus on only one language, make it less ideal for web use. It is used on the clients to reduce memory consumption of software-centers and to allow some advanced search features, like stemming. Using AppStream and libappstream without Xapian is perfectly possible though (just use AsDataPool instead of AsDatabase in the API).

[probonopd]

Thanks @ximion for the detailed explanation. As for what exactly we are trying to do, well, we are still trying to figure out what would make most sense for a decentral, community-based/crowd-sourced/chain-of-trust type of system.

What seems clear is that with more and more options for users to get software (native distribution package, ppa, COPR, Flatpak, Snappy, AppImage - stable, rolling, nightly) if seems to be common sense that it would be beneficial to offer these choices to the user in a convenient way inside the software centers.

[ximion]

Well, AppStream is tying all of these together and allows them to share metadata.
See the "bundle" tag for metainfo files: https://www.freedesktop.org/software/appstream/docs/chap-DistroData.html#tag-distro-bundle
So, displaying that stuff in software-centers in a way which is nice for users will be a rather simple task, if everything uses AppStream.
For Snappy, this might take a while because apparently someone higher up overruled the engineers to not implement AppStream - but I will be at a Snappy sprint this summer, maybe there still is a way to come to an agreement.

[probonopd]

Looks like you have thought of everything @ximion, yay for the <bundle> tag.

Would something along the lines of

<bundle type="appimage" channel="releases" >Foobar (releases)</bundle>
<bundle type="appimage" channel="development" >Foobar (development builds)</bundle>
<bundle type="appimage" channel="continuous" >Foobar (continuous builds)</bundle>

be allowable, with the actual version numbers being determined on demand, in order to save bandwidth? I am especially thinking of continuous where you'd have to check for the version essentially all the time.

Or is the <bundle/> tag meant to be used inside the <releases> tree structure? In that case, would a release called "continuous" be allowable?

[probonopd]

Note to self, also check https://sessellift.wordpress.com/2016/04/25/a-usability-guys-journey-to-creating-his-first-kde-tool-part-2-a-vision/

[aleixpol]

Don't you want to specify the URL instead of a channel?

[probonopd]

Correct @aleixpol.
In fact, not only direct http(s) download links to the AppImage but alternatively also the update information which can not only be used to update an AppImage but also to download it from scratch.

Will think about how to do this best.

Edit: Maybe it would be best to link to the download page, and have the user download the AppImage from there. This way the website could count downloads, etc. in any way it wants.

[probonopd]

@hughsie had been mentioning the need for a "central registry of AppImages" on https://blogs.gnome.org/hughsie/2016/04/27/3rd-party-fedora-repositories-and-appstream/#comment-5183

So, how would one do a "central registry of AppImages" without being too restrictive about what gets "in"? Right now I can imagine

1. Set clear and lightweight rules and every AppImage that matches those rules gets in
2. Have the possibility to add more than one "central registry of AppImages"

Thoughts welcome.

[ximion]

@probonopd Please read the AppStream spec, especially the distinction from metainfo files vs. distro-metadata.
To put it simply, right now AppStream only allows one installation candidate to be there by design. The component-id (org.example.MyApp.desktop) is supposed to be unique for this particular application. Bundling systems can offer AppStream metadata via some way, having a <bundle type="whatever">${ID}</bundle> tag, where ${ID} refers to some identifier used by the bundling system to install the particular app. Could be, for example, a web URL, or anything the software-center frontend can pass to the app-installer to actually install an application.

If you want to have different "channels" of an application, the only way to realize that right now would be giving them a different component-id. For example org.mozilla.Firefox vs. org.mozilla.FirefoxAurora. Those would have their individual bundle-tag to identify what needs to be installed to make that app available. It also allows an easy way of displaying them in the UI, and, most importantly for me, allows upstreams to set a different description and other metadata for their development versions (e.g. include a warning message that this is untested software).

In my personal opinion, alpha and beta releases should not be in the software sources at all, because having them is incredibly confusing for people, and it creates overcomplicated UI and presents users a decision of taking the newest version, or some older release, which they might no be able to make and usually don't need to make, because there is a released, known to be woring version available.
For the cases where these snapshots are useful, e.g. when working on bugs, providing a bundle outside of the software centers main sources does make a lot of sense.
For the rare cases where having different release channels is useful, giving the software different IDs is justified, IMHO.

[probonopd]

If you want to have different "channels" of an application, the only way to realize that right now would be giving them a different component-id. For example org.mozilla.Firefox vs. org.mozilla.FirefoxAurora.

I can see how this would work for Aurora (which in fact can be seen as a "different application" (has its own name, icon etc.). But if you take Scribus or the GIMP then i'm not sure that treating the development version as a "different application" is the best way.

I also understand that in order not to confuse ordinary users, development/nightly/continuous versions should not be shown too prominently, and there might be a warning when installing them, or a GUI checkbox that would beed to be checked in order for the user to see such versions. Especially with AppImages, I expect one major use case to be development versions (for the very reason that one app = one file makes it super easy to just try something out). So I am convinced that it could be beneficial to have no limit on the number of different versions possible to be offered, including historical ones. Take web browsers, for example - I am sure that web designers, for example, might want to test something not only on the latest but also on other versions of browsers.

That being said, I am even more inclined now to just put in the web link to the project's download page instead of a deeplink to the AppImage. This way, the project can explain the different versions on the download page and provide as many options, versions, or variants as they want, give warnings regarding development versions, etc.

On the following question I would really appreciate some input - what exactly is supposed to go into the bundle tag. An URL from where the bundle(s) can be downloaded? Pro: User can decide which bundle to download, contra: cannot automatically download/update/"install". An identifier that allows an AppImage helper (AppImageUpdate or a gnome-software/discover plugin) to download/update the AppImage? Pro: More automation. Con: How to handle more than 1 version/channel?

Is there any existing convention on what goes into the bundle tag?

<bundle type="appimage">https://krita.org/en/download/krita-desktop/</bundle>

Correct? Or is this reserved for non-HTTP-links and we should use something like

<url type="appimage">https://krita.org/en/download/krita-desktop/</url>

[probonopd]

@ximion One open question is, how do software centers "learn" about bundled software (Flatpak, AppImage etc.)

An AppImage that contains AppStream data should be able to place this to a location in the system where software centers can pick it up.
Looking at the AppStream spec, XMLs are expected to live in /usr/share/app-info/xmls/ or /var/cache/app-info/xmls but for AppImage we need a way to get data in as non-root.

https://mail.gnome.org/archives/gnome-software-list/2016-June/msg00000.html talks about ~/.local/share/app-info/xmls/ - @ximion is this an allowable location?

[ximion]

@probonopd

Looking at the AppStream spec, XMLs are expected to live in /usr/share/app-info/xmls/ but for AppImage we need a way to get data in as non-root.

https://mail.gnome.org/archives/gnome-software-list/2016-June/msg00000.html talks about ~/.local/share/app-info/xmls/ - is this an allowable location?

Not yet, it's something GNOME Software specific, so you can not rely on it being present or used at all.
However, it's on my todo-list to maybe add it to the spec and libappstream.
(in general, in my opinion installing executable apps into the user's $HOME directory is a really bad thing to do, and they should always be stored in a system location (for enhanced security, sharing of resources, ...) But I do see that having some metadata in .local might be useful, so that path might be available for string distro XML in future.
(I will comment on your other post later, lunchtime now!)

[probonopd]

@ximion at least for AppImage, being able to download and run applications without the need for root rights is essential. So I would really appreciate ~/.local/share/app-info/xmls/ becoming available.

[probonopd]

Can someone clarify the relationship between AppStream and Appdata? ("AppData is a subset of the AppStream schema, which is also XML." is the official explanation, but I would be interested in what I should provide, and why there are two...) Especially, if an upstream project such as Krita provides AppStream data, is it converted to Appdata before it is included in Fedora, and would any <bundle> tags be "thrown out" in the process?

[probonopd]

Remains the question, how can the system learn about not-yet-installed AppImages for software that are not part of the distro.

• The snap gs plugin seems to communicate directly with the Snapcraft store via snapd, hence opening a side channel and not using AppStream.
• Debian uses /etc/apt/apt.conf.d/50appstream to fetch DEP-11 (a YAML variant of AppStream) on each apt-get update.
• We could have a list of HTTP links to AppStream files in the upstream's repositories that a tiny plugin in the software centers (e.g., gnome-software and Discover) would download and use? This list could be fed by manual submissions and/or after (automated) testing of submitted AppImages. The same list could be used by websites (e.g., Freshmeat, OpenDesktop,etc.) that want to provide an "app store" like experience.

[ximion]

@probonopd Read the spec, it's all explained there ;-)
AppStream == Metainfo files + AppStream distro metadata
Appdata == MetaInfo file

When Richard started the upstream metadata as "appdata", it was a subset of AppStream, today it isn't anymore.

The Distro-Metadata is a refined Metainfo file, which allows more tags (e.g. <bundle/> is part of the distro-data but not metainfo files) and has a different structure (see the <description/> tag for an example).

Upstream Metadata (Metainfo/Appdata):
https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html

Distro Metadata (shipped by Linux distributions, Limba and Flatpak):
https://www.freedesktop.org/software/appstream/docs/chap-DistroData.html

Quickstart Guides (helping upstream projects to write good metadata, not covering the full metainfo spec):
https://www.freedesktop.org/software/appstream/docs/chap-Quickstart.html

[ximion]

@probonopd

at least for AppImage, being able to download and run applications without the need for root rights is essential.

What do you do if /home is mounted noexec? What if the company has a policy disallowing executing stuff from /home? How do you deduplicate data in multi-user setups (where every user might install the very same app)?

Anyway, searching the locations in $HOME is cheap and will come eventually. Some other issues need to be resolved first though, before this can happen.

[probonopd]

This is what I see today when I search for Krita in Ubuntu 16.06:

So apparently the only version being displayed is 3.0-snap12 because the gnome-software snap plugin has got this version from snapd with which it has communicated. Apparently this results in the version provided by the distribution in not being shown at all, nor the description or any of the URLs. So if Krita would put in a <url type="appimage"> it would most likely also not be shown.

How are software centers supposed to decide what to show and what not to show?

[probonopd]

@ximion

What do you do if /home is mounted noexec?

Then the administrator has deliberately made the choice to disallow this kind of thing, which I think is fine.

How do you deduplicate data in multi-user setups (where every user might install the very same app)?

Put AppImages in /opt or better /usr/local, for example, to make them available for everyone.

[probonopd]

@ximion

AppStream == Metainfo files + AppStream distro metadata
Appdata == MetaInfo file

Please help me wrap my head around this. I don't quite understand the separation between upstream-provided and distro-provided information but I suspect it doesn't allow upstreams to offer their own binaries. Right now upstreams are supposed to supply only a subset of the total information; and unfortunately this subset doesn't include the <bundle> tag, right?

So what is an upstream supposed to do who wants to provide his own bundle, from his own website, without interference from distributions, and wants to have this displayed in the software centers?

Can you show me, e.g., using the Krita AppStream file, what you would recommend the Krita upstream authors to do in this case?

[ximion]

(Why is there no quote button on Github? Quoting manually is really annoying...)

@probonopd

I also understand that in order not to confuse ordinary users, development/nightly/continuous versions should not be shown too prominently, and there might be a warning when installing them, or a GUI checkbox that would beed to be checked in order for the user to see such versions. Especially with AppImages, I expect one major use case to be development versions (for the very reason that one app = one file makes it super easy to just try something out). So I am convinced that it could be beneficial to have no limit on the number of different versions possible to be offered, including historical ones. Take web browsers, for example - I am sure that web designers, for example, might want to test something not only on the latest but also on other versions of browsers.

For web browsers maybe, and for very advanced users. But for the rest, installing ancient versions with security issues is a bad thing. Especially for software centers, which AppStream is primarily designed for. AppStream really is designed with having a single "current" version where the metadata is made for, which will be installed. Installing different versions is not supported at all, and will never be. There are also other issues with this approach: For example, most apps are forward-compatible and can read the configuration files and data an older version of the app has created. Once you switch back to the older app, you might not be able to read the data anymore, which can cause data loss in the worst case (think of e.g. an email app with a non-standard mail storage method).

That being said, I am even more inclined now to just put in the web link to the project's download page instead of a deeplink to the AppImage.

Agreed - old-version links on the projects home page do make sense - in software centers, not so much.

On the following question I would really appreciate some input - what exactly is supposed to go into the bundle tag.

It's relatively free-form. Software centers link to a shared library the bundling system provides, and pass it the id found in the bundle tag to install or remove an app. What this id is exactly is up to the bundling system.

An URL from where the bundle(s) can be downloaded? Pro: User can decide which bundle to download

No, they can't - there is always one "current" bundle per component-ID.
The only way one could realize this is by making the bundle tag a child of a <release/>, but that would be weird...
(any such change would need quite a lot of changes in AppStream, and wouldn't be supported right now)

contra: cannot automatically download/update/"install". An identifier that allows an AppImage helper (AppImageUpdate or a gnome-software/discover plugin) to download/update the AppImage? Pro: More automation.

Sounds good to me.

Con: How to handle more than 1 version/channel?

You don't offer that feature. User who want a different version can get it from the developer's website.

Is there any existing convention on what goes into the bundle tag?

No, use whatever makes sense for you.

<url type="appimage">https://krita.org/en/download/krita-desktop/</url>

That would be invalid use of the <url/> tag (only for webpages), so, no, that's not the right thing to do ;-)

Greetings from Debconf/Debcamp :-)

[ximion]

@probonopd

This is what I see today when I search for Krita in Ubuntu 16.06:

Snappy doesn't use AppStream, so their metadata sucks. See https://bugs.launchpad.net/ubuntu/+source/gnome-software/+bug/1588266
(might get fixed, let's see...)

There seems to be a misconception on your side: The <bundle/> tag can't be in the metainfo file. Also, upstream metadata shouldn't see a link to 3rd-party software which software centers use. I am pretty certain that even if this was supported, the security team at Debian would beat me for not filtering this out.
Downloading things from untrusted 3rd-party sources is a no-go. Also, we can't update the data once the distro is released (at least not unless a new package is released), so this would mean a lot of dead links quickly, which is not a sign of good quality.

The <bundle/> tag is designed for scenarios where you have a repository of bundles, and then ship one software source which references those.
That's similar to a distro repo, and the way Flatpak and Limba operate as well.

Put AppImages in /opt or better /usr/local, for example, to make them available for everyone.

Yes - this should be the default mode of operation when installing stuff via GNOME Software, e.g. place the bundles in /opt/appimage and create a .desktop file in a XDG_DATA_PATH location.

Please help me wrap my head around this. Right now upstreams are supposed to supply only a subset of the total information; and unfortunately this subset doesn't include the tag, right?

Yes - the remaining data is filled in via tools like appstream-generator1 or appstream-builder2 which operate on repositories of software.
They also transform the metainfo/upstream-metadata data and information from various other sources (.desktop files!) into distro-metadata that can be shipped to the users as index of software.

So what is an upstream supposed to do who wants to provide his own bundle, from his own website, without interference from distributions, and wants to have this displayed in the software centers?

Upload it to some repository service which is generating the data and the source of which can be added as ttrusted download source to a SC. OR just submit a link to a service like that. Adding the download link to metainfo files is impractical, see the reasons given above.

[probonopd]

After IRC conversation with @ximion I understand the situation as follows:

The bundle system should provide AppStream data in the "distribution" format (AppStream distribution XML files = upstream-provided AppStream metadata = "metainfo", "appdata") from a central location. The service at the "central location" should use upstream-provided AppStream data
and enrich it, e.g., with version information and tags. This data would be downloaded by the gs plugin, and the gs plugin would use the ID given in the tag to identiy the software. The ID could be the same as the AppStream ID, but typically also includes the version; it might however include a channel instead ("krita-nightlies").

Using the existing data flows from the upstream to the distributions is probably not suitable to carry information regarding a) "a bundle exists" b) "here is how to download the bundle" c) "the following version(s) are available as a bundle" because AppStream information in distributions is frozen when the distribution is released, and because distributions might not like/remove it (viewing binaries provided by upstreams like "untrusted 3rd parties").

The only problem: We don't want a "central location", but rather want upstreams to provide this information themselves. But hey, somewhere the information must come from... any ideas appreciated.

[probonopd]

Only half-knowingly what I am doing, am I on track here @ximion?

#
# This script is supposed to populate a "central repository" of AppStream information
# by gathering this information from AppImages provided by upstream application authors
# hosted on their own servers
#

# Install prerequisites
sudo apt-get -y install appstream-util xmlstarlet httpfs2 fuseiso

APPIMAGEURL="http://ftp5.gwdg.de/pub/linux/kde/extrafiles/krita/3/linux/devbuilds/krita-3.0-7dc551d-x86_64.appimage"

# Make sure we always use http because httpfs2 doesn't like https
APPIMAGEURL=$(echo "${APPIMAGEURL}" | sed -e 's|https://|http://|g')

# Mount the AppImage over http, this way we don't have to download it entirely
mkdir mnt1 mnt2
httpfs2 "${APPIMAGEURL}" mnt1
APPIMAGE=$(find mnt1 -type f)
fuseiso "${APPIMAGE}" mnt2

# Get Update Information embedded in the AppImage,
# we want to use this for the AppStream <bundle> tag
# https://github.com/AppImage/AppImageSpec/blob/master/draft.md#updateinformation
UPDATE_INFORMATION=$(dd if="${APPIMAGE}" bs=1 skip=33651 count=512 2>/dev/null)
echo "${UPDATE_INFORMATION}"

# If there is no Update Information embedded in the AppImage, then we use the 
# download URL for the AppStream <bundle> tag
# (this will NOT allow for binary delta updates but at least we can download)
if [ -z "${UPDATE_INFORMATION}" ] ; then
  TRANSPORT=$(echo "${APPIMAGEURL}" | cut -d ":" -f 1)
  UPDATE_INFORMATION="${TRANSPORT}|${APPIMAGEURL}"
fi

# Find the relevant appdata.xml file;
# according to ximion, usr/share/appdata is a legacy path replaced by usr/share/metainfo
APPDATA=$(find mnt2/usr/share/metainfo -name "*appdata.xml" | head -n 1) # TODO: Handle more than one better
if [ -z "$APPDATA" ] ; then
  APPDATA=$(find mnt2/usr/share/appdata -name "*appdata.xml" | head -n 1) # TODO: Handle more than one better
fi
echo "${APPDATA}"

# Validate
appstreamcli validate "${APPDATA}"
# appstream-util validate-relax "${APPDATA}" # Hangs on Krita's xml

# Copy the relevant files that are necessary to put together distribution AppStream information
# so that we can edit it (i.e., insert the <bundle> tag)
mkdir -p temp/usr/share/metainfo
cp -r mnt2/usr/share/{applications,icons,metainfo,appdata} temp/usr/share

# Insert "<bundle type="appimageupdate">$UPDATE_INFORMATION</bundle>"
XMLNAME=$(basename "${APPDATA}")
TARGET="temp/usr/share/metainfo/$XMLNAME"
grep "</component>" "${APPDATA}" && xmlstarlet ed -s /component -t elem -n bundle -v "${UPDATE_INFORMATION}" -i /component/bundle -t attr -n type -v appimageupdate "${APPDATA}" > "${TARGET}"
# Sometimes there is no <component> but <application> - why is this?
grep "</application>" "${APPDATA}" && xmlstarlet ed -s /application -t elem -n bundle -v "${UPDATE_INFORMATION}" -i /application/bundle -t attr -n type -v appimageupdate "${APPDATA}" > "${TARGET}"

# Check if we have successfully inserted the <bundle> tag into the upstream AppStream information
cat "${TARGET}" | grep "</bundle>"

# Convert the upstream AppStrem information to distribution AppStream information
ID=$(basename "${APPDATA}" | sed -e 's|.appdata.xml||g')
echo "${ID}"
appstream-compose -v --origin=appimage --output-dir=xml --icons-dir=icons --prefix=temp/usr "${ID}"

# Check if we have successfully inserted the <bundle> tag into the distribution AppStream information
zcat xml/*xml.gz | grep "</bundle>"
# FIXME: Why is the type attribute lost?

# Unmount the AppImage
killall fuseiso
killall httpfs2

EDIT: Updated the script and added descriptive comments

[ximion]

Not sure what I am seeing here ^^
Can you explain a bit more about the script and what you expect it to be doing?
Validation can be done with appstreamcli validate too (maybe use both if you're really pedantic).
The compose step is only necessary to build distro-data, not sure if you want to do that here... (if so, for what purpose? does this get exported somehow?)

Additionally, the /usr/share/appdata path is a legacy path, so you should support /usr/share/metainfo only (if you can enforce people to support the new path (which they should do anyway)), or both paths.

[probonopd]

Thanks @ximion.

What do I want to achieve? I want to gather all necessary information about AppImages that are hosted by upstreams, insert the <bundle> tag, and convert it to a database in the "distribution AppStream metadata" format that could be used by software centers (or plugins thereof).

I have updated the script above and added descriptive comments. Would be great if you could check what I am doing there. The <bundle> tag seems to get lost in the process...

[ximion]

Doesn't look bad - I don't know how the appstream-compose command is working exactly, that is hughsie's doing ^^ (you probably need to pass it a spec version to always have it output the right format).

To avoid the hackery with <bundle/>, you could simply request its inclusion into the spec :P Before it's in there, it can't be supported anyway.
Just file a bug at https://github.com/ximion/appstream/issues , or count on me not forgetting to do this.

The origin property of the distro-metadata file shouldn't be the name of the bundling system, but the repository name, for example debian-jessie-main (maybe make that configurable).

Since I just finished preparing my AppStream talk, I might have time tomorrow to work on AppStream a bit (before starting to prepare my other talk on bundling ^^).
There are a few things I wanted to change in AppStream anyway, adding another bundling system type would be trivial.
(for the compose thing to work, asglib will need to implement this as well, though)

[probonopd]

There is now official support for AppImage in AppStream: ximion/appstream@1df590d

Thanks @ximion!

[probonopd]

https://github.com/openSUSE/software-o-o/ seems to be the code that powers https://software.opensuse.org/appstore - a ready implementation of an app store

[probonopd]

@ximion @aleixpol @hughsie
Continuing from a Gitter discussion. Looks like AppStream has AppImage bundles in place, now the question is, where in the system can a user (not: root) put additional AppStream XML files so that they get picked up from software centers like KDE Discover and GNOME Software.

aleixpol: I saw you mentioned software centers, is the appstream part ready?

probonopd: @aleixpol according to https://github.com/ximion/appstream/search?utf8=✓&q=appimage there is now an AppImageKit bundle type. The only question is, where does appimaged have to put the xml files for discover to pick them up? I am voting for $HOME/.local/share/metainfo/, agree?

aleixpol: I agree, I would discuss it with ximion though, his software needs to pick it up after all.
We also need to have a service for searching somewhere, do you plan to provide it? Or you just want to integrate for updates?

probonopd: @aleixpol to be more precise, "$XDG_DATA_HOME defines the base directory relative to which user specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used."
@aleixpol can you point me to the docs/api for searching? is it a local or a server search?

aleixpol: @probonopd https://www.freedesktop.org/software/appstream/docs/chap-CollectionData.html#sect-AppStream-XML

So far I was under the impression that it would be enough to populate the bundle tag in a metainfo file and put that into $HOME/.local/share/metainfo/, without the need for a collection data (distribution) XML. Perhaps @ximion can shed some light on this? I am still struggling with the architectural distinction of the two.

[ximion]

@probonopd AppStream doesn't support data in $HOME directories yet, mainly due to my very strong aversion to installing anything in user's home. Adding this will happen though, since it's useful for testing and Flatpak needs it too, I think appstream-glib does support it already even. The directory will be ~/.local/share/metainfo/, nothing else makes sense.

Collection metadata in /var/lib/app-info etc. describes available software components. The metainfo files are only on the system and can only be read when a software is installed. Collection data is generated from the metainfo files.

So, if you just want to register your app once it is installed, dropping a metainfo file is enough. If you want a software catalog-like feature, a collection file is usually what you need.

[hughsie]

Err, appstream-glib has supported ~/.local/share/app-info/xmls since its inception...

[ximion]

@hughsie I find it disturbing that people would be able to override system metadata with stuff in their home directory. When I add it (highly likely with the next release), there will be a couple of checks for that.
Back then when I considered it, the benefit of it was zero and the concept of "scope" for metadata didn't exist yet and nobody signaled that this would be used. That's quite different now.

[hughsie]

gnome-software uses the "user" app-info location for the gnome-shell extensions support (which is per-user) and also for the steam database (which is also per-user). The other thing it's hugely useful for is testing the various things...

[probonopd]

@hughsie I find it disturbing that people would be able to override system metadata with stuff in their home directory. When I add it (highly likely with the next release), there will be a couple of checks for that.

With most configuration, it is actually the other way around: The system provides r/o defaults which the user can override in r/w $HOME.

I would like to provide per-user catalog data (as in: the user doesn't need root rights to read the catalog data and the application that gets/downloads/generates the catalog data also doesn't need root rights). This doesn't have to override system-provided information, but it should augment it.

For example: LTS distribution comes with Krita 2.5 but my per-user catalog data has 3.1. The software center (GS, Discover, etc.) would either

• Show 2 entries (product detail pages) for Krita, or (even better)
• Merge this information on one product detail page, saying "The following versions are available: 2.5 through the distribution, 3.0 through Flatpak, 3.1 through Snap and AppImage", or (even better)
• Allow for the concept of "channels", saying on the product detail page "The following versions are available: 2.5 through the distribution, 3.0 through Flatpak and in the AppImage Releases Channel, 3.1 in the AppImage Beta Channel, and git.230576 in the AppImage Continuous Builds Channel.

One can dream... :-)

[ximion]

That channels stuff doesn't exist, but everything else is pretty much there.

[TheAssassin]

There hasn't been any activity for quite a while. Is any of this still relevant?

[probonopd]

Yes. This is still to be figured out.

[probonopd]

Started working on https://github.com/AppImage/AppImageHub/. Is there an appstreamcli for trusty 14.04 available @ximion or could you provide a ppa/backport?

[ximion]

@probonopd Trusty has AppStream, but the version in there is ancient and should really not be used. Ideally, use Xenial, which is much better maintained.
A PPA for Trusty would be possible if the GLib version in Trusty was high enough. Otherwise, we would need to backport that as well and maybe other components, which is a bit annoying (GLib is a pretty central component, afterall).

[probonopd]

We could make it into an AppImage that bundles everything, https://github.com/lvml/makeaoi :-)

[probonopd]

Side note, it seems to me that using a scripting language for linting would be better suitable since it would allow to lint AppStream files on LTS/Enterprise distributions while still getting the latest rules for linting more easily...

[probonopd]

It's being implemented now at https://github.com/AppImage/appimage.github.io/, hence closing here.