Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AppStream 1.0 #47

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open

AppStream 1.0 #47

wants to merge 24 commits into from

Conversation

probonopd
Copy link
Owner

@probonopd probonopd commented Mar 15, 2024

This PR shall build AppStream v1.0.x instead of what we have been using so far.

probonopd/go-appimage#272

Note to self: If this doesn't work, a possibly less painful alternative might be https://github.com/probonopd/appstreamlint

@probonopd
Copy link
Owner Author

It uses meson as a build time tool. Every time meson is used, building the latest version means that meson probably needs to be updated as well (which, in practical terms, means that the whole Linux distribution needs to be updated). So annoying. I don't like moving targets.

@probonopd
Copy link
Owner Author

probonopd commented Mar 15, 2024

Newer versions of things always seem to require more dependencies. Argh!

Even with curl we get errors:

Run-time dependency libcurl found: YES 8.5.0
WARNING: Static library 'xml2' not found for dependency 'libxml-2.0', may not be statically linked
WARNING: Static library 'lzma' not found for dependency 'libxml-2.0', may not be statically linked
Run-time dependency libxml-2.0 found: YES 2.11.7
Run-time dependency yaml-0.1 found: YES 0.2.5
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency xmlb found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency xmlb
Building fallback subproject with default_library=static

Executing subproject libxmlb 

libxmlb| Project name: libxmlb
libxmlb| Project version: 0.3.16
libxmlb| C compiler for the host machine: cc (gcc 13.2.1 "cc (Alpine 13.2.1_git20231014) 13.2.1 20231014")
libxmlb| C linker for the host machine: cc ld.bfd 2.41
libxmlb| Compiler for C supports arguments -Wno-nonnull-compare: YES
libxmlb| Compiler for C supports arguments -Wno-aggregate-return: YES
libxmlb| Compiler for C supports arguments -Wunused: YES
libxmlb| Compiler for C supports arguments -Warray-bounds: YES
libxmlb| Compiler for C supports arguments -Wcast-align: YES
libxmlb| Compiler for C supports arguments -Wclobbered: YES
libxmlb| Compiler for C supports arguments -Wdeclaration-after-statement: YES
libxmlb| Compiler for C supports arguments -Wduplicated-branches: YES
libxmlb| Compiler for C supports arguments -Wduplicated-cond: YES
libxmlb| Compiler for C supports arguments -Wempty-body: YES
libxmlb| Program python3 found: YES (/usr/bin/python3)
libxmlb| subprojects/libxmlb/data/meson.build:2: WARNING: Project targets '>=0.47.0' but uses feature introduced in '0.50.0': install arg in configure_file.
libxmlb| Configuring libxmlb.test using configuration
libxmlb| Configuring xb-version.h using configuration
libxmlb| Compiler for C supports link arguments -Wl,--version-script,/appstream-1.0.2/subprojects/libxmlb/src/libxmlb.map: YES
libxmlb| subprojects/libxmlb/src/meson.build:121: WARNING: Project targets '>=0.47.0' but uses feature introduced in '0.50.0': install arg in configure_file.
libxmlb| Configuring xb-tool.1 using configuration
libxmlb| subprojects/libxmlb/src/meson.build:230: WARNING: Project targets '>=0.47.0' but uses feature introduced in '0.56.0': meson.project_source_root.
libxmlb| subprojects/libxmlb/src/meson.build:265: WARNING: Project targets '>=0.47.0' but uses feature introduced in '0.56.0': meson.project_source_root.
libxmlb| Build targets in project: 4
libxmlb| WARNING: Project specifies a minimum meson_version '>=0.47.0' but uses features which were added in newer versions:
libxmlb| * 0.50.0: {'install arg in configure_file'}
libxmlb| * 0.56.0: {'meson.project_source_root'}
libxmlb| NOTICE: Future-deprecated features used:
libxmlb| * 0.64.0: {'copy arg in configure_file'}
libxmlb| Subproject libxmlb finished.

Dependency xmlb from subproject subprojects/libxmlb found: YES 0.3.16

meson.build:169:47: ERROR: Function does not take keyword arguments.

A full log can be found at /appstream-1.0.2/build/meson-logs/meson-log.txt
Error: Process completed with exit code 1.

Meson is never new enough. Reason enough to avoid Meson altogether.
Developers always seem to assume that people building their software run the latest and greatest distributions. A never ending update cycle, a.k.a. "moving target".

@probonopd
Copy link
Owner Author

probonopd commented Mar 15, 2024

@ximion is there a way to build appstream 1.0 without curl?

Right now we are using a static build of a pre-1.0 version, and would like to go to 1.0 in the hope that the AppStream format is no longer subject to change.

@ximion
Copy link

ximion commented Mar 15, 2024

is there a way to build appstream 1.0 without curl?

No, it is required for the portions of the spec that require network access.
The format will of course still change, what happened with 1.0 was the removal of deprecated features. Removal of deprecated stuff will (very likely) never happen again.

@probonopd
Copy link
Owner Author

probonopd commented Mar 25, 2024

Thanks @ximion. Do you happen to know how to produce a static build of appstreamcli on Alpine? We are using an older version of it, built I am so far not succeeding in getting 1.x built statically, since it seems to pull in additional dependencies which the earlier version did not need.

@ximion
Copy link

ximion commented Mar 25, 2024

What kind of issue do you have? AppStream 1.0 doesn't have dependencies that 0.16.4 didn't also have, I think.

@probonopd
Copy link
Owner Author

Coming from 0.12.9. Currently running into meson.build:169:47: ERROR: Function does not take keyword arguments, whatever that might mean.

https://github.com/probonopd/static-tools/actions/runs/8422233832/job/23061071735#step:4:1516

@probonopd
Copy link
Owner Author

probonopd commented Mar 25, 2024

All I need is the appstreamcli command to check the validity of ("lint") AppStream MetaInfo files. However, the AppStream repo now includes much more functionality that draws in many more dependencies, e.g.,:

  • libxmlb is now used for caching things. I think for appstreamcli to check files, we could do without a cache ximion/appstream@a8186ed
  • libzstd is now used to compress (which?) things. I think for appstreamcli to check files, we could do without compression ximion/appstream@1bde5bf

Is there a way to just compile the appstreamcli command with the minimal dependencies needed to validate ("lint") files?

@ximion
Copy link

ximion commented Mar 25, 2024

Coming from 0.12.9. Currently running into meson.build:169:47: ERROR: Function does not take keyword arguments, whatever that might mean.

That error indeed makes no sense, the function on that line isn't passed any kwargs. Can you easily upgrade to Meson 1.3.2, just to check if that helps?

@probonopd
Copy link
Owner Author

I don't think I can (easily) update Meson any further, as I have already updated to the latest version of Alpine as part of this PR.

@ximion
Copy link

ximion commented Mar 25, 2024

I was asking because this smells a bit like a Meson bug. Do you apply any patches to AppStream? What is in the line that it complains about in your copy? Because the upstream line does not make any sense here.

@probonopd
Copy link
Owner Author

probonopd commented Mar 25, 2024

Yes, I was indeed wondering the same thing. No, I am not applying any patches, except for some ad-hoc meson.build file patches (I don't have much clue about Meson, really). My code is here:

static-tools/build.sh

Lines 90 to 115 in 0aea9a2

# Build appstreamcli
trap 'cat /appstream-1.0.2/build/meson-logs/meson-log.txt' EXIT
apk add glib-static meson libxml2-dev yaml-dev yaml-static gperf curl-dev curl-static curl libxmlb-dev
# Compile liblmdb from source as Alpine only ship it as a .so
wget https://git.openldap.org/openldap/openldap/-/archive/LMDB_0.9.29/openldap-LMDB_0.9.29.tar.gz
tar xf openldap-LMDB_*.tar.gz
cd openldap-LMDB_*/libraries/liblmdb
make liblmdb.a
install -D -m 644 liblmdb.a /usr/local/lib/liblmdb.a
install -D -m 644 lmdb.h /usr/local/include/lmdb.h
cd -
wget -O appstream.tar.gz https://github.com/ximion/appstream/archive/v1.0.2.tar.gz # Keep at v1.0.x so as to not have a moving target
tar xf appstream.tar.gz
cd appstream-*/
# Ask for static dependencies
sed -i -E -e "s|(dependency\('.*')|\1, static: true|g" meson.build
# Disable po, docs and tests
sed -i -e "s|subdir('po/')||" meson.build
sed -i -e "s|subdir('docs/')||" meson.build
sed -i -e "s|subdir('tests/')||" meson.build
# -no-pie is required to statically link to libc
CFLAGS=-no-pie LDFLAGS=-static meson setup build --buildtype=release --default-library=static --prefix="$(pwd)/prefix" --strip -Db_lto=true -Db_ndebug=if-release -Dstemming=false -Dgir=false -Dapidocs=false
# Install in a staging enviroment
meson install -C build
file prefix/bin/appstreamcli
cd -

@ximion
Copy link

ximion commented Mar 25, 2024

Can you please drop the

sed -i -e "s|subdir('po/')||" meson.build
sed -i -e "s|subdir('docs/')||" meson.build
sed -i -e "s|subdir('tests/')||" meson.build

lines? That is no way to disable anything, there is a lot of interdependencies here which can cause all kinds of issues. Dropping po/ is probably harmless, but the rest isn't! Use -Dapidocs=false -Dinstall-docs=false for the same effect instead (and just remove translations after installation to a known location).

You can also drop the entire LMDB block, unless something else needs it - because AppStream does not require it anymore for a long time.

Besides all of that, sed -i -E -e "s|(dependency\('.*')|\1, static: true|g" meson.build is indeed the line that wrecks this build, it creates an invalid definition. Instead of that hack, just pass -Ddefault_library=static to Meson and you should be set.

@probonopd
Copy link
Owner Author

probonopd commented Mar 25, 2024

Thanks for your help @ximion.
Now it also needs systemd, which doesn't even exist on Alpine Linux. Argh!
I just want to check some files...

The original command asks for a password interactively, which does not work on CI
@ximion
Copy link

ximion commented Mar 25, 2024

Now it also needs systemd, which doesn't even exist on Alpine Linux. Argh!
I just want to check some files...

It's not essential if you just need validation and don't need hardware information. Just pass -Dsystemd=false (device capability checking will not work then, but you won't need that for validation).

@ximion
Copy link

ximion commented Mar 25, 2024

libxmlb has -Dintrospection=false -Dgtkdoc=false -Dcli=false switches that you may like ;-)

@probonopd
Copy link
Owner Author

Just was gonna elaborate... erm, complain about, that we are now firmly in Gnome land, with stuff like gobject-introspection-dev...

In an effort to fix `/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object '/usr/lib/libglib-2.0.so'`
@probonopd
Copy link
Owner Author

probonopd commented Mar 25, 2024

Almost there...

[141/146] Linking target tests/as-test_xml
ninja: job failed: cc  -o tests/as-test_yaml tests/as-test_yaml.p/meson-generated_.._.._src_as-enum-types.c.o tests/as-test_yaml.p/test-yamldata.c.o tests/as-test_yaml.p/as-test-utils.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -static -no-pie -Wl,--start-group src/libappstream.a /usr/lib/libglib-2.0.so /usr/lib/libintl.so /usr/lib/libgobject-2.0.so /usr/lib/libgio-2.0.so /usr/lib/libxml2.so /usr/lib/libcurl.so /usr/local/lib/libxmlb.a /usr/lib/liblzma.so /usr/lib/libzstd.so /usr/lib/libyaml.so -Wl,--end-group
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/usr/lib/libglib-2.0.so'
collect2: error: ld returned 1 exit status
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/usr/lib/libglib-2.0.so'
collect2: error: ld returned 1 exit status
ninja: job failed: cc  -o tests/as-test_xml tests/as-test_xml.p/meson-generated_.._.._src_as-enum-types.c.o tests/as-test_xml.p/test-xmldata.c.o tests/as-test_xml.p/as-test-utils.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -static -no-pie -Wl,--start-group src/libappstream.a /usr/lib/libglib-2.0.so /usr/lib/libintl.so /usr/lib/libgobject-2.0.so /usr/lib/libgio-2.0.so /usr/lib/libxml2.so /usr/lib/libcurl.so /usr/local/lib/libxmlb.a /usr/lib/liblzma.so /usr/lib/libzstd.so /usr/lib/libyaml.so -Wl,--end-group
ninja: job failed: cc  -o tests/as-test_validate tests/as-test_validate.p/meson-generated_.._.._src_as-enum-types.c.o tests/as-test_validate.p/test-validate.c.o tests/as-test_validate.p/as-test-utils.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -static -no-pie -Wl,--start-group src/libappstream.a /usr/lib/libglib-2.0.so /usr/lib/libintl.so /usr/lib/libgobject-2.0.so /usr/lib/libgio-2.0.so /usr/lib/libxml2.so /usr/lib/libcurl.so /usr/local/lib/libxmlb.a /usr/lib/liblzma.so /usr/lib/libzstd.so /usr/lib/libyaml.so -Wl,--end-group
ninja: job failed: cc  -o tests/as-test_misc tests/as-test_misc.p/meson-generated_.._.._src_as-enum-types.c.o tests/as-test_misc.p/test-misc.c.o tests/as-test_misc.p/as-test-utils.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -static -no-pie -Wl,--start-group src/libappstream.a /usr/lib/libglib-2.0.so /usr/lib/libintl.so /usr/lib/libgobject-2.0.so /usr/lib/libgio-2.0.so /usr/lib/libxml2.so /usr/lib/libcurl.so /usr/local/lib/libxmlb.a /usr/lib/liblzma.so /usr/lib/libzstd.so /usr/lib/libyaml.so -Wl,--end-group
ninja: job failed: cc  -o tests/as-test_perf tests/as-test_perf.p/meson-generated_.._.._src_as-enum-types.c.o tests/as-test_perf.p/test-performance.c.o tests/as-test_perf.p/as-test-utils.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -static -no-pie -Wl,--start-group src/libappstream.a /usr/lib/libglib-2.0.so /usr/lib/libintl.so /usr/lib/libgobject-2.0.so /usr/lib/libgio-2.0.so /usr/lib/libxml2.so /usr/lib/libcurl.so /usr/local/lib/libxmlb.a /usr/lib/liblzma.so /usr/lib/libzstd.so /usr/lib/libyaml.so -Wl,--end-group
ninja: job failed: cc  -o tests/as-test_pool tests/as-test_pool.p/meson-generated_.._.._src_as-enum-types.c.o tests/as-test_pool.p/test-pool.c.o tests/as-test_pool.p/as-test-utils.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -static -no-pie -Wl,--start-group src/libappstream.a /usr/lib/libglib-2.0.so /usr/lib/libintl.so /usr/lib/libgobject-2.0.so /usr/lib/libgio-2.0.so /usr/lib/libxml2.so /usr/lib/libcurl.so /usr/local/lib/libxmlb.a /usr/lib/liblzma.so /usr/lib/libzstd.so /usr/lib/libyaml.so -Wl,--end-group
ninja: subcommands failed
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: as-validator.c.o (symbol from plugin): in function `as_validator_issue_tag_list':
(.text+0x0): multiple definition of `as_validator_issue_tag_list'; tests/as-test_validate.p/test-validate.c.o (symbol from plugin):(.text+0x0): first defined here
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/usr/lib/libglib-2.0.so'
collect2: error: ld returned 1 exit status
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/usr/lib/libglib-2.0.so'
collect2: error: ld returned 1 exit status
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/usr/lib/libglib-2.0.so'
collect2: error: ld returned 1 exit status
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: attempted static link of dynamic object `/usr/lib/libglib-2.0.so'
collect2: error: ld returned 1 exit status
Could not rebuild /appstream-1.0.2/build
Error: Process completed with exit code 255.

What is it this time?
Why is it trying to link .so files instead of .a?

This whole exercise is a great reminder why I like Golang so much. Writing https://github.com/probonopd/appstreamlint from scratch was a breeze compared to getting appstreamcli 1.0 built as a static binary.

@ximion
Copy link

ximion commented Mar 25, 2024

If you would just use LD_LIBRARY_PATH, you could avoid the static linking annoyance completely ;-)

@probonopd
Copy link
Owner Author

probonopd commented Nov 3, 2024

Playing around with bundling the appstreamcli binary from Alpine Linux, it seems like the following libraries are all direct or indirect dependencies - does this look about right?

4.5M    ./usr/lib/libcrypto.so.3
2.1M    ./usr/lib/libgio-2.0.so.0.8200.2
1.7M    ./usr/lib/libunistring.so.5.1.0
1.4M    ./usr/lib/libglib-2.0.so.0.8200.2
1.1M    ./usr/lib/libxml2.so.2.13.4
812K    ./usr/lib/libssl.so.3
728K    ./usr/lib/libbrotlienc.so.1.1.0
704K    ./usr/lib/libzstd.so.1.5.6
688K    ./usr/lib/libpcre2-8.so.0.12.0
640K    ./usr/lib/libcurl.so.4.8.0
636K    ./lib/ld-musl-x86_64.so.1
600K    ./usr/lib/libappstream.so.1.0.3
408K    ./usr/lib/libgobject-2.0.so.0.8200.2
280K    ./usr/lib/libmount.so.1.1.0
264K    ./usr/lib/libgirepository-2.0.so.0.8200.2
232K    ./usr/lib/liblzma.so.5.6.3
208K    ./usr/lib/libcares.so.2.19.1
200K    ./usr/lib/libidn2.so.0.4.0
200K    ./usr/lib/libblkid.so.1.1.0
160K    ./usr/lib/libxmlb.so.2.0.0
152K    ./usr/lib/libnghttp2.so.14.28.3
144K    ./usr/lib/libbrotlicommon.so.1.1.0
112K    ./usr/lib/libyaml-0.so.2.0.9
108K    ./usr/lib/libz.so.1.3.1
104K    ./usr/lib/ossl-modules/legacy.so
80K     ./usr/lib/libpsl.so.5.3.5
72K     ./usr/lib/libintl.so.8.4.0
60K     ./usr/lib/libbrotlidec.so.1.1.0
60K     ./usr/lib/engines-3/loader_attic.so
44K     ./usr/lib/libffi.so.8.1.4
44K     ./usr/lib/libeconf.so.0.6.2
28K     ./usr/lib/libgmodule-2.0.so.0.8200.2
28K     ./usr/lib/engines-3/padlock.so
24K     ./usr/lib/engines-3/afalg.so
20K     ./usr/lib/libpcre2-posix.so.3.0.5
20K     ./usr/lib/libgthread-2.0.so.0.8200.2
20K     ./usr/lib/engines-3/capi.so

Getting rid of libcrypto.so.3 alone would already reduce the AppImage significantly.

@ximion
Copy link

ximion commented Nov 3, 2024

Looks fine, most dependencies likely come in via Curl to support loading data from the web.

@Samueru-sama
Copy link

Relevant info

Looks like you can build it fully static on musl if you remove the systemd dependency.

@probonopd
Copy link
Owner Author

probonopd commented Nov 7, 2024

Interesting. I am already using -Dsystemd=false, so we need to change something else in this PR. Based on what you linked to, can you find out what exactly we need to change @Samueru-sama?

@Azathothas
Copy link

As per @leleliu008,
The build recipe looks like this:
https://github.com/leleliu008/ppkg-formula-repository-official-core/blob/master/formula/appstream.yml

As per the comment on: leleliu008/ppkg-formula-repository-official-core#7 ,

You had just need to remove the systemd depdency and change systemd to false:
https://github.com/leleliu008/test/blob/master/.github/workflows/appstream.yml#L46

@probonopd
Copy link
Owner Author

We don't have a ~/.ppkg/repos.d/official-core/formula/appstream.yml, so the question is which changes do we have to make to https://github.com/probonopd/static-tools/pull/47/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants