Prebuilt mediasoup-worker fails in Debian Bullseye 11: version `GLIBC_2.33' not found

[ibc]

Bug Report

After #1087 is merged and released, npm i mediasoup in Debian Bullseye 11 fetches mediasoup-worker-3.12.0-linux-x64.tgz from mediasoup GitHub Releases:https://github.com/versatica/mediasoup/releases.

Then it fails in runtime. Easy to reproduce by just running the worker directly:

~/node_modules/mediasoup$ ./worker/out/Release/mediasoup-worker

./worker/out/Release/mediasoup-worker: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./worker/out/Release/mediasoup-worker)
./worker/out/Release/mediasoup-worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./worker/out/Release/mediasoup-worker)
./worker/out/Release/mediasoup-worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./worker/out/Release/mediasoup-worker)
./worker/out/Release/mediasoup-worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./worker/out/Release/mediasoup-worker)

Your environment

• Operating system: Debian Bullseye 11
• Node version: 18.16.0
• npm version: 9.6.7
• gcc/clang version: gcc (Debian 10.2.1-6) 10.2.1 20210110
• mediasoup version: 3.12.0

[ibc]

@barlock help please

[ibc]

I've found a probably completely unrelated fix (no idea if it makes sense, haven't tested it because no idea about it): https://github.com/anbox/anbox/pull/1892/files

[ibc]

Problem seems to be that my Debian host has different version of libc (2.31) installed:

/lib/x86_64-linux-gnu$ ls -l libc*

-rwxr-xr-x 1 root root 1.9M Apr 19 23:17 libc-2.31.so*
lrwxrwxrwx 1 root root   12 Apr 19 23:17 libc.so.6 -> libc-2.31.so*
lrwxrwxrwx 1 root root   18 Nov 20  2020 libcap-ng.so.0 -> libcap-ng.so.0.0.0
-rw-r--r-- 1 root root  27K Nov 20  2020 libcap-ng.so.0.0.0
lrwxrwxrwx 1 root root   14 Oct 14  2020 libcap.so.2 -> libcap.so.2.44
-rw-r--r-- 1 root root  38K Oct 14  2020 libcap.so.2.44
lrwxrwxrwx 1 root root   17 Jun  7  2021 libcom_err.so.2 -> libcom_err.so.2.1
-rw-r--r-- 1 root root  18K Jun  7  2021 libcom_err.so.2.1
lrwxrwxrwx 1 root root   17 Apr 19  2021 libcrypt.so.1 -> libcrypt.so.1.1.0
-rw-r--r-- 1 root root 198K Apr 19  2021 libcrypt.so.1.1.0
lrwxrwxrwx 1 root root   23 Feb  1  2022 libcryptsetup.so.12 -> libcryptsetup.so.12.6.0
-rw-r--r-- 1 root root 484K Feb  1  2022 libcryptsetup.so.12.6.0

while prebuilt mediasoup-worker requires... 4 different versions???

./worker/out/Release/mediasoup-worker: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./worker/out/Release/mediasoup-worker)
./worker/out/Release/mediasoup-worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./worker/out/Release/mediasoup-worker)
./worker/out/Release/mediasoup-worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./worker/out/Release/mediasoup-worker)
./worker/out/Release/mediasoup-worker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./worker/out/Release/mediasoup-worker)

[ibc]

Working on a workaround: #1090

[barlock]

Interesting, I'm afraid this type of error is far out of my wheel house. I have a few folks on my team who know c++, I'll have someone take a look to see if they can make sense of it and make better suggestions.

Immediate things that come to mind would be:

• Can the prebuilds be entirely statically linked? They would be bigger, I don't know by how much, but that could be the price of shorter build times. The repo gives plenty of hooks to build your own simply if the downstream user cared about saving a few kb in deployments.
• Remove linux support for prebuilds for the time being? If you remove the prebuild asset from the github release you'll keep that specific version working. I think you'd only need to remove the linux variant unless we hear of other reports.

[ibc]

1. Can we compile the worker statically against libc??? It would surprise me but who knows. Problem is that this may require same thing to be done in every C subproject (libuv, libsrtp, openssl, etc) and built in deps such libwebrtc.
2. Removing Linux support would be... wow, terrible 😀. BTW take a look to the PR I've merged right now. It fallbacks to locally building the worker if the prebuilt one doesn't run in local host.

[barlock]

libc??? It would surprise me but who knows.

See my ignorance of c projects 😅

Removing Linux support would be... wow, terrible

I meant just for prebuilds, not entirely. Then linux would function the same way it always had. Your fix is a clever one for sure and probably better than just removing it. I was thinking more as you'll likely need a new version of the project to publish your fix (3.12.1?) for 3.12.0, you can remove the linux asset and anyone using it won't be broken.

[ibc]

Yep, 3.12.1 should workaround the issue in Linux. Just wondering if we can avoid the issue entirely. Perhaps prebuilt binaries should be more OS specific. Absolutely no idea.

[nazar-pc]

As I was commenting in one of the older PRs, downloading should happen for a specific distro, specific version. If either distro or version do not match, it should be cmpiled frm scratch.

[nazar-pc]

You shold be able to compile static binary with musl libc, with Glibc it is not possible AFAIK.

[barlock]

downloading should happen for a specific distro, specific version.

Easy enough to add more runners across the spectrum that GitHub offers. Will also add a lot of good "test cases" for supported platforms.

Honest question here, is it really distro/version though? I would assume there's something more runtime specific to check against (like libc version as mentioned here).

[ibc]

And how to the get OS name and versión to name the prebuilts? Some uname command (not in Windows AFAIU) with specific options? I have bad experience using it.

[nazar-pc]

Glibc generally is forwards compatible. So if you compile for Ubuntu 18.04, it should also run on 20.04, 22.04 and all versions in between. And we don't depend on specific versions of other dependencies, we compile them from source. And for Linux you don't care about host OS, it is possible to build software for any distro and any architecture in a container.

/etc/issue should be present on most Linux distros, there is also https://www.freedesktop.org/software/systemd/man/os-release.html, but I think someone must have already built NPM package for this purpose that we can use rather than rolling a custom solution.

Overall worth looking at other projects that ship prebuilt stuff, this is not a new problem, it was solved many times before.

[ibc]

Glibc generally is forwards compatible. So if you compile for Ubuntu 18.04, it should also run on 20.04, 22.04 and all versions in between

So maybe the problem is that we are building in Ubuntu 22.04 and then trying to use in Debian stable which comes with an older version of glib?

[snnz]

So maybe the problem is that we are building in Ubuntu 22.04 and then trying to use in Debian stable which comes with an older version of glib?

Yes, glibc uses symbol versioning. The program references the latest available versions of functions/structures provided by libc in the build environment. Possible workarounds are:

• build on the platform with the oldest libc that has all necessary features;
• use symver pseudo-op to link to a particular symbol version;
• compile against the fake libc built without versioning information, to fall back to default versions at runtime;
• strip versioning information with a tool like patchelf.

[ibc]

build on the platform with the oldest libc that has all necessary features;

I'll write a PR today using older Ubuntu 20.04 to generate the prebuilt and will check it.

use symver pseudo-op to link to a particular symbol version;

I have no idea about gLib. Any tip about how to do it in Ubuntu 22.04, please?

compile against the fake libc built without versioning information, to fall back to default versions at runtime;

And how to do that? 😀

strip versioning information with a tool like patchelf.

How to do it?

Thanks a lot.

[ibc]

build on the platform with the oldest libc that has all necessary features;

I'll write a PR today using older Ubuntu 20.04 to generate the prebuilt and will check it.

Trying it here: #1091

[ibc]

I confirm that using Ubuntu 20.04 (instead of 22.04) to build the Linux prebuilt makes the binary work in Debian 11.

[JeDaYoshi]

I say it'd be better to build a statically-linked executable with musl libc like @nazar-pc suggested, rather than praying that the glibc version is compatible, to avoid issues of the kind.

It'd be worth trying out.

[ibc]

I say it'd be better to build a statically-linked executable with musl libc like @nazar-pc suggested, rather than praying that the glibc version is compatible, to avoid issues of the kind.

I have no idea how to do it. Also, in case the prebuilt worker is not compatible with the host the mediasoup install process builds it from scratch, so this is no longer an urgent issue. Said that, feel free to comment how to statically link the worker with such a "musl glib" (I don't know what that is).

[nazar-pc]

I have no idea how to do it. Also, in case the prebuilt worker is not compatible with the host the mediasoup install process builds it from scratch, so this is no longer an urgent issue. Said that, feel free to comment how to statically link the worker with such a "musl glib" (I don't know what that is).

I think the easiest is to compile under Alpine Linux, then IIRC it should result in statically linked executable, Alpine Linux is using musl by default.

[JeDaYoshi]

@nazar-pc It will not result in a static executable as-is, since you'd need to pass the variables to the compiler, so it builds a static binary. It'd just depend on musl libc instead of glibc.

It could be done on Alpine Linux, but musl can also be installed on Debian/Ubuntu, and is packaged on the repos: https://packages.ubuntu.com/jammy/musl

[nazar-pc]

On Ubuntu musl seems to be significantly harder to use comparing to Alpine Linux. But yes, we'd have to explicitly compile static version (we might already, I'm too lazy to check).

[ibc]

So if we use an alpine machine in CI to build the worker:

• Do we need to setup something else to make it use musl?
• Do we need to setup something else to get the worker super statically built so the binary can work in any Linux host with same arch without having to install musl in the target machine?

[nazar-pc]

No need for a machine, any Linux able to run Alpine container image will work. I'd try the container and then inspect the binary afterwards.

[ibc]

Do you mean that we should install Docker in the ubuntu-22.04 machine of CI then run all those things?

[nazar-pc]

It is already installed by default, but yeah

[ibc]

I'll put this at the bottom of my endless TODO list because I have very little knowledge about GitHub CI and I don't know what "it's already installed" means.

[barlock]

FWIW, if someone can create the build target to be run in alpine that will generate the binary, I'm happy to submit the PR to automate the build.

[ibc]

create the build target to be run in alpine that will generate the binary

Not sure if I understand. Why can't that be part of the PR you may write?

[barlock]

I don't know how to compile c projects, but I do know how to orchestrate builds and write automations with docker and GHA.

I was offering that, if someone else could create the script to build the binary with a statically linked musl libc that we could use for a generic linux binary, I'd be happy to automate it.

If there's a human who has both of those skills, I'm obviously not necessary.