Ubuntu 22.10 - invalid entry point / GLIBC_2.34 not found

[sebastianschauenburg]

Describe the bug
Trying to use/run toolbox on Ubuntu 22.10 with rootless podman and it fails. The Ubuntu image from docker fails, but found a couple of bugreports for that. The Debian image from docker hub seems to have a similar issue as the Fedora image below:

toolbox enter --verbose fedora-test
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user swat 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /home/user/.config/toolbox 
DEBU Current Podman version is 3.4.4              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.4.4                  
DEBU Migration not needed: Podman version 3.4.4 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'fedora-test'                   
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'fedora-test'                   
DEBU Image: 'fedora-toolbox:33'                   
DEBU Release: '33'                                
DEBU Checking if container fedora-test exists   
DEBU Inspecting mounts of container fedora-test 
DEBU Starting container fedora-test             
DEBU Inspecting entry point of container fedora-test
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-test is toolbox (PID=0) 
Error: invalid entry point PID of container fedora-test

podman start --attach fedora-test
toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by toolbox)  

Steps how to reproduce the behaviour
See above.

Expected behaviour
I expected to be entered into a toolbox instance.

Actual behaviour
See above.

Output of toolbox --version (v0.0.90+)
toolbox version 0.0.99.2

Toolbox package info (rpm -q toolbox)
podman-toolbox is 0.0.99.2-2ubuntu1

Output of podman version

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Info about your OS
Ubuntu 22.10, x64

Additional context
Have not had toolbox running on this machine before and want to use it. podman / podman-compose works though (rootless).

[angiglesias]

@HarryMichal @sebastianschauenburg This problem it's probably related to #1001 due to conflicting libs versions in the host and the container and DT_RUNPATH not being transitive for dynamic loaded libs during runtime

[angiglesias]

@sebastianschauenburg @HarryMichal @debarshiray I've looked it up and binaries shipped on Ubuntu 22.10 in the package podman-toolbox are missing the custom RUNPATH pointing to host libs.

$ readelf -d /usr/bin/toolbox

Dynamic section at offset 0x44ee58 contains 20 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x402000
 0x000000000000000d (FINI)               0x61ab74
 0x000000006ffffef5 (GNU_HASH)           0x400438
 0x0000000000000005 (STRTAB)             0x401050
 0x0000000000000006 (SYMTAB)             0x400600
 0x000000000000000a (STRSZ)              1583 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x850000
 0x0000000000000002 (PLTRELSZ)           1200 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x401808
 0x0000000000000007 (RELA)               0x4017c0
 0x0000000000000008 (RELASZ)             72 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x401760
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x401680
 0x0000000000000000 (NULL)   

Meanwhile, on fedora 37, for comparision:

$ readelf -d /usr/bin/toolbox

Dynamic section at offset 0x6febf8 contains 28 entries:
  Marca      Tipo                         Nombre/Valor
 0x0000000000000001 (NEEDED)             Biblioteca compartida: [libc.so.6]
 0x000000000000001d (RUNPATH)            Ruta de ejecución de la biblioteca: [/run/host/usr/lib64]
 0x000000000000000c (INIT)               0x95000
 0x000000000000000d (FINI)               0x3da8ec
 0x0000000000000019 (INIT_ARRAY)         0x48a650
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x48a658
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x448
 0x0000000000000005 (STRTAB)             0x12a0
 0x0000000000000006 (SYMTAB)             0x6b8
 0x000000000000000a (STRSZ)              2014 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x6ffdf8
 0x0000000000000002 (PLTRELSZ)           1272 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x93fb0
 0x0000000000000007 (RELA)               0x1be0
 0x0000000000000008 (RELASZ)             598992 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Opciones: NOW PIE
 0x000000006ffffffe (VERNEED)            0x1b80
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x1a7e
 0x000000006ffffff9 (RELACOUNT)          24952
 0x0000000000000000 (NULL)               0x0

UPDATE: sources for the package are from version 0.0.99.2 previous to the inclusion of the RUNPATH pointing to host libs. Upgrading manually to the version packaged in Lunar (23.04) solves this problem: https://packages.ubuntu.com/lunar/podman-toolbox

$ podman start --attach fedora-33-test
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Migrating to newer Podman"
level=debug msg="Setting up configuration"
level=debug msg="Setting up configuration: file /etc/containers/toolbox.conf not found"
level=debug msg="Setting up configuration: file /root/.config/containers/toolbox.conf not found"
level=debug msg="Resolving container and image names"
level=debug msg="Container: ''"
level=debug msg="Distribution (CLI): ''"
level=debug msg="Image (CLI): ''"
level=debug msg="Release (CLI): ''"
level=debug msg="Resolved container and image names"
level=debug msg="Container: 'fedora-toolbox-33'"
level=debug msg="Image: 'fedora-toolbox:33'"
level=debug msg="Release: '33'"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Preparing to redirect /etc/host.conf to /run/host/etc/host.conf"
level=debug msg="/run/host/etc/host.conf isn't a symbolic link"
level=debug msg="Redirecting /etc/host.conf to /run/host/etc/host.conf"
level=debug msg="Preparing to redirect /etc/hosts to /run/host/etc/hosts"
level=debug msg="/run/host/etc/hosts isn't a symbolic link"
level=debug msg="Redirecting /etc/hosts to /run/host/etc/hosts"
level=debug msg="Preparing to redirect /etc/localtime to /run/host/etc/localtime"
level=debug msg="/run/host/etc/localtime is a symbolic link"
level=debug msg="Redirecting /etc/localtime to /run/host/etc/localtime"
level=debug msg="Resolved /etc/localtime to /usr/share/zoneinfo/Europe/Madrid"
level=debug msg="Preparing to redirect /etc/resolv.conf to /run/host/etc/resolv.conf"
level=debug msg="/run/host/etc/resolv.conf is a symbolic link"
level=debug msg="Redirecting /etc/resolv.conf to /run/host/etc/resolv.conf"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating directory /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating directory /run/systemd/resolve"
level=debug msg="Binding /run/systemd/resolve to /run/host/run/systemd/resolve"
level=debug msg="Creating directory /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating directory /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating directory /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating directory /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating directory /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user test-vm with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/home/test-vm
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=test-vm
level=debug msg="Removing password for user test-vm"
level=debug msg="Removing password for user root"
passwd: Note: deleting a password also unlocks the password.
level=debug msg="Setting KCM as the default Kerberos credential cache"
level=debug msg="Configuring RPM to ignore bind mounts"
level=debug msg="Setting up daily ticker"
level=debug msg="Setting up watches for file system events"
level=debug msg="Finished initializing container"
level=debug msg="Creating runtime directory /run/user/1000/toolbox"
level=debug msg="Creating initialization stamp /run/user/1000/toolbox/container-initialized-33053"
level=debug msg="Listening to file system and ticker events"

^Ctest-vm@testvm:~$ toolbox enter fedora-33-test

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

test-vm@toolbox:~$ 

[HarryMichal]

Thanks @angiglesias for the investigation! So, considering version 0.0.99.3 does solve the issue, we can close this ticket.