'Error: failed to initialize container fedora-toolbox-39', 'Error: container fedora-toolbox-38 not found'

[comods]

Describe the bug
Fedora 39 container was working on my Fedora 38 silverblue OS but now it is not.

Steps how to reproduce the behaviour

[localhost ~]$ toolbox enter fedora-toolbox-39
Error: failed to initialize container fedora-toolbox-39

[localhost ~]$ toolbox run fedora-toolbox-39 ls
Error: container fedora-toolbox-38 not found
Use the 'create' command to create a toolbox.
Run 'toolbox --help' for usage.

https://registry.fedoraproject.org/repo/fedora-toolbox/tags/ shows 39 and 39-1 as options.

Screenshots

[localhost ~]$ toolbox list
IMAGE NAME                                                          CREATED
registry.fedoraproject.org/fedora-toolbox:39  3 weeks ago

CONTAINER NAME     CREATED        STATUS  IMAGE NAME
fedora-toolbox-39      6 days ago     exited    registry.fedoraproject.org/fedora-toolbox:39

[localhost ~]$ toolbox enter fedora-toolbox-39 -v
DEBU Running as real user ID *
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user user
DEBU Validating sub-ID file /etc/subuid
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /home/user/.config/toolbox 
DEBU Current Podman version is 4.4.1              
DEBU Creating runtime directory /run/user/*/toolbox 
DEBU Old Podman version is 4.4.1                  
DEBU Migration not needed: Podman version 4.4.1 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /home/user/.config/containers/toolbox.conf not found 
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Resolved container name                      
DEBU Container: 'fedora-toolbox-38'               
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Resolving container name                     
DEBU Container: 'fedora-toolbox-39'               
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Resolved container name                      
DEBU Container: 'fedora-toolbox-39'               
DEBU Checking if container fedora-toolbox-39 exists 
DEBU Inspecting mounts of container fedora-toolbox-39 
DEBU Starting container fedora-toolbox-39         
DEBU Inspecting entry point of container fedora-toolbox-39 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-39 is toolbox (PID=5641) 
DEBU Waiting for container fedora-toolbox-39 to finish initializing
DEBU Creating runtime directory /run/user/*/toolbox 
DEBU Checking if initialization stamp /run/user/*/toolbox/container-initialized-5641 exists 
Error: failed to initialize container fedora-toolbox-39

Output of toolbox --version (v0.0.90+)
toolbox version 0.0.99.3

Toolbox package info (rpm -q toolbox)
toolbox-0.0.99.3-10.fc38.x86_64

Output of podman version

Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.20.1
Built:        Fri Feb 17 03:26:16 2023
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)
podman-4.4.1-3.fc38.x86_64

Info about your OS
Fedora Silverblue 38

Additional context
My Fedora 35 container still works. Since toolbox upgrade doesn't exist I want to use the newest image possible to postpone when I will have to create yet another container.

[comods]

$XDG_RUNTIME_DIR is setup correctly.

Rebooting did not fix.

Editing ~/.config/containers/containers.conf to the following did not work:

[engine]
cgroup_manager = "cgroupfs"

[debarshiray]

[localhost ~]$ toolbox run fedora-toolbox-39 ls
Error: container fedora-toolbox-38 not found
Use the 'create' command to create a toolbox.
Run 'toolbox --help' for usage.

Note that this syntax is wrong. For run, the name of the container must be specified using the --container option:

$ toolbox run --container fedora-toolbox-39 ls

[debarshiray]

For the actual bug report, what's the output from:

$ podman start --attach fedora-toolbox-39
...
...

[comods]

Thanks @debarshiray, here is output:

$ podman start --attach fedora-toolbox-39
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Migrating to newer Podman"
level=debug msg="Setting up configuration"
level=debug msg="Setting up configuration: file /etc/containers/toolbox.conf not found"
level=debug msg="Setting up configuration: file /root/.config/containers/toolbox.conf not found"
level=debug msg="Resolving container and image names"
level=debug msg="Container: ''"
level=debug msg="Distribution (CLI): ''"
level=debug msg="Image (CLI): ''"
level=debug msg="Release (CLI): ''"
level=debug msg="Resolved container and image names"
level=debug msg="Container: 'fedora-toolbox-39'"
level=debug msg="Image: 'fedora-toolbox:39'"
level=debug msg="Release: '39'"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/Pacific/Rarotonga"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating directory /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating directory /run/systemd/resolve"
level=debug msg="Binding /run/systemd/resolve to /run/host/run/systemd/resolve"
level=debug msg="Creating directory /run/systemd/sessions"
level=debug msg="Binding /run/systemd/sessions to /run/host/run/systemd/sessions"
level=debug msg="Creating directory /run/systemd/system"
level=debug msg="Binding /run/systemd/system to /run/host/run/systemd/system"
level=debug msg="Creating directory /run/systemd/users"
level=debug msg="Binding /run/systemd/users to /run/host/run/systemd/users"
level=debug msg="Creating directory /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating directory /run/udev/tags"
level=debug msg="Binding /run/udev/tags to /run/host/run/udev/tags"
level=debug msg="Creating directory /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating directory /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating directory /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating directory /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating directory /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating directory /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /home to /var/home"
level=debug msg="/var/home isn't a symbolic link"
level=debug msg="Redirecting /home to /var/home"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user michael with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/home/michael
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=michael
level=debug msg="Removing password for user michael"
passwd: libuser initialization error: could not open configuration file `/etc/login.defs': No such file or directory
Error: failed to remove password for user michael: failed to invoke passwd(1)
$

[comods]

$ toolbox run --container fedora-toolbox-39 ls
Error: failed to initialize container fedora-toolbox-39
$ 

[heyakyra]

Same for me:

$ toolbox run --container dnf ls
Error: failed to initialize container dnf
$ podman start --attach dnf
WARN[0000] Failed to decode the keys ["storage.options.ostree_repo"] from "/var/home/myself/.config/containers/storage.conf" 
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Migrating to newer Podman"
level=debug msg="Setting up configuration"
level=debug msg="Setting up configuration: file /etc/containers/toolbox.conf not found"
level=debug msg="Setting up configuration: file /root/.config/containers/toolbox.conf not found"
level=debug msg="Resolving container and image names"
level=debug msg="Container: ''"
level=debug msg="Distribution (CLI): ''"
level=debug msg="Image (CLI): ''"
level=debug msg="Release (CLI): ''"
level=debug msg="Resolved container and image names"
level=debug msg="Container: 'fedora-toolbox-38'"
level=debug msg="Image: 'fedora-toolbox:38'"
level=debug msg="Release: '38'"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/America/Chicago"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating directory /run/libvirt"
level=debug msg="Binding /run/libvirt to /run/host/run/libvirt"
level=debug msg="Creating directory /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating directory /run/systemd/resolve"
level=debug msg="Binding /run/systemd/resolve to /run/host/run/systemd/resolve"
level=debug msg="Creating directory /run/systemd/sessions"
level=debug msg="Binding /run/systemd/sessions to /run/host/run/systemd/sessions"
level=debug msg="Creating directory /run/systemd/system"
level=debug msg="Binding /run/systemd/system to /run/host/run/systemd/system"
level=debug msg="Creating directory /run/systemd/users"
level=debug msg="Binding /run/systemd/users to /run/host/run/systemd/users"
level=debug msg="Creating directory /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating directory /run/udev/tags"
level=debug msg="Binding /run/udev/tags to /run/host/run/udev/tags"
level=debug msg="Creating directory /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating directory /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating directory /var/lib/libvirt"
level=debug msg="Binding /var/lib/libvirt to /run/host/var/lib/libvirt"
level=debug msg="Creating directory /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating directory /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating directory /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating directory /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Preparing to redirect /home to /var/home"
level=debug msg="/var/home isn't a symbolic link"
level=debug msg="Redirecting /home to /var/home"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user myself with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/var/home/myself
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=myself
level=debug msg="Removing password for user myself"
passwd: libuser initialization error: could not open configuration file `/etc/login.defs': No such file or directory
Error: failed to remove password for user myself: failed to invoke passwd(1)

Issue i've been hitting since #687 (comment)

[m666m]

For my Fedora 38, every time the os upgrades the kernel, the toolbox enter f38 cannot run and can only be rebuilt, so I naming it F381, F382... How can I solve this problem?

$ toolbox enter f382 -vvv
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Looking for sub-GID and sub-UID ranges for user m666m
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /home/m666m/.config/toolbox 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called version.PersistentPreRunE(podman --log-level debug version --format json) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/m666m/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/m666m/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/m666m/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/m666m/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called version.PersistentPostRunE(podman --log-level debug version --format json) 
DEBU[0000] Shutting down engines                        
DEBU Current Podman version is 4.6.0              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 4.6.0                  
DEBU Migration not needed: Podman version 4.6.0 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /home/m666m/.config/containers/toolbox.conf not found 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-38'               
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Resolving container and image names          
DEBU Container: 'f382'                            
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'f382'                            
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Checking if container f382 exists            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exists.PersistentPreRunE(podman --log-level debug container exists f382) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/m666m/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/m666m/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/m666m/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/m666m/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called exists.PersistentPostRunE(podman --log-level debug container exists f382) 
DEBU[0000] Shutting down engines                        
DEBU Inspecting mounts of container f382          
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container f382) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/m666m/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/m666m/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/m666m/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/m666m/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Looking up image "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" in local containers storage 
DEBU[0000] Trying "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" ... 
DEBU[0000] parsed reference into "[overlay@/home/m666m/.local/share/containers/storage+/run/user/1000/containers]@997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" 
DEBU[0000] Found image "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" as "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" in local containers storage 
DEBU[0000] Found image "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" as "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" in local containers storage ([overlay@/home/m666m/.local/share/containers/storage+/run/user/1000/containers]@997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f) 
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container f382) 
DEBU[0000] Shutting down engines                        
DEBU Starting container f382                      
Error: failed to start container f382