Map /var/lib/systemd/coredump/ into the toolbox

[thmo]

In order to be able to debug something that has segfaulted in a toolbox, access to the core file is needed.

# crash
$ perl -e 'unpack p,1x8'
Segmentation fault (core dumped)

# coredumpctl sees it, but cannot load it
$ coredumpctl dump
[...]
 Command Line: perl -e unpack p,1x8
    Executable: /usr/bin/perl
[...]
       Storage: /var/lib/systemd/coredump/core.perl.1000.1db7fdb8f1944cfabeca888d0594f389.65777.1598546178000000000000.lz4 (inaccessible)
[...]
File "/var/lib/systemd/coredump/core.perl.1000.1db7fdb8f1944cfabeca888d0594f389.65777.1598546178000000000000.lz4" is not readable: No such file or directory

as /var/lib/systemd/coredump/ is empty inside the toolbox container.

[thmo]

Hm, ok, seems the mapping is already there, as part of a more general /var → /run/host/var/ mapping.
Now, how to tell coredumpctl that it should also search in /run/host/var/lib/systemd/coredump/?

[debarshiray]

From the outset, this does look a sane idea.

Do you want to submit a pull request? Look at initContainerMounts in src/cmd/initContainer.go.

[eiffel-fl]

Hi.

I hope you are fine and the same for your relatives.

I would like to work on this issue.

Best regards.

[eiffel-fl]

I managed to add /var/lib/systemd/coredump in createArgs in create.go(I did not understand how to correctly use init-container).
The directory is correctly mounted but coredumpctl does not find any journal and systemctl complains:

System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

Its complaint is false because it is PID 1 on my machine (and though inside toolbox).
This problem seems related to #176 and PR #177.

[debarshiray]

I hope you are fine and the same for your relatives.

Thank you for asking. Yes, I am good. :)

I managed to add /var/lib/systemd/coredump in createArgs in create.go
(I did not understand how to correctly use init-container).

Cool.

However, bind mounting a location using podman create ... has the problem that it won't take effect for existing containers. It will only work for newly created ones. Therefore, we try to bind mount the location as part of the toolbox init-container command which is run as the container's entry point. You will see that inside the container, the /run/host location already has parts of the host's file system. So we take the subset that we are interested in from there, and bind mount it at the right location.

Look at initContainerMounts in src/cmd/initContainer.go. It has the list of such bind mounts. We need to add another entry to that list for /var/lib/systemd/coredump.

[eiffel-fl]

Thank you for asking. Yes, I am good. :)

You are welcome :)

I dived more into the code and I understood that toolbox calls podman to create a toolbox container whom entrypoint is toolbox init-container.
So I will try to add my modification in initContainer.go.

Do you have some information about using systemd related command into toolbox?

[debarshiray]

It's a known problem that we can't use systemctl inside Toolbox containers. However, coredumpctl should work. I did this inside one of my containers, and it was enough to get it going:

$ sudo mount --rbind /run/host/var/lib/systemd/coredump /var/lib/systemd/coredump

[eiffel-fl]

Hum...
On my side, even with your command coredumpctl still complains that there are no journal.
I will investigate it a bit more!

[debarshiray]

Really? Could you paste the full error? We do make the systemd journal available inside the container from the host, but you can only access entries written by your UID inside the container.

[eiffel-fl]

Sorry for my late answer!
The error message is quite simple:

me@toolbox:~/Codes/toolbox$ coredumpctl
No journal files were found.
No coredumps found.

While the coredump files are accessible through a toolbox mount:

me@toolbox:~/Codes/toolbox$ ls /run/host/var/lib/systemd/coredump/
core.processus.X.Y.Z.W.lz4

I have the same problem with journalctl though:

me@toolbox:~/Codes/toolbox$ journalctl
No journal files were found.
-- No entries --

[debarshiray]

me@toolbox:~/Codes/toolbox$ coredumpctl
No journal files were found.
No coredumps found.

While the coredump files are accessible through a toolbox mount:

me@toolbox:~/Codes/toolbox$ ls /run/host/var/lib/systemd/coredump/
core.processus.X.Y.Z.W.lz4

Just having /run/host/var/lib/systemd/coredump isn't enough. It needs to be available at /var/lib/systemd/coredump inside the container. This is what the mount --rbind ... above does. Then you should be able to see things in coredumpctl.

I have the same problem with journalctl though:

me@toolbox:~/Codes/toolbox$ journalctl
No journal files were found.
-- No entries --

Do you have anything inside /var/log/journal inside the container? It's supposed to work since commit ee82b94

[eiffel-fl]

My comment was not clear but I tried after mounting with rbind:

me@toolbox:~/Codes/toolbox$ sudo mount --rbind /run/host/var/lib/systemd/coredump /var/lib/systemd/coredump
me@toolbox:~/Codes/toolbox$ coredumpctl
No journal files were found.
No coredumps found.

Do you have anything inside /var/log/journal inside the container? It's supposed to work since commit ee82b94

Good catch! I have nothing inside this directory... I should have checked before writing.

[debarshiray]

In that case, I think /var/log/journal being empty is the actual cause of all your problems. Is this an old container? Or something you created recently?

[eiffel-fl]

I just created it with ./toolbox create.
Do you know how I can fill it quickly so I can check if the problem comes from here?

And, just to be sure I understood correctly, var/log/journal being empty is also the root cause of coredumpctl complaining?

[debarshiray]

Do you know how I can fill it quickly so I can check
if the problem comes from here?

Maybe first check if /var/log/journal is empty on your host or not. If you are running a full graphical session, then usually there's something.

And, just to be sure I understood correctly, /var/log/journal being empty
is also the root cause of coredumpctl complaining?

Yes, I think so.

[eiffel-fl]

Maybe first check if /var/log/journal is empty on your host or not. If you are running a full graphical session, then usually there's something.

/var/log/journnal is empty on my machine.
I will try to fill it so I can try to use correctly journalctl and coredumpctl.
I keep you updated when I have some news about it.

[debarshiray]

Maybe try something like this to write something to your systemd journal:

$ logger "syslog: hello world"

[eiffel-fl]

I wanted to test your command, sorry for the delay I was a bit busy, but when I try to mount --rbind I am asked for a password when using sudo.
Entering my usual password fails and I did not remember when I had to give a password when I worked on toolbox 2 weeks ago.
Can it be related to #523 (my podman version is 2.1.1)?

About this command $ logger "syslog: hello world", on debian it seems that /var/log/journal is empty but the syslog is in /var/log/syslog.
So how can I cleanly move it inside the toolbox so I can test journalctl?

[debarshiray]

when I try to mount --rbind I am asked for a password when using sudo.
Entering my usual password fails and I did not remember when I had to
give a password when I worked on toolbox 2 weeks ago.

You mean, it's asking for a sudo password once enter the container with toolbox enter? That shouldn't happen.

Can it be related to #523 (my podman version is 2.1.1)?

This should have been fixed. You need to update your toolbox package and restart the container.

What's your toolbox version?

About this command $ logger "syslog: hello world", on debian it seems that /var/log/journal
is empty but the syslog is in /var/log/syslog.
So how can I cleanly move it inside the toolbox so I can test journalctl?

You can modify the mount --rbind ... command that I mentioned before to bind mount /run/host/var/log/syslog at /var/log/syslog inside the container.

[eiffel-fl]

You mean, it's asking for a sudo password once enter the container with toolbox enter? That shouldn't happen.

No, I am asked for a sudo password when I run a sudo command like this:

me@toolbox:~/Codes/toolbox$ sudo mount --rbind /run/host/var/lib/

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for me:

Can it be related to #523 (my podman version is 2.1.1)?

This should have been fixed. You need to update your toolbox package and restart the container.

What's your toolbox version?

I am building it from the source, so this is "master".

You can modify the mount --rbind ... command that I mentioned before to bind mount /run/host/var/log/syslog at /var/log/syslog inside the container.

Thank you for the answer but I should have think before asking this...
About the syslog, on debian syslog is a file and not a directory (I do not know how it is on fedora-like distribution).

[debarshiray]

No, I am asked for a sudo password when I run a sudo command like this

And that's inside a toolbox container, right?

It sounds like systemd-journald isn't enabled on your system. What does this show:

$ ps aux | grep systemd-journald

You will probably find it easier to use a Fedora host (a virtual machine is fine) to contribute to Toolbox.

[eiffel-fl]

No, I am asked for a sudo password when I run a sudo command like this

And that's inside a toolbox container, right?

Yes!

It sounds like systemd-journald isn't enabled on your system. What does this show:

$ ps aux | grep systemd-journald

The above command gives me back:

root       336  0.2  0.1  29720  9400 ?        Ss   10:36   0:00 /lib/systemd/systemd-journald
me   2762  0.0  0.0   6224  2520 pts/1    S+   10:38   0:00 grep --color=auto systemd-journald

which means that systemd-journald seems to work on my system.

You will probably find it easier to use a Fedora host (a virtual machine is fine) to contribute to Toolbox.

You are right, I will setup a Fedora VM and I update you here when I have some news!
Thank you for your help!