Podman Service is Not Active (When it is)

[xer0-1ne]

After a recent update, I am now getting an error message that the Podman service isn't running. I tried downgrading without any luck. It now seems to be consistent with this error. The issue is that Podman is running.

Error message:

Cockpit status:

Terminal status or cockpit.service and cockpit.socket:

cockpit-bridge --packages

installed versions:

I am running an Arch Linux machine:
banshee ~ # uname -a
Linux banshee 5.8.10-arch1-1 #1 SMP PREEMPT Thu, 17 Sep 2020 18:01:06 +0000 x86_64 GNU/Linux

All other services related to Cockpit seem to be functioning properly.

[marusak]

What version of podman are you running?

[xer0-1ne]

Podman version is 2.1.1-1

[marusak]

Podman version is 2.1.1-1

ok, so my guess was wrong. There was a broken podman version, but this one should be fine. (Possibly the arch version was never patched?)

Do you see anything in your browser logs?

Can you check output of this command? (It should stay listening on events and when you do some action like start a new container it should produce some output and still stay listening)
sudo curl -s --unix-socket /run/podman/podman.sock http://d/v1.12/libpod/events

[xer0-1ne]

Marusak,
I downgraded podman back to 2.0.6 and it worked.
It works for now. I'll wait for the next update and report back.

[ardevd]

I'm having the same issue now on Podman 2.1.1 on Fedora 32. "Podman service is not active" message shows up in Cockpit even though it indeed is active.

[marusak]

I'm having the same issue now on Podman 2.1.1 on Fedora 32.

Can you please provide the following items? Thanks!

• rpm -q podman
• rpm -q cockpit-podman
• systemctl --user status podman.service (run as non-root please)
• sudo systemctl status podman.service
• Screenshot of your cockpit podman page (ideally the whole screen)
• Any logs in browser console

[j2mcv]

I have the same problem with Fedora 33:

• podman-2.1.1-10.fc33.x86_64
• cockpit-podman-24-1.fc33.noarch
• curl -s --unix-socket /run/podman/podman.sock http://d/v1.12/libpod/events
  {"cause":"failed to read message field: bad message","message":"failed to read message field: bad message","response":500}

[marusak]

curl -s --unix-socket /run/podman/podman.sock http://d/v1.12/libpod/events
{"cause":"failed to read message field: bad message","message":"failed to read message field: bad message","response":500}

This would indeed explain why we say it is not active. This is however unrelated to cockpit. Since I cannot reproduce would you please mind opening issue in https://github.com/containers/podman and linking it then back here? Thanks!

[j2mcv]

curl -s --unix-socket /run/podman/podman.sock http://d/v1.12/libpod/events
{"cause":"failed to read message field: bad message","message":"failed to read message field: bad message","response":500}

This would indeed explain why we say it is not active. This is however unrelated to cockpit. Since I cannot reproduce would you please mind opening issue in https://github.com/containers/podman and linking it then back here? Thanks!

No problem but are you sure this endpoint has not been deprecated? looking at the REST API reference it seems to always return an error code 500.

https://docs.podman.io/en/latest/_static/api.html#operation/libpodGetEvents

[marusak]

No problem but are you sure this endpoint has not been deprecated? looking at the REST API reference it seems to always return an error code 500.

I see there on the page:

Responses:
200 returns a string of json data describing an event
500 Internal server error

[j2mcv]

The origin of my problem was two corrupted files in the journal.

I removed them and now cockpit-podman is working.

[mplinuxgeek]

I just wanted to share that corrupted journal entries was also my problem.

podman.service was failing with the below error:

Oct 28 20:49:35 nottherealhostname systemd[5424]: podman.service: Failed with result 'protocol'.
Oct 28 20:49:35 nottherealhostname systemd[5424]: podman.service: Unit process 5626 (podman pause) remains running after unit stopped.
Oct 28 20:49:35 nottherealhostname systemd[5424]: Failed to start Podman API Service.

Running sudo journalctl --verify showed that there were corrupted journal files so the fix was:

sudo journalctl --vacuum-time=7d
sudo journalctl --verify

After the above 2 commands podman.service started without issue.

[marusak]

This does not seems to be cockpit-podman specific. Partly maybe podman specific as it cannot start when journal is corrupted.