auto-update no PODMAN_SYSTEMD_UNIT label found #6793

q-b · 2020-06-26T18:57:08Z

Hi,

While playing with podman last version on CentOS 8, I was unable to use the auto-update feature.

# podman auto-update
Trying to pull ...
<redacted>
... signatures
Error: 1 error occurred:
	* error auto-updating container "<redacted>": no PODMAN_SYSTEMD_UNIT label found

https://github.com/containers/libpod/blob/0968f25988624699f2db559f0a984cf84c8a284f/pkg/autoupdate/autoupdate.go#L184

The service file looks good (I checked it on disk and daemon-reloaded tu be sure):

...
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
...

# podman inspect ...
...
            "Labels": {
                "io.containers.autoupdate": "image",
...
            "CreateCommand": [
...
                "--label",
                "io.containers.autoupdate=image",
...

# podman version
Version:      2.0.1
API Version:  1
Go Version:   go1.13.4
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Let me know if you need more input to troubleshoot this issue!

The text was updated successfully, but these errors were encountered:

mheon · 2020-06-26T19:38:25Z

@vrothberg PTAL

vrothberg · 2020-06-29T07:37:43Z

Thanks for opening the issue!

Indeed, we regressed on it. Absolute time to extend the systemd tests...

vrothberg · 2020-06-29T07:39:23Z

And as it looks like we regressed with 2.0.1. 2.0.0 is working...

vrothberg · 2020-06-29T07:58:31Z

Master is fine though.

vrothberg · 2020-06-29T08:46:58Z

I now fail to reproduce again. Very curious.

I used the following script to reproduce:

./bin/podman rm -af
CID=`./bin/podman create --label "io.containers.autoupdate=image" -d busybox:latest top`
./bin/podman generate systemd --new $CID > ~/.config/systemd/user/container-test.service
systemctl --user daemon-reload
systemctl --user stop container-test && systemctl --user start container-test && ./bin/podman inspect --format "{{.Config.Labels}}" -l
./bin/podman auto-update
systemctl --user stop container-test

vrothberg · 2020-06-29T08:47:49Z

@edsantiago could you have a quick look and try to reproduce with 2.0.1?

vrothberg · 2020-06-29T09:02:39Z

Opened #6810 to add a test that would catch that case.

edsantiago · 2020-06-29T12:46:35Z

I can't find any way to reproduce; am trying root on f32.

Run `podman auto-update` in the systemd system tests. Note that this is a first step to at least exercise parts of `auto-update` in the CI. The service won't get updated just yet as we need to set up a local registry, and push a new image. I do not have enough time at the moment to do that but consider this change already as an improvement. We are experiencing some issues in containers#6793 w.r.t. to auto-updates but couldn't track down the root cause yet. Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg · 2020-06-29T15:09:25Z

I now fail to reproduce again. Very curious.

I used the following script to reproduce:

./bin/podman rm -af
CID=`./bin/podman create --label "io.containers.autoupdate=image" -d busybox:latest top`
./bin/podman generate systemd --new $CID > ~/.config/systemd/user/container-test.service
systemctl --user daemon-reload
systemctl --user stop container-test && systemctl --user start container-test && ./bin/podman inspect --format "{{.Config.Labels}}" -l
./bin/podman auto-update
systemctl --user stop container-test

@q-b can you try this reproducer?

edsantiago · 2020-06-29T16:19:10Z

Giving up. For posterity, here is what I tried:

podman run instead of podman create
setting up a local registry, adding it as insecure in registries.conf, using localhost:5000/foobar:latest as source image.
The above, plus pushing/tagging various images as foobar:latest to confirm that podman auto-update would re-pull.
In all of those, many iterations of test loops intended to catch flakes

vrothberg · 2020-07-01T11:28:17Z

Thanks a lot for checking, @edsantiago!

q-b · 2020-07-03T12:38:36Z

@vrothberg

As myself:

[qb@p1-dc1-node-1 ~]$ podman rm -af
[qb@p1-dc1-node-1 ~]$ CID=`podman create --label "io.containers.autoupdate=image" -d busybox:latest top`
Trying to pull registry.fedoraproject.org/busybox:latest...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/busybox:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/busybox:latest...
  manifest unknown: manifest unknown
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 91f30d776fb2 done
Copying config c7c37e472d done
Writing manifest to image destination
Storing signatures
[qb@p1-dc1-node-1 ~]$ mkdir -p ~/.config/systemd/user/
[qb@p1-dc1-node-1 ~]$ podman generate systemd --new $CID > ~/.config/systemd/user/container-test.service
[qb@p1-dc1-node-1 ~]$ systemctl --user daemon-reload
[qb@p1-dc1-node-1 ~]$ systemctl --user stop container-test && systemctl --user start container-test && podman inspect --format "{{.Config.Labels}}" -l
map[PODMAN_SYSTEMD_UNIT:container-test.service io.containers.autoupdate:image]
[qb@p1-dc1-node-1 ~]$ podman auto-update
[qb@p1-dc1-node-1 ~]$ systemctl --user stop container-test

I need to try as root next.

q-b · 2020-07-03T12:56:34Z

[root@p1-dc1-node-1 ~]# podman rm -af
[root@p1-dc1-node-1 ~]# CID=`podman create --label "io.containers.autoupdate=image" -d busybox:latest top`
[root@p1-dc1-node-1 ~]# podman generate systemd --new $CID > /usr/lib/systemd/system/container-test.service
[root@p1-dc1-node-1 ~]# restorecon -RFv /usr/lib/systemd/system/container-test.service
Relabeled /usr/lib/systemd/system/container-test.service from unconfined_u:object_r:systemd_unit_file_t:s0 to system_u:object_r:systemd_unit_file_t:s0
[root@p1-dc1-node-1 ~]# systemctl daemon-reload
[root@p1-dc1-node-1 ~]# systemctl stop container-test && systemctl start container-test && podman inspect --format "{{.Config.Labels}}" -l
map[PODMAN_SYSTEMD_UNIT:container-test.service io.containers.autoupdate:image]
[root@p1-dc1-node-1 ~]# podman auto-update
[root@p1-dc1-node-1 ~]# systemctl stop container-test.service

q-b · 2020-07-03T13:03:56Z

Reproducible:

[root@p1-dc1-node-1 ~]# podman container create \
>   --name ctn-caddy \
>   --label io.containers.autoupdate=image \
>   --volume /etc/caddy:/etc/caddy:Z \
>   --volume /var/lib/caddy:/data:Z \
>   --volume /var/www:/usr/share/caddy:Z \
>   --network host \
>   --publish 80 \
>   --publish 443 \
>   docker.io/library/caddy:2
ab698f14f3a6291b4e71985eba7477aa85a55fed8337a7b2240f35cb6ee3282b
[root@p1-dc1-node-1 ~]# podman generate systemd \
>   --files \
>   --name \
>   --container-prefix srv \
>   --pod-prefix srv \
>   ctn-caddy
/root/srv-ctn-caddy.service
[root@p1-dc1-node-1 ~]# mv -i srv-*.service /usr/lib/systemd/system/
[root@p1-dc1-node-1 ~]# restorecon -FRv /usr/lib/systemd/system
Relabeled /usr/lib/systemd/system/srv-ctn-caddy.service from unconfined_u:object_r:admin_home_t:s0 to system_u:object_r:systemd_unit_file_t:s0
[root@p1-dc1-node-1 ~]# systemctl daemon-reload
[root@p1-dc1-node-1 ~]# systemctl enable --now srv-ctn-caddy.service
Created symlink /etc/systemd/system/multi-user.target.wants/srv-ctn-caddy.service → /usr/lib/systemd/system/srv-ctn-caddy.service.
Created symlink /etc/systemd/system/default.target.wants/srv-ctn-caddy.service → /usr/lib/systemd/system/srv-ctn-caddy.service.
[root@p1-dc1-node-1 ~]# podman inspect --format "{{.Config.Labels}}" -l
map[io.containers.autoupdate:image org.opencontainers.image.description:a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go org.opencontainers.image.documentation:https://caddyserver.com/docs org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.source:https://github.com/caddyserver/caddy-docker org.opencontainers.image.title:Caddy org.opencontainers.image.url:https://caddyserver.com org.opencontainers.image.vendor:Light Code Labs org.opencontainers.image.version:v2.0.0]
[root@p1-dc1-node-1 ~]# podman auto-update
Trying to pull docker://docker.io/library/caddy:2...
Getting image source signatures
Copying blob 639b35073727 done
Copying blob df20fa9351a1 done
Copying blob b1e49a2f7ba4 done
Copying blob 2ed237faf7b9 done
Copying blob 57d6f0d7c084 done
Copying config 90a12b5c95 done
Writing manifest to image destination
Storing signatures
Error: 1 error occurred:
	* error auto-updating container "ab698f14f3a6291b4e71985eba7477aa85a55fed8337a7b2240f35cb6ee3282b": no PODMAN_SYSTEMD_UNIT label found

q-b · 2020-07-04T09:01:13Z

Ok, so it seems related to the --newflag. The description does not mention auto-update, and the podman-auto-update man page does not explicitly state its requirement either.

If diffed both unit files:

[root@p1-dc1-node-1 ~]# diff without-new with-new
3c3
< # Sat Jul  4 10:52:24 CEST 2020
---
> # Sat Jul  4 10:48:30 CEST 2020
14,17c14,18
< ExecStart=/usr/bin/podman start ctn-caddy
< ExecStop=/usr/bin/podman stop -t 10 ctn-caddy
< ExecStopPost=/usr/bin/podman stop -t 10 ctn-caddy
< PIDFile=/run/srv-ctn-caddy.pid
---
> ExecStartPre=/bin/rm -f %t/srv-ctn-caddy.pid %t/srv-ctn-caddy.ctr-id
> ExecStart=/usr/bin/podman run --conmon-pidfile %t/srv-ctn-caddy.pid --cidfile %t/srv-ctn-caddy.ctr-id --cgroups=no-conmon -d --replace --name ctn-caddy --label io.containers.autoupdate=image --volume /etc/caddy:/etc/caddy:Z --volume /var/lib/caddy:/data:Z --volume /var/www:/usr/share/caddy:Z --network host --publish 80 --publish 443 docker.io/library/caddy:2
> ExecStop=/usr/bin/podman stop --ignore --cidfile %t/srv-ctn-caddy.ctr-id -t 10
> ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/srv-ctn-caddy.ctr-id
> PIDFile=%t/srv-ctn-caddy.pid

I checked the labels with --new as well and it looks OK:

[root@p1-dc1-node-1 ~]# podman inspect --format "{{.Config.Labels}}" -l
map[PODMAN_SYSTEMD_UNIT:srv-ctn-caddy.service io.containers.autoupdate:image ...]

vrothberg · 2020-07-06T09:05:18Z

Thanks a lot for checking! Using unit files that create new containers (as done with --new) is a requirement as we need to create and start new containers in order to use the updated image.

I'll open a PR to improve the docs and make this requirement explicit.

Clarify in the help message and the man page that auto updates only work with systemd units that are similar to the ones from `generate systemd --new`. Units that merely start/stop a container do not work as they will use the same image. Fixes: containers#6793 Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg · 2020-07-06T09:17:00Z

Opened #6858

Clarify in the help message and the man page that auto updates only work with systemd units that are similar to the ones from `generate systemd --new`. Units that merely start/stop a container do not work as they will use the same image. Fixes: containers#6793 Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg self-assigned this Jun 29, 2020

vrothberg added the bug label Jun 29, 2020

vrothberg mentioned this issue Jun 29, 2020

systemd system test: run auto-update #6810

Merged

vrothberg mentioned this issue Jul 6, 2020

auto-update: clarify systemd-unit requirements #6858

Merged

openshift-merge-robot closed this as completed in #6858 Jul 6, 2020

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

auto-update no PODMAN_SYSTEMD_UNIT label found #6793

auto-update no PODMAN_SYSTEMD_UNIT label found #6793

q-b commented Jun 26, 2020

mheon commented Jun 26, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

edsantiago commented Jun 29, 2020

vrothberg commented Jun 29, 2020

edsantiago commented Jun 29, 2020

vrothberg commented Jul 1, 2020

q-b commented Jul 3, 2020

q-b commented Jul 3, 2020

q-b commented Jul 3, 2020

q-b commented Jul 4, 2020

vrothberg commented Jul 6, 2020

vrothberg commented Jul 6, 2020

auto-update no PODMAN_SYSTEMD_UNIT label found #6793

auto-update no PODMAN_SYSTEMD_UNIT label found #6793

Comments

q-b commented Jun 26, 2020

mheon commented Jun 26, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

vrothberg commented Jun 29, 2020

edsantiago commented Jun 29, 2020

vrothberg commented Jun 29, 2020

edsantiago commented Jun 29, 2020

vrothberg commented Jul 1, 2020

q-b commented Jul 3, 2020

q-b commented Jul 3, 2020

q-b commented Jul 3, 2020

q-b commented Jul 4, 2020

vrothberg commented Jul 6, 2020

vrothberg commented Jul 6, 2020