[additionalimagestores] Allow prune to operate on additional stores

[srcshelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

A use-case for a configuration with additionalimagestores defined is to separate persistent images which will survive a reboot in one store from ephemeral images which are lost on reboot and would need to be rebuilt in another.

In this situation, it would be helpful if podman image prune and podman system prune gained an additional option to easily operate on specific image stores, or all defined image stores.

e.g.

$ podman image ls
REPOSITORY                              TAG         IMAGE ID       CREATED          SIZE      STORE
localhost/gentoo-build                  latest      c2c952975d5e   53 seconds ago   680 MB    main
<none>                                  <none>      c9dfaa697d1b   2 minutes ago    1.56 GB   main
localhost/service.net-misc.dhcp         4.4.2-r3    7cc352e402c7   2 days ago       59.8 MB   /storage/images
<none>                                  <none>      8be457d547f6   2 days ago       51.1 MB   /storage/images
localhost/service.net-dns.bind          9.16.12     1abeea31ed8d   2 days ago       351 MB    /storage/build
<none>                                  <none>      d88173308c16   2 days ago       102 MB    /storage/build

$ podman image prune -f
Deleted Images
c9dfaa697d1b[…]
Total reclaimed space: 1.56GB

$ podman image ls
REPOSITORY                              TAG         IMAGE ID       CREATED          SIZE      STORE
localhost/gentoo-build                  latest      c2c952975d5e   1 minute ago     680 MB    main
localhost/service.net-misc.dhcp         4.4.2-r3    7cc352e402c7   2 days ago       59.8 MB   /storage/images
<none>                                  <none>      8be457d547f6   2 days ago       51.1 MB   /storage/images
localhost/service.net-dns.bind          9.16.12     1abeea31ed8d   2 days ago       351 MB    /storage/build
<none>                                  <none>      d88173308c16   2 days ago       102 MB    /storage/build

$ podman image prune -f --store=/storage/images
Deleted Images from store /storage/images
8be457d547f6[…]
Total reclaimed space: 51.1MB

$ podman image ls
REPOSITORY                              TAG         IMAGE ID       CREATED          SIZE      STORE
localhost/gentoo-build                  latest      c2c952975d5e   1 minute ago     680 MB    main
localhost/service.net-misc.dhcp         4.4.2-r3    7cc352e402c7   2 days ago       59.8 MB   /storage/images
localhost/service.net-dns.bind          9.16.12     1abeea31ed8d   2 days ago       351 MB    /storage/build
<none>                                  <none>      d88173308c16   2 days ago       102 MB    /storage/build

$ podman image prune -f --store=all
Deleted Images from stores main, /storage/images, /storage/build
d88173308c16[…]
Total reclaimed space: 102MB

(… and there's a question of whether --store=all or --all-stores would be the preferred form of the latter invocation)

This result can currently be achieved by looping over the entries in additionalimagestores and running the prune operation having reset storage-opt and set root… but it would be much more convenient to be given the option to perform this operation in a single command.

For context see #9852, for the suggested change in ls output see #10391.

[rhatdan]

Recursively running on --store would remove images that might be used in the main store.

You could have an image in an additional store "fedora", and then have an image in the primary read/write store based on top of Fedora. If I want and did

podman --root ADDITIONALSTORE image prune
it will remove fedora, since it is not used by any images of containers, correct? Which would then break the primary.

[srcshelton]

This is perhaps beyond my architectural understanding of how the storage layers operate!

So we're saying that if we have an additional location with a image and we build from that into a different store than rather than copying the relevant layers into the destination store on creation, there is instead a link to the data in the original additional location - such that if this original additional storage image is removed, the new image which inherited from it will break?

(And so by extension, in the original use-case for additional imagestores - having read-only images on a network share - if the network or the originating host is own, any local images based on them are also unusable?)

It may be that a (stronger/more explicit?) link is needed back from leaf-node images to their parents in this case, which would have the side-effect of resolving this question as the prune operation would then be aware of the dependancy of primary image.

Alternatively, prune could operate on a best-effort based on what is currently visible to it… although this would be much more effective if the second element of #10393 were implemented - to provide a simple means to promote additionalimagestores, so that using root doesn't simply give a different primary store, but instead promotes one store to primary and keeps the original primary as a additional store. prune would then have the additional visibility to avoid breakage.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I don't think this is something we want to do. I think this is something administrators would need to manage.

[srcshelton]

To confirm my understanding, though, you're saying that prune can never be a safe operation if there's any chance of additionalimagestores being built-off, since images are neither linked to non-primary dependencies nor are the dependencies copied into the primary store on reference?

If this is (broadly) correct then should we consider an enhancement whereby non-primary images referenced by a new image to be written to the primary store are also automatically copied to the primary store, to ensure referential integrity if the additional store is in the future cleared or unavailable?

[rhatdan]

No I don't believe we should. I think we should treat this the same way the kernel treats overlay, if the underlying storage is modified then what happens in the upper is undefined. It would be interesting to see how Podman would react if an dependent image is removed. Would it attempt to pull it again from a registry if it could?