Signed image save #6265

maage · 2020-05-17T15:14:33Z

I'm trying to implement setup where every image is signed and every time image is run, I want to be able to move images from one host to another without registry (using podman image save, podman image load and keep signatures intact).

I'm using buildah to create an image and sign it. I use rootless setup with podman / buildah.

But it seems I am not able to save a image as policy denies it even if signature seems to be okay when tested by hand.

It would also help if there was a documentation of all required transports at: docs/containers-policy.json.5.md

If policy has: "default": [{"type": "reject"}]

Then I need to define containers-storage transport to get this deny. Otherwise podman image save ends with:

ERRO[0000] unable to save "localhost/foo:latest": Error copying image to the remote destination: Source image rejected: Running image containers-storage:[vfs@/home/ubuntu/.local/share/containers/storage+/run/user/1000/containers]@0a806fbe41e1e6f7a305b65ab832d3e0c3dbbb5c6bfdbcd10cb5f8c335244a13 is rejected by policy.

So all possible transports should be supported, not just those listed.

I can also run images when only policy is:

{ "default": [ { "type": "reject" } ] }

I would assume it should deny running any image then. You can test this with your own image.

I'm submitting this issue here as it mainly is about setting /etc/containers/policy.json and interpreting it.

$ podman --version
podman version 1.9.1
$ buildah --version
buildah version 1.14.8 (image-spec 1.0.1-dev, runtime-spec 1.0.1-dev)

/etc/containers/policy.json

{
  "default": [
    {
      "type": "reject"
    }
  ],
  "transports": {
    "containers-storage": {
      "": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/DEB-GPG-KEY-local-2020.asc"
        }
      ]
    },
    "docker": {
      "localhost": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/DEB-GPG-KEY-local-2020.asc"
        }
      ]
    }
  }
}

$key is fingerprint of key for of /etc/pki/rpm-gpg/DEB-GPG-KEY-local-2020.asc
And its keyring is at "$HOME"/.debgpg.

$ c="$(buildah from scratch)"
$ (export GNUPGHOME="$HOME"/.debgpg;buildah commit --sign-by "$key" "$c" foo)
$ podman image list foo
REPOSITORY      TAG      IMAGE ID       CREATED          SIZE
localhost/foo   latest   0a806fbe41e1   37 minutes ago   3.73 kB
$ podman image save -o foo foo:latest
Error: unable to save "foo:latest": Error copying image to the remote destination: Source image rejected: Signature for identity localhost/foo:latest is not accepted

If I import key, I can verify signature by hand:

$ gpg --import /etc/pki/rpm-gpg/DEB-GPG-KEY-local-2020.asc 
gpg: key 09FD2BF532E95A05: public key "<hidden>" imported
gpg: Total number processed: 1
gpg:               imported: 1
$ gpg --verify /home/ubuntu/.local/share/containers/storage/vfs-images/0a806fbe41e1*/signatures 
gpg: Signature made Sun May 17 17:17:01 2020 EEST
gpg:                using RSA key A277C6F62361690019E1591409FD2BF532E95A05
gpg: Good signature from "<hidden>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A277 C6F6 2361 6900 19E1  5914 09FD 2BF5 32E9 5A05

Save with debugging

$ podman --log-level=debug image save -o foo localhost/foo:latest
DEBU[0000] Ignoring lipod.conf EventsLogger setting "journald". Use containers.conf if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/ubuntu/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/ubuntu/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/ubuntu/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/ubuntu/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/sbin/runc"               
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] running as rootless                          
DEBU[0000] Ignoring lipod.conf EventsLogger setting "journald". Use containers.conf if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/ubuntu/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/ubuntu/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/ubuntu/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/ubuntu/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "vfs"   
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/sbin/runc"               
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] parsed reference into "[vfs@/home/ubuntu/.local/share/containers/storage+/run/user/1000/containers]localhost/foo:latest" 
DEBU[0000] parsed reference into "[vfs@/home/ubuntu/.local/share/containers/storage+/run/user/1000/containers]@0a806fbe41e1e6f7a305b65ab832d3e0c3dbbb5c6bfdbcd10cb5f8c335244a13" 
DEBU[0000] Using blob info cache at /home/ubuntu/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[vfs@/home/ubuntu/.local/share/containers/storage]@0a806fbe41e1e6f7a305b65ab832d3e0c3dbbb5c6bfdbcd10cb5f8c335244a13 
DEBU[0000]  Using transport "containers-storage" policy section "" 
DEBU[0000] Requirement 0: denied, done                  
ERRO[0000] unable to save "localhost/foo:latest": Error copying image to the remote destination: Source image rejected: Signature for identity localhost/foo:latest is not accepted

The text was updated successfully, but these errors were encountered:

mtrmac · 2020-05-18T16:57:54Z

Thanks for your report.

Signatures don’t quite work the way the process above assumes: what is signed isn’t a “abstract image”, but a specific representation of an image. So, it does not really make sense to sign the image stored locally (as an overlay set of filesystem trees of uncompressed files) and then expect to publish/reuse the image. E.g. every push to a registry will compress the image, changing the representation and invalidate the signatures.

And the “docker archive” format used by default by (podman image save) (besides being notably less efficient than a registry) does not preserve signatures at all. So this is just not going to work that way.

Basically signatures only make sense when used together with registries. (You can run a temporary registry in a container, backed by a volume, and transfer the backing volume to a different computer and run a temporary registry there. That’s going to be more efficient than docker-archive anyway.)

A few specific comments.

I'm trying to implement setup where every image is signed and every time image is run, I want to be able to move images from one host to another without registry (using podman image save, podman image load and keep signatures intact).

Signatures are created on registry manifests; the (docker-archive) format does not have registry manifests (c/image currently creates a fake one for purposes of interoperability with the rest of the code, but even that isn’t a guaranteed behavior of the library), so there is nothing to sign in the docker-archive format and even if it made sense for a locally-built image (which is questionable), and even if it were contained in the docker-archive format (which isn’t the case), no way to ensure that a signature will work once the image is transferred using a docker-archive. It might happen to work for you right now with some specific versions of the code involved but that’s accidental and may break any time.

It would also help if there was a documentation of all required transports at: docs/containers-policy.json.5.md

https://github.com/containers/image/blob/master/docs/containers-transports.5.md ?

If policy has: "default": [{"type": "reject"}]

Then I need to define containers-storage transport to get this deny. Otherwise podman image save ends with:
ERRO[0000] unable to save "localhost/foo:latest": Error copying image to the remote destination: Source image rejected: Running image containers-storage:[vfs@/home/ubuntu/.local/share/containers/storage+/run/user/1000/containers]@0a806fbe41e1e6f7a305b65ab832d3e0c3dbbb5c6bfdbcd10cb5f8c335244a13 is rejected by policy. 

Yeah, that’s arguably a Podman bug / missing feature. Right now you can just set containers-storage: to insecureAcceptAnything, but that would of course interact badly with requiring signatures on every container start.

I can also run images when only policy is:
{ "default": [ { "type": "reject" } ] }
I would assume it should deny running any image then.

That kind of enforcement does not currently exist. (It only really matters if a key is removed from policy.json, or the policy changes after pulling the image in some other way — and it would not be very practical to verify the gigabytes of image data on every container start; it isn’t even possible at all for schema1 images). That’s a long-standing, but not a high-priority, RFE.

$ c="$(buildah from scratch)"
$ (export GNUPGHOME="$HOME"/.debgpg;buildah commit --sign-by "$key" "$c" foo)
$ podman image save -o foo foo:latest
Error: unable to save "foo:latest": Error copying image to the remote destination: Source image rejected: Signature for identity localhost/foo:latest is not accepted

That’s interesting.

$ podman --log-level=debug image save -o foo localhost/foo:latest
…
DEBU[0000] parsed reference into "[vfs@/home/ubuntu/.local/share/containers/storage+/run/user/1000/containers]localhost/foo:latest" 
DEBU[0000] parsed reference into "[vfs@/home/ubuntu/.local/share/containers/storage+/run/user/1000/containers]@0a806fbe41e1e6f7a305b65ab832d3e0c3dbbb5c6bfdbcd10cb5f8c335244a13" 
…
DEBU[0000] IsRunningImageAllowed for image containers-storage:[vfs@/home/ubuntu/.local/share/containers/storage]@0a806fbe41e1e6f7a305b65ab832d3e0c3dbbb5c6bfdbcd10cb5f8c335244a13 
…
ERRO[0000] unable to save "localhost/foo:latest": Error copying image to the remote destination: Source image rejected: Signature for identity localhost/foo:latest is not accepted

Podman completely loses the name:tag@digest when referring to images by the resolved immutable ID. That might eventually be a problem when we want to add a “verify image signature before starting it”.

Moving to Podman, both to hard-code ignoring the containers-storage policy on push/save, and to discuss the loss of name:tag@digest.

github-actions · 2020-07-03T00:10:58Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2020-07-06T13:56:57Z

@vrothberg Could you look at this problem?

vrothberg · 2020-07-06T14:02:32Z

@vrothberg Could you look at this problem?

Sorry, I won't find time for that in the foreseeable future.

rhatdan · 2020-07-06T18:39:32Z

@QiWang19 PTAL

rhatdan · 2020-09-10T20:13:58Z

@QiWang19 Did you ever get a chance to look at this?

lmgray · 2020-10-15T12:07:59Z

I have a related question... I'm running into a similar problem and would like to programatically add an accept clause to /etc/containers/policy.json for containers-storage transport. I don't see a way to add/modify a transport other than "docker" using podman image trust set ... I'm imagining being able to do something like podman image trust set --type=accept --transport=containers-storage ... Is there another method to add/modify /etc/containers/policy.json or does everyone just manually edit the file? I need to do this dynamically in a jenkins pipeline so editing the file is problematic...

basically, want to add the following to policy.json from cli without resorting to some awful sed/jq/... hack:

        "containers-storage": {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }

QiWang19 · 2020-10-15T14:33:33Z

Current podman image trust set -t accept containers-storage only set the scope under the docker transport. It would be possible to have --transport=containers-storage, and if --transport not specified, it defaults to docker transport.

lmgray · 2020-10-15T17:24:10Z

Current podman image trust set -t accept containers-storage only set the scope under the docker transport. It would be possible to have --transport=containers-storage, and if --transport not specified, it defaults to docker transport.

I have (ahem) "a workaround" in my automation right now that I'm not proud of:

            jq '.transports."containers-storage" += .transports."docker-daemon"' /etc/containers/policy.json > policy.json.new
            sudo mv policy.json.new /etc/containers/policy.json

This duplicates the "docker-daemon" transport policy (which is insecureAcceptAnything in my case) to a new "containers-storage" transport policy for better or worse.

I'd certainly like to upvote the idea of adding support for podman container trust set --transport= ... to create/manage other transports since that seems to be the only CLI tool for managing /etc/containers/policy.json.

rhatdan · 2020-12-24T12:13:55Z

@QiWang19 Any new thoughts on this?

github-actions · 2021-01-25T00:49:11Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-01-25T15:05:34Z

@QiWang19 What is the scoop on this?

QiWang19 · 2021-01-25T16:25:00Z

I didn't work on this.

rhatdan · 2021-01-25T17:07:33Z

@ashley-cui Interested in taking a look at this?

github-actions · 2021-02-25T00:16:23Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-02-26T10:59:24Z

@ashley-cui Did you get a chance to look at this?

ashley-cui · 2021-03-02T22:00:37Z

Haven't gotten to look at it, might find time soon

github-actions · 2021-04-02T00:13:35Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-04-05T16:56:02Z

@ashley-cui Reminder...

github-actions · 2021-05-06T00:02:16Z

A friendly reminder that this issue had no activity for 30 days.

lmgray · 2021-07-01T14:01:00Z

Any new on this? The issue came up again for us -- we're still using jq as a hack to accomplish this:

    jq '.transports."containers-storage" += .transports."dir"' policy.json

Would be nice to be able to use podman image trust vs. munging policy.json manually

github-actions · 2021-08-01T00:03:53Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-08-01T11:32:01Z

@vrothberg @mtrmac PTAL

github-actions · 2021-09-01T00:03:53Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-09-01T14:13:20Z

@vrothberg @mtrmac @lsm5 Any thoughts?

github-actions · 2021-10-02T00:06:12Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-10-04T19:16:43Z

Time keeps on ticking, ticking, ticking...
@vrothberg @mtrmac @lsm5 @ashley-cui Any ideas on this?

vrothberg · 2021-10-05T08:45:08Z

Unless priorities change, I won't see this on my pipeline for a long time.

github-actions · 2021-11-24T00:04:20Z

A friendly reminder that this issue had no activity for 30 days.

github-actions · 2022-09-14T00:09:51Z

A friendly reminder that this issue had no activity for 30 days.

mtrmac transferred this issue from containers/image May 18, 2020

mheon added the kind/feature Categorizes issue or PR as related to a new feature. label Jun 2, 2020

github-actions bot added the stale-issue label Jul 3, 2020

rhatdan assigned vrothberg Jul 6, 2020

vrothberg removed their assignment Jul 6, 2020

rhatdan assigned QiWang19 Jul 6, 2020

rhatdan removed the stale-issue label Dec 24, 2020

github-actions bot added the stale-issue label Jan 25, 2021

rhatdan removed the stale-issue label Jan 25, 2021

QiWang19 removed their assignment Jan 25, 2021

github-actions bot added the stale-issue label Feb 25, 2021

rhatdan removed the stale-issue label Feb 26, 2021

github-actions bot added the stale-issue label Apr 2, 2021

rhatdan removed the stale-issue label Apr 5, 2021

github-actions bot added the stale-issue label May 6, 2021

ashley-cui removed the stale-issue label May 11, 2021

github-actions bot added the stale-issue label Aug 1, 2021

rhatdan removed the stale-issue label Aug 1, 2021

github-actions bot added the stale-issue label Sep 1, 2021

rhatdan removed the stale-issue label Sep 1, 2021

github-actions bot added the stale-issue label Oct 2, 2021

rhatdan removed the stale-issue label Oct 4, 2021

github-actions bot added the stale-issue label Nov 24, 2021

vrothberg mentioned this issue Jul 13, 2022

podman save - signature support #11718

Closed

umohnani8 added priority/low and removed stale-issue labels Jul 13, 2022

github-actions bot added the stale-issue label Sep 14, 2022

rhatdan removed the stale-issue label Sep 14, 2022

vrothberg removed the priority/low label Aug 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Signed image save #6265

Signed image save #6265

maage commented May 17, 2020

mtrmac commented May 18, 2020

github-actions bot commented Jul 3, 2020

rhatdan commented Jul 6, 2020

vrothberg commented Jul 6, 2020

rhatdan commented Jul 6, 2020

rhatdan commented Sep 10, 2020

lmgray commented Oct 15, 2020 •

edited

Loading

QiWang19 commented Oct 15, 2020

lmgray commented Oct 15, 2020 •

edited

Loading

rhatdan commented Dec 24, 2020

github-actions bot commented Jan 25, 2021

rhatdan commented Jan 25, 2021

QiWang19 commented Jan 25, 2021

rhatdan commented Jan 25, 2021

github-actions bot commented Feb 25, 2021

rhatdan commented Feb 26, 2021

ashley-cui commented Mar 2, 2021

github-actions bot commented Apr 2, 2021

rhatdan commented Apr 5, 2021

github-actions bot commented May 6, 2021

lmgray commented Jul 1, 2021

github-actions bot commented Aug 1, 2021

rhatdan commented Aug 1, 2021

github-actions bot commented Sep 1, 2021

rhatdan commented Sep 1, 2021

github-actions bot commented Oct 2, 2021

rhatdan commented Oct 4, 2021

vrothberg commented Oct 5, 2021

github-actions bot commented Nov 24, 2021

github-actions bot commented Sep 14, 2022

Signed image save #6265

Signed image save #6265

Comments

maage commented May 17, 2020

mtrmac commented May 18, 2020

github-actions bot commented Jul 3, 2020

rhatdan commented Jul 6, 2020

vrothberg commented Jul 6, 2020

rhatdan commented Jul 6, 2020

rhatdan commented Sep 10, 2020

lmgray commented Oct 15, 2020 • edited Loading

QiWang19 commented Oct 15, 2020

lmgray commented Oct 15, 2020 • edited Loading

rhatdan commented Dec 24, 2020

github-actions bot commented Jan 25, 2021

rhatdan commented Jan 25, 2021

QiWang19 commented Jan 25, 2021

rhatdan commented Jan 25, 2021

github-actions bot commented Feb 25, 2021

rhatdan commented Feb 26, 2021

ashley-cui commented Mar 2, 2021

github-actions bot commented Apr 2, 2021

rhatdan commented Apr 5, 2021

github-actions bot commented May 6, 2021

lmgray commented Jul 1, 2021

github-actions bot commented Aug 1, 2021

rhatdan commented Aug 1, 2021

github-actions bot commented Sep 1, 2021

rhatdan commented Sep 1, 2021

github-actions bot commented Oct 2, 2021

rhatdan commented Oct 4, 2021

vrothberg commented Oct 5, 2021

github-actions bot commented Nov 24, 2021

github-actions bot commented Sep 14, 2022

lmgray commented Oct 15, 2020 •

edited

Loading

lmgray commented Oct 15, 2020 •

edited

Loading