podman save not saving multiple images

[Prodian0013]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug

Description
When executing a podman save with multiple images only one image is saved.

Steps to reproduce the issue:

1. podman save > images.tar docker.io/busybox:1.27.2 docker.io/metallb/controller:v0.3.1 docker.io/metallb/speaker:v0.3.1

2. podman import images.tar

3. podman images

Describe the results you received:
Only the first image is saved and imported.

Describe the results you expected:
All three images should be saved and imported.

The documentation states podman save is the equivalent to docker save. Where as docker save can actually save multiple images at once.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.1.2

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.2
  podman version: 1.1.2
host:
  BuildahVersion: 1.7.1
  Conmon:
    package: podman-1.1.2-2.git0ad9b6b.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: 6e07c13bf86885ba6d71fdbdff90f436e18abe39-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 96034816
  MemTotal: 3973865472
  OCIRuntime:
    package: runc-1.0.0-59.dev.git2abd837.el7.centos.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 2147217408
  SwapTotal: 2147479552
  arch: amd64
  cpus: 2
  hostname: controller.lan
  kernel: 3.10.0-957.5.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 59m 48.22s
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 28
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

@haircommander PTAL

[afbjorklund]

I got bitten by this bug as well. Saved multiple entries, all pointing to the same image...

Guess I will use multiple files as a workaround, but would be good if it could be fixed ?

[rhatdan]

@haircommander Any update on this?

[baude]

@haircommander Any update on this?

[haircommander]

still waiting on containers/image#610 . I don't have time currently to tackle it on c/image side

[afbjorklund]

My workaround was to use one tarball for each image, and a for loop to load them one-by-one.

find ... -type f | xargs -n 1 podman load -i

[rhatdan]

@haircommander any update on this?

[haircommander]

Unfortunately not, I am not sure I have the bandwidth to tackle it both here and in c/image.

[rhatdan]

Ok @vrothberg You want to grab it?

[vrothberg]

Let's move discussion over to containers/image#610. I add it to my backlog but I can't commit to a schedule at the moment.

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

This is still a good issue, not sure when we can get someone to work on it?
@mtrmac @vrothberg PTAL

[YuLimin]

podman version

Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.10.3
OS/Arch:            linux/amd64

podman info --debug

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.3
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.3.0, commit: unknown'
  Distribution:
    distribution: '"rhel"'
    version: "7.7"
  MemFree: 4073623552
  MemTotal: 8350715904
  OCIRuntime:
    package: containerd.io-1.2.10-3.2.el7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 12
  hostname: registry
  kernel: 3.10.0-1062.el7.x86_64
  os: linux
  rootless: false
  uptime: 56m 35.65s
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker save multiple images work fine.

IDS=$(docker images | awk '{if ($1 ~ /^(registry)/) print $3}')
echo $IDS
docker save $IDS -o xxx.tar

podman save multiple images work failed.

IDS=$(podman images | awk '{if ($1 ~ /^(registry)/) print $3}')
echo $IDS
podman save $IDS -o xxx.tar

[vrothberg]

This is still a good issue, not sure when we can get someone to work on it?
@mtrmac @vrothberg PTAL

I fear we have other tasks with higher priority on our tables at the moment. Let's move discussion over to containers/image#610.

[rhatdan]

Sadly no progress.

[marcfiu]

I am trying to champion podman over docker; however, colleagues use "docker save" to air gap images to a docker-archive tarball. This translates into "podman" not being a drop-in replacement "docker", which is suboptimal. At the least, "podman save" should error out indicating "can't do multiple images" rather than silently succeeding with something that is simply not equivalent to "docker save".

[vrothberg]

Thanks for the feedback, @marcfiu! Throwing an error is definitely better.

[vrothberg]

I opened #5659 to address your feedback. Note that I didn't error out but log when more than one argument is passed as I don't want others to regress on the current behavior.

[vrothberg]

@baude, afaiu this issue should get some attention soon, right?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This is being worked on .

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Most of the work is happening in containers/image right now.

[ncapps]

Is this issue still being worked? We would like the ability to load a tar file containing multiple images.

[mheon]

It's been resolved since Podman 2.2.0

[afages]

I'm sorry but this is not resolved (or regression occured in 2.2.1).
Running a RHEL8 with Podman 2.2.1 and should I use "docker save -o", "podman save -o", "docker/podman >" only the first image is included in the .tar file. Whether I use 2 images or more.

[rhatdan]

Could you check it against podman 3.*, which should be in RHEL8.4 which is about to release.

[vrothberg]

It works when specifying the -m flag. We had to remain backwards compatible with older Podmans, so users have to opt-in to saving multiple image archives.

[rhatdan]

Should we add a containers.conf value for this?

[vrothberg]

Should we add a containers.conf value for this?

Very good thought! We actually implemented that already. The default can be changed via multi_image_archive=true in /etc/containers/containers.conf.

[podmod001]

It works when specifying the -m flag. We had to remain backwards compatible with older Podmans, so users have to opt-in to saving multiple image archives.

@vrothberg
I strongly suggest to display some kind of warning or error with docker-archives, when specifying multiple images without using the -m option.

It is very confusing that the command seems runs successfully without -m, but triggers an endless loop of runtime errors, when actually trying to start containers.

I needed to invest an hour to realize, there is an extra option for multiple images. See https://unix.stackexchange.com/questions/703355/how-to-do-offline-installation-of-multiple-images-via-podman-load-save for a reproduction.

Thanks for considering!

[vrothberg]

@podmod001, thanks for reaching out.

It is very confusing that the command seems runs successfully without -m, but triggers an endless loop of runtime errors, when actually trying to start containers.

That sounds like a bug and is definitely not part of the design. Please open a new issue, ideally with a reproducer, if you want to see it fixed.