podman import writes the image 3 times

[jfilak]

/kind bug

Description

I have an exported container (48GB). I run podman import ctr.tar myimg and the podman fails because it writes the data 3 times but my disk has only 120GB.

Steps to reproduce the issue:

1. podman import ctr.tar myimg

Describe the results you received:

Getting image source signatures
Copying blob 0687615e2b9e done
Copying config bb8c6cd121 done
Writing manifest to image destination
Storing signatures
Error: Error committing the finished image: error adding layer with blob "sha256:0687615e2b9ea4615907a71d92be5d4266431fb87e06047a39d0babd193b12a7": Error processing tar file(exit status 1): write /sybase/NPL/saplog_1/NPL_log_001.dat: no space left on device

Describe the results you expected:
IMPORTED

Additional information you deem important (e.g. issue happens only occasionally):

[root@buran /]# grep root /etc/containers/storage.conf
runroot = "/containers/runroot"
graphroot = "/containers/graphroot"
[root@buran containers]# findmnt /dev/mapper/containers-data
TARGET      SOURCE                            FSTYPE OPTIONS
/containers /dev/mapper/containers-data       xfs    rw,relatime,attr2,inode64,noquota
/var/tmp    /dev/mapper/containers-data[/tmp] xfs    rw,relatime,attr2,inode64,noquota
[root@buran containers]# du -sh /containers/*
35G     graphroot
8.0K    runroot
89G     tmp
[root@buran tmp]# du -sh /containers/tmp/*
45G     import052406256
45G     storage125424783

Output of podman version:

Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.10.3
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.3
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.3.0, commit: unknown'
  Distribution:
    distribution: '"rhel"'
    version: "7.6"
  MemFree: 7854403584
  MemTotal: 8184705024
  OCIRuntime:
    package: runc-1.0.0-65.rc8.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  hostname: buran.openstack.eu-ru-1.cloud.sap
  kernel: 3.10.0-957.5.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 18h 53m 57.33s (Approximately 0.75 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /containers/graphroot
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /containers/runroot
  VolumePath: /containers/graphroot/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.4.4-4.el7.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

OpenStack VM, VMWare

[rhatdan]

@QiWang19 Could you take a look at this, seems pretty strange.

@jfilak You are running as root correct?

[jfilak]

@rhatdan Yep, root.

BTW: Docker does not need any temporary file for import and Docker needs only 1 temporary file for load.

[rhatdan]

@nalind Any ideas what is going on?

[rhatdan]

@mtrmac ?

[mtrmac]

*shrug*

• Runtime.Import writes it to file (the import temporary file)
• c/image/tarball reads all of it to compute a digest and create a fake manifest [X]
• c/image/storage.PutBlob makes a separate on-disk copy, to be applied in Commit (the storage directory; this might be better after one of the parts of blob-copy detection image#611)

The second copy could be avoidable, with some pretty tricky code, though containers/image#611 is doing a lot and it may not be easy to extricate from the rest.

But really, my first recommendation is to use a real image distribution protocol that is random-access and can avoid pulling redundant data, like the docker/distribution one, or even dir: on a network filesystem, not distributing images by streaming unseekable tarballs.

Even if we fix the second temporary copy, [X] makes it necessary to create a copy anyway, just to compute its digest to see whether the layer already exists locally. That’s always one extra copy just because the transport method does not contain a manifest.

(Or maybe c/storage can handle optimistically creating a layer, on the fly computing its digest, and then deleting it if it turns out to be redundant… note that that is much more expensive than the single large temporary file if the layer ends up being redundant; and it would require Podman import to be implemented directly using c/storage instead of via c/image, which is inherently whole-image = requires a manifest to be created.)

[jfilak]

@mtrmac The thing is that I have the container in Docker on a builder VM and I want do deploy it on a machine I can ssh only through a jumpserver. So I do docker export on the one end and [ssh tunnel] and podman import on the other end.

@mtrmac How can I create the dir: from a docker image stored in docker storage? (I will tar-gzip-ncat the dir through an ssh tunnel and ncat-ungzip-untar it on the target host). What command I have to issue to load/import/pull the dir?

[mtrmac]

@mtrmac The thing is that I have the container in Docker on a builder VM and I want do deploy it on a machine I can ssh only through a jumpserver.

(To make things easy: reconsider the jumpserver, or tunnel a HTTP proxy through that? I suppose that’s not an option.)

So I do docker export on the one end and [ssh tunnel] and podman import on the other end.

• Run local container with docker/distribution, storage mounted from an external volume
• docker push all of the images to that server
• Terminate container
• scp the backing storage through the jump host to the destination
• On the destination, run a local container with docker/distribution, storage mounted from an external volume pointing to the copy
• podman pull all of the images
• Terminate container.

Yes, that’s quite a bit more work… but it uses the optimized/optimizable path, naturally avoids redundant copies of the same layers for multiple images, and it should be possible to automate.

(There’s still one extra copy without containers/image#611, I’m afraid.)

@mtrmac How can I create the dir: from a docker image stored in docker storage?

skopeo copy docker-daemon:$name dir:$path should work. (OTOH, because it uses docker save-like unseekable tar streams internally, it will create large temporary on-disk copies. (IIRC one inside Docker, and one in Skopeo.) Basically, right now, you can choose whether the extra disk usage happens at the source or at the destination.)

(I will tar-gzip-ncat the dir through an ssh tunnel and ncat-ungzip-untar it on the target host). What command I have to issue to load/import/pull the dir?

Hum, podman pull can’t do that it seems. I suppose something like skopeo copy dir:$path containers-storage:$name would be the direct equivalent, but I haven’t tested it now, and it could be much more tricky with rootless. Or skopeo copy dir:$path docker://$local-registry/repo and then pull from there — but at that point the “copy a registry backing volume” approach would be easier.

[mtrmac]

(None of this is to say that we are happy to have the extra copies, and it should be fixed as time/priorities allow; it’s just to be clear that that at least one of them will probably need to remain anyway, and the only way to avoid that one is to avoid the tar stream approach.)

[jfilak]

You are right. I run out of disk space on the build host because skope copy docker-daemon: ... creates an extra temp docker directory in /var/tmp/.

[vrothberg]

@jfilak, does the provided workaround work for you?

[jfilak]

Well, I would have to configure docker and podman on the destination system and while it is technically possible, I didn't bother to mess up with this setup. Fortunately, I realized that I can point container storage to system partition (can hold 1 image), tmp to additional disk space (can hold 2 images) and I can pipe the original image from the source system via ssh - that way I managed to import the image.

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[mheon]

No, bot, this is still a fairly serious bug

[mheon]

@vrothberg So it's closing bugs despite user activity? That seems wrong?

[vrothberg]

@vrothberg So it's closing bugs despite user activity? That seems wrong?

I was surprised as well. Need to check if we can tweak it.

[vrothberg]

Closing as the provided workaround does the trick.