local repository is locked during podman push

[avikivity]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

During a podman push operation, other operations (like podman run) are blocked.

Steps to reproduce the issue:

1. podman push some huge image

2. while this is chugging away, run podman run, even on the same image

3. wait for the push to complete

Describe the results you received:

push and run operations are serialized

Describe the results you expected:

podman run can proceed while the push is pushing

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.1.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 12
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: avi
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.15-301.fc33.x86_64
  linkmode: dynamic
  memFree: 9043312640
  memTotal: 33541640192
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-4.dev.giteecccdb.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 12773634048
  swapTotal: 16869486592
  uptime: 224h 47m 10.04s (Approximately 9.33 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/avi/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 0
    stopped: 8
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/avi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 110
  runRoot: /run/user/1000/containers
  volumePath: /home/avi/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Wed Oct  7 19:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

(truncated - in fact even that waits for the push)

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-12.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical, with a non-stellar connection (100 Mb/s upload).

[rhatdan]

This is a well known bug that @saschagrunert and @vrothberg have been working to fix.

[avikivity]

If I may complain a little more about large image handling, I notice that podman is quite slow when compressing large images. It would be nice to compress in parallel, and use a native library.

[rhatdan]

PRs would be appreciated. :^)

[avikivity]

Only an endless stream of complaints, unfortunately

[vrothberg]

If I may complain a little more about large image handling, I notice that podman is quite slow when compressing large images. It would be nice to compress in parallel, and use a native library.

Thanks for the feedback. We have that on our radar as well. Unfortunately, time is scarce. Layers are pulled and (de)compressed in parallel already but they are first stored in a tmp dir and are then committed to the storage in sequence. The sequential committing may be what's slowing your large image down.

[avikivity]

I have one giant layer, and a few tiny layers, so inter-layer parallelism does not help.

[vrothberg]

Is Docker noticeable faster?

I am asking since even single layers are (de)compressed in parallel. I don't think that's a bottleneck but I want to follow your suspicion and make sure.

[avikivity]

I can no longer run docker since I'm on Fedora 33 with v2 cgroups. I notice that podman rarely takes more than 140% cpu, so even if there is some parallelism, it's not nearly enough to exploit even a desktop.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@vrothberg did you ever finish your diagnosis?

[vrothberg]

@vrothberg did you ever finish your diagnosis?

Unfortunately not. Without having a baseline, it's hard to judge what's fast and what's slow.

@avikivity could you share the image? Is it accessible on Docker Hub or Quay or some other public registry?

[avikivity]

https://hub.docker.com/repository/docker/scylladb/scylla-toolchain tag fedora-33-20201214

[vrothberg]

Thanks!

I can reproduce. Took me a while to put the pieces back together, but this is blocked on containers/storage#473. Not only run is blocked, but also listing images and other commands accessing the image store.

@rhatdan @nalind @saschagrunert, I think we need to set us a time line to get the work in containers/storage#473 done.

[saschagrunert]

@rhatdan @nalind @saschagrunert, I think we need to set us a time line to get the work in containers/storage#473 done.

Yes, there was some review done recently and I can pick up the work next week. 👍

[rhatdan]

I would love to get that one in, as well as @giuseppe patches. Now that 3.0 is out, we should be able to get this merged.

[vrothberg]

I am consolidating issues. This is effectively a duplicate of #5356 so I am closing this one here.