Error doing COPY of empty directory in multi-stage build

[AlanCoding]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Our real Containerfile is in the project here:

https://github.com/ansible/awx-ee/blob/devel/Containerfile

My problem is that I can't build it with the relatively recent podman I have on my computer. Older versions seem to be able to build it. The error occurs in a line COPY --from=galaxy /usr/share/ansible/roles /usr/share/ansible/roles where /usr/share/ansible/roles in the galaxy image either does't exist, or is empty.

Steps to reproduce the issue:

This is a modified version of our Containerfile to demonstrate the issue:

FROM quay.io/ansible/ansible-runner:devel as galaxy

ADD _build/requirements.yml /build/_build/requirements.yml

RUN mkdir -p /usr/share/ansible/roles /usr/share/ansible/collections
RUN echo "bar"
RUN echo "foo" > /usr/share/ansible/collections/file.txt

FROM quay.io/ansible/ansible-runner:devel


RUN mkdir -p /usr/share/ansible/roles /usr/share/ansible/collections
COPY --from=galaxy /usr/share/ansible/roles /usr/share/ansible/roles
COPY --from=galaxy /usr/share/ansible/collections /usr/share/ansible/collections

1. run podman build -f Containerfile . -t quay.io/ansible/awx-ee:latest, passing that Containerfile

Describe the results you received:

$ podman build -f Containerfile . -t quay.io/ansible/awx-ee:latest
STEP 1: FROM quay.io/ansible/ansible-runner:devel AS galaxy
STEP 2: ADD _build/requirements.yml /build/_build/requirements.yml
--> Using cache ec7652b091ac27342183727049f5a5eff4db32b1dc17d2940b2bf8beaf53db33
--> ec7652b091a
STEP 3: RUN mkdir -p /usr/share/ansible/roles /usr/share/ansible/collections
--> Using cache 1e522b93200c9f4748861523eaad9831d695c91ed0bbcf54b3fbe585f0b2e4b8
--> 1e522b93200
STEP 4: RUN echo "bar"
--> Using cache 5ee8fd1c00fe78fab47ab6f63df709f4a3aac0bea3f0d42b2e3941e7eb50bbc7
--> 5ee8fd1c00f
STEP 5: RUN echo "foo" > /usr/share/ansible/collections/file.txt
--> Using cache 43aab374738bcb0ee0c58e7c8cb9682acaf7a487f04e688987b2868caee037b5
--> 43aab374738
STEP 6: FROM quay.io/ansible/ansible-runner:devel
STEP 7: RUN mkdir -p /usr/share/ansible/roles /usr/share/ansible/collections
--> Using cache a5a3bd69a915b1208c3bb4a22b218d7d700321c57a0d4e5b9958bf455b24b7a7
--> a5a3bd69a91
STEP 8: COPY --from=galaxy /usr/share/ansible/roles /usr/share/ansible/roles
Error: error building at STEP "COPY --from=galaxy /usr/share/ansible/roles /usr/share/ansible/roles": error adding sources [/home/alancoding/.local/share/containers/storage/overlay/9eace1a33965254739d6514e26df17b4925c5319680abff1e6c90ff9ba1116ef/merged/usr/share/ansible/roles]: no items matching glob "/home/alancoding/.local/share/containers/storage/overlay/9eace1a33965254739d6514e26df17b4925c5319680abff1e6c90ff9ba1116ef/merged/usr/share/ansible/roles" copied (1 filtered out): no such file or directory

return code:

$ echo $?
125

Describe the results you expected:

Return code of 0.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens deterministically.

I tinkered with my example Containerfile, found that I could replace the line:

RUN echo "bar"

with

RUN echo "bar" > /usr/share/ansible/roles/file.txt

And then it worked without error. In other words, the COPY worked when the directory wasn't empty.

Docker doesn't seem have a problem with my example Containerfile.

Output of podman version:

$ podman version
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 09:37:50 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: alan-red-hat
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.9.16-200.fc33.x86_64
  linkmode: dynamic
  memFree: 1972752384
  memTotal: 33405517824
  ociRuntime:
    name: runc
    package: containerd.io-1.4.3-3.1.fc31.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc92
      commit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
      spec: 1.0.2-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 21064048640
  swapTotal: 21067980800
  uptime: 186h 2m 11.74s (Approximately 7.75 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/alancoding/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/alancoding/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 75
  runRoot: /run/user/1000/containers
  volumePath: /home/alancoding/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 09:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-2.2.1-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

We currently are enforcing that the source direcrtory exists, I wonder if that is what broke?

@nalind PTAL

[nalind]

The source directory does exist, though, but we're treating it as an error when we try to copy from a directory to a directory, but the source directory is empty.

[AlanCoding]

FWIW, we were able to work around it with the kind of obvious fix of:

COPY --from=galaxy /usr/share/ansible /usr/share/ansible

Since there is always a collection subfolder, the /usr/share/ansible is not considered to be empty, and the COPY works.

So that's the workaround - assure that the source folder always has something in it.

But this still breaks pattern with docker, which from my testing accepted an empty source folder. Using:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        831ebea
 Built:             Tue Dec 15 04:35:53 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:33:10 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[nalind]

But this still breaks pattern with docker, which from my testing accepted an empty source folder. Using:

Oh, yes, it's a bug, sorry if I wasn't clear on that.

[rhatdan]

This is now fixed in the main branch, although it is causing us some other errors that we are working through.

[TomSweeneyRedHat]

Reopening as #3002 will revert the change completely and I need to dig further.

[rhatdan]

I think this conflicts with what Docker does. That is why everything is blowing up.

[TomSweeneyRedHat]

Looks like Docker does NOT blow up with this Dockerfile, which leads me to believe that something might be off either with the dockerignore test, or perhaps the way we process dockerignore files?

# docker build -t tom -f ~/Dockerfile.galaxy .
Sending build context to Docker daemon  13.82kB
Step 1/8 : FROM alpine as galaxy
latest: Pulling from library/alpine
4c0d98bf9879: Pull complete 
Digest: sha256:08d6ca16c60fe7490c03d10dc339d9fd8ea67c6466dea8d558526b1330a85930
Status: Downloaded newer image for alpine:latest
 ---> e50c909a8df2
Step 2/8 : RUN mkdir -p /usr/share/ansible/roles /usr/share/ansible/collections
 ---> Running in f07bae9df092
Removing intermediate container f07bae9df092
 ---> 1644586ab9bf
Step 3/8 : RUN echo "bar"
 ---> Running in 07f2982ea5c6
bar
Removing intermediate container 07f2982ea5c6
 ---> 843904b2d3e6
Step 4/8 : RUN echo "foo" > /usr/share/ansible/collections/file.txt
 ---> Running in 3a1bf5357e7f
Removing intermediate container 3a1bf5357e7f
 ---> d6554aec3950
Step 5/8 : FROM galaxy
 ---> d6554aec3950
Step 6/8 : RUN mkdir -p /usr/share/ansible/roles /usr/share/ansible/collections
 ---> Running in 530e52beaab9
Removing intermediate container 530e52beaab9
 ---> bc89239181bf
Step 7/8 : COPY --from=galaxy /usr/share/ansible/roles /usr/share/ansible/roles
 ---> 3f3dec850470
Step 8/8 : COPY --from=galaxy /usr/share/ansible/collections /usr/share/ansible/collections
 ---> d5d2da6a3134
Successfully built d5d2da6a3134
Successfully tagged tom:latest

[TomSweeneyRedHat]

I made a note in the PR, but thought I'd tie up here. The new processing for an emptydir was not checking the .dockerignore file before making the directory. I've now fixed that in #2978