podman build COPY reuses cache even when files have changed

[shaicoleman]

/kind bug

Description

Similar to issue #2544 , might be related to #3921

Steps to reproduce the issue:

After running the initial build

In the Dockerfile I have a COPY line, e.g.
COPY --chown=myuser:myuser config myapp/config/

I run the build command:
sudo podman build --rm --build-arg RAILS_ENV=development --build-arg NODE_ENV=development --file rails/Dockerfile -t myimage ..

After changing the files, and rerunning the build, it reuses the cache, even though it shouldn't.

Additional information you deem important (e.g. issue happens only occasionally):
When this happens I have to delete /var/lib/containers to fix the issue.

Output of podman version:

Version:            1.5.0
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:
debug:
compiler: gc
git commit: ""
go version: go1.10.4
podman version: 1.5.0
host:
BuildahVersion: 1.10.1
Conmon:
package: 'conmon: /usr/libexec/podman/conmon'
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.0, commit: unknown'
Distribution:
distribution: linuxmint
version: "19.2"
MemFree: 5022597120
MemTotal: 16622612480
OCIRuntime:
package: 'containerd.io: /usr/bin/runc'
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc8
commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
spec: 1.0.1-dev
SwapFree: 16986927104
SwapTotal: 16986927104
arch: amd64
cpus: 12
eventlogger: journald
hostname: shai-5591
kernel: 5.0.0-25-generic
os: linux
rootless: false
uptime: 21m 22.48s
registries:
blocked: null
insecure: null
search: null
store:
ConfigFile: /etc/containers/storage.conf
ContainerStore:
number: 4
GraphDriverName: overlay
GraphOptions: null
GraphRoot: /var/lib/containers/storage
GraphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
ImageStore:
number: 35
RunRoot: /var/run/containers/storage
VolumePath: /var/lib/containers/storage/volumes

**Package info:**

podman/bionic,now 1.5.0-2~ubuntu18.04~ppa5 amd64 [installed]

[mheon]

@TomSweeneyRedHat PTAL

[shaicoleman]

Happens also with 1.5.1

[E100Beta]

Same problem using multi-stage build and using COPY . . in second stage - I change files, but cache still used. Works fine in single-stage builds.

[rhatdan]

Probably best if these issues were opened on github.com:containers/buildah then on podman, since podman is just vendoring in buildah code.

[TomSweeneyRedHat]

I'm going to leave this open for now, but believe it's a duplicate of containers/buildah#1780

[TomSweeneyRedHat]

@shaicoleman could you please verify my test methodology on containers/buildah#1780 and let me know if that's off from yours?

@E100Beta you mentioned single-stage builds were OK for you, but multi-stage ones were not. Do you have an example of a bad multi-stage build that you could share?

[shaicoleman]

@TomSweeneyRedHat, I used the podman build command rather than calling buildah bud. podman v1.5.0 and v1.5.1 both include buildah 1.10.1 . are you sure containers/buildah#1792 is included in that version? That might fix it

[E100Beta]

@TomSweeneyRedHat will Dockerfile be enough?
I use this to build my Django app:

FROM python:3.7-alpine AS builder
COPY ./requirements/ /requirements/
RUN set -ex \
	&& apk add --no-cache \
		build-base \
		git \
		pcre-dev \
		zlib-dev libjpeg-turbo-dev postgresql-dev \
	&& python3 -m venv /venv \
	&& /venv/bin/pip install -U pip \
	&& /venv/bin/pip install -r /requirements/production.txt
###########################
FROM python:3.7-alpine
RUN apk add --no-cache zlib libjpeg-turbo postgresql-libs
COPY --from=builder /venv /venv
WORKDIR /usr/src/app
COPY . .
EXPOSE 8000
RUN DATABASE_URL=postgres://none /venv/bin/python manage.py collectstatic --noinput
ENTRYPOINT ["/docker-entrypoint.sh"]

[TomSweeneyRedHat]

@shaicoleman I can confirm that containers/buildah#1792 was not part of v1.10.1. That landed in v1.11.0

@E100Beta Dockerfiles always wor the best for me when trying to track down build issues. I'll try yours above and/or refactor mine to get an "AS" clause in play. There may still be an issue in that scenario.

Thanks all!

[TomSweeneyRedHat]

I've not been able to duplicate this using Buildah v1.11.0 which should be in the next release of Podman. I'm not sure if that will be Podman 1.5.1 or Podman 1.6.0, but it will be contained that next version. I'm going to close this now as containers/buildah#1792 appears to have fixed the issue. If you find otherwise, please feel free to reopen this or create a new issue as appropriate.

[shaicoleman]

Still happening to me with podman 1.6.1 (comes with buildah 1.11.2) on Ubuntu, with the following command
ADD --chown=myuser:mygroup archive.tar.xz /home/myuser

Are archives also checksummed?

[rhatdan]

@TomSweeneyRedHat Could you check this out?

[TomSweeneyRedHat]

@rhatdan ack, added on to the list to check into today/tomorrow.

[TomSweeneyRedHat]

Issue reopened in Buildah at: containers/buildah#1906