caches: buildah bud --layers sucks on --build-arg=VAR=VAL

[praiskup]

Description

Changing the --build-arg option doesn't invalidate --layers caches.

Steps to reproduce the issue:

1. provide dockerfile:

   $ cat Dockerfile
   FROM busybox
   ARG user
   RUN echo $user
   

2. first run (no caches => no hit => correct)

   $ buildah bud --layers --build-arg=user=0 .
   STEP 1: FROM busybox
   Getting image source signatures
   Copying blob 697743189b6d: 738.18 KiB / 738.18 KiB [========================] 0s
   Copying config d8233ab899d4: 1.46 KiB / 1.46 KiB [==========================] 0s
   Writing manifest to image destination
   Storing signatures
   STEP 2: ARG user
   --> 61e60952494142fd9d32723c5eb0dad8adb526ff18f991642d66aed5dc783627
   STEP 3: FROM 61e60952494142fd9d32723c5eb0dad8adb526ff18f991642d66aed5dc783627
   STEP 4: RUN echo $user
   0
   --> 6dfeb0be1b508772f699f32606a04c5f5fdc2909b1684f247f9c2e97de3e33dd
   STEP 5: COMMIT
   

3. repeat the build with the same --build-arg=user=0 (cache hit => correct)

   $ buildah bud --layers --build-arg=user=0 .
   STEP 1: FROM busybox
   STEP 2: ARG user
   --> Using cache 61e60952494142fd9d32723c5eb0dad8adb526ff18f991642d66aed5dc783627
   STEP 3: FROM 61e60952494142fd9d32723c5eb0dad8adb526ff18f991642d66aed5dc783627
   STEP 4: RUN echo $user
   --> Using cache 6dfeb0be1b508772f699f32606a04c5f5fdc2909b1684f247f9c2e97de3e33dd
   STEP 5: COMMIT 
   

4. repeat the build with changed --build=arg=1 (cache hit => incorrect!)

   $ buildah bud --layers --build-arg=user=1 .
   STEP 1: FROM busybox
   STEP 2: ARG user
   --> Using cache 61e60952494142fd9d32723c5eb0dad8adb526ff18f991642d66aed5dc783627
   STEP 3: FROM 61e60952494142fd9d32723c5eb0dad8adb526ff18f991642d66aed5dc783627
   STEP 4: RUN echo $user
   --> Using cache 6dfeb0be1b508772f699f32606a04c5f5fdc2909b1684f247f9c2e97de3e33dd
   STEP 5: COMMIT 
   

Describe the results you received:

The --build-arg option doesn't invalidate the caches.

Describe the results you expected:

Changed --build-arg option invalidates caches.

Output of rpm -q buildah or apt list buildah:

buildah-1.7-10.1.praiskup.dev.git310363c.fc29.x86_64

Output of buildah version:

buildah version 1.7-dev (image-spec 1.0.0, runtime-spec 1.0.0)

Output of podman version if reporting a podman build issue:

podman version 1.2.0-dev

Output of cat /etc/*release:

Fedora 29 x86_64

Output of cat /etc/containers/storage.conf:

[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
size = ""
override_kernel_check = "true"
mountopt = "nodev"
[storage.options.thinpool]
ostree_repo = ""
skip_mount_home = "false"

[praiskup]

Can I help with this? It causes fatal build failures in @sclorg/postgresql-container:

$ git clone https://github.com/sclorg/postgresql-container.git
$ cd postgresql-container
$ git submodule update --init
$ make VERSIONS=9.6 tag # succeeds
...
$ make VERSIONS=9.6 tag # fails

I intentionally created easier reproducer, but it is not a low-prio thing.

[TomSweeneyRedHat]

@praiskup sorry for the delay on this one. It's high on the priority list, but I've had a few others that were prioritized a bit higher. Also @nalind is busy reworking the image layering code and I think his work will at least make this easier to figure out this issue with a small chance that it might fix it. Part of that work is in #1473 and more to come. I hope to make more progress on this one next week. Honestly I looked at my todo list earlier today and saw this one and said to myself, 'really need to get to this".

Again apologies for the tardiness.

[TomSweeneyRedHat]

@praiskup I was hoping that @nalind's latest rounds of PR's was going to fix this as a side effect, but unfortunately it did not, I just tested with upstream. Will dive in more.

[TomSweeneyRedHat]

@praiskup the issue appears to be that the value of the user variable, in this example either 1 or 0, is not being tucked away in the layers as it's probably supposed to be. So when we check our current image against previous ones we get a match as that doesn't differentiate them.

I wasn't able to find a quick fix, but will dig further on Monday.

[umohnani8]

Fixed in #1521