Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

With Dockerfile ADD .tar.xz reuses cache even when files changed #1906

Closed
shaicoleman opened this issue Oct 11, 2019 · 8 comments
Closed

With Dockerfile ADD .tar.xz reuses cache even when files changed #1906

shaicoleman opened this issue Oct 11, 2019 · 8 comments
Assignees
@TomSweeneyRedHat
Labels
from Podman This issue was either first reported on the Podman issue list or when running 'podman build' locked - please file new issue/PR

Comments

@shaicoleman
Copy link

shaicoleman commented Oct 11, 2019 •
edited
Loading

Description

Dockerfile ADD command reuses cache even when files have changed

Steps to reproduce the issue:

  1. On Ubuntu 18.04, install podman 1.6.1 from PPA (Buildah 1.11.2)
  2. Create the following Dockerfile
FROM docker.io/busybox

ADD test.tar.xz /
  1. Build a test archive, and run podman build
$ date>test; tar cJf - test > test.tar.xz; md5sum test.tar.xz
158a9e025b54fe66713a443504821a8a  test.tar.xz
$ sudo podman build --file Dockerfile -t podman-test .
STEP 1: FROM docker.io/busybox
STEP 2: ADD test.tar.xz /
STEP 3: COMMIT podman-test
0f7320b3df37c68a91455477f27ec042b832eb589a44bc98365e85449c92cff5
  1. Create a new test archive, run podman build again
$ date>test; tar cJf - test > test.tar.xz; md5sum test.tar.xz
cd07d63afd4136c141d49b74f92e1247  test.tar.xz
$ sudo podman build --file Dockerfile -t podman-test .
STEP 1: FROM docker.io/busybox
STEP 2: ADD test.tar.xz /
--> Using cache 2cab1dafa7b9b6bb91760fd2325da32df0d7846e02798739049026d3ba92e2de
STEP 3: COMMIT podman-test
2cab1dafa7b9b6bb91760fd2325da32df0d7846e02798739049026d3ba92e2de

Describe the results you expected:
It is expected not to reuse the cache if the files have changed.

Version:

podman/bionic,now 1.6.1-1~ubuntu18.04~ppa3 amd64 [installed]
Version:            1.6.1
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64
BuildahVersion: 1.11.2
kernel: 5.0.0-31-generic
arch: amd64
runc version 1.0.0-rc8+dev
@TomSweeneyRedHat
Copy link
Member

First reported in containers/podman#3920. See that issue for further background information. We'll track the issue here going forward though.

@TomSweeneyRedHat TomSweeneyRedHat mentioned this issue Oct 11, 2019
Closed
@TomSweeneyRedHat TomSweeneyRedHat self-assigned this Oct 11, 2019
@TomSweeneyRedHat TomSweeneyRedHat added the from Podman This issue was either first reported on the Podman issue list or when running 'podman build' label Oct 11, 2019
@TomSweeneyRedHat
Copy link
Member

TomSweeneyRedHat commented Oct 11, 2019 •
edited
Loading

@shaicoleman I'm replicating now with upstream and latest. As you noted it should not cache. As a work around, you can add the --no-cache option to the second build command, that should work as you expect.

$ sudo podman build --no-cache --file Dockerfile -t podman-test .

In the meantime I'll see if I can figure out why the build process isn't.

@edsantiago
Copy link
Member

FWIW this is a regression from podman 1.5.0 (buildah v1.10.1, according to podman's go.mod). That is: running this reproducer on v1.5.0 will correctly pick up the new tarfile.

Also FWIW I was never able to reproduce containers/podman#3920 in v1.5.0, no matter what Dockerfile incantations I used.

TomSweeneyRedHat added a commit to TomSweeneyRedHat/buildah that referenced this issue Oct 16, 2019
@TomSweeneyRedHat
Re-Add file check to cacheing algorithm
a14ea45 
Add back the file checking mechanisms that were removed as part of the
reworking of the cache in containers#1792.  If we have a 'COPY mytar.tar.xz' command
in the Containerfile and the tar file was regenerated after an intial build,
secondary 'buildah bud' commands would use the cache from the first build
for the tar file instead of regenerating it.

The historyMatches() function (https://github.com/containers/buildah/blob/master/imagebuildah/executor.go#L287)
does not seem to pick up this type of change and I wasn't able to quickly
rework that to do so.  We may want to revert back to this check for now,
then tweak the historyMatches to work properly in a follow up PR.

Regardless, I've added a test as part of this PR to catch this issue
going forward.

Fixes:  containers#1906

Signed-off-by: TomSweeneyRedHat <[email protected]>
@TomSweeneyRedHat TomSweeneyRedHat mentioned this issue Oct 16, 2019
Closed
TomSweeneyRedHat added a commit to TomSweeneyRedHat/buildah that referenced this issue Oct 16, 2019
@TomSweeneyRedHat
Re-Add file check to cacheing algorithm
baf8007 
Add back the file checking mechanisms that were removed as part of the
reworking of the cache in containers#1792.  If we have a 'COPY mytar.tar.xz' command
in the Containerfile and the tar file was regenerated after an intial build,
secondary 'buildah bud' commands would use the cache from the first build
for the tar file instead of regenerating it.

The historyMatches() function (https://github.com/containers/buildah/blob/master/imagebuildah/executor.go#L287)
does not seem to pick up this type of change and I wasn't able to quickly
rework that to do so.  We may want to revert back to this check for now,
then tweak the historyMatches to work properly in a follow up PR.

Regardless, I've added a test as part of this PR to catch this issue
going forward.

Fixes:  containers#1906

Signed-off-by: TomSweeneyRedHat <[email protected]>
TomSweeneyRedHat added a commit to TomSweeneyRedHat/buildah that referenced this issue Oct 16, 2019
@TomSweeneyRedHat
Re-Add file check to cacheing algorithm
2d7aa9d 
Add back the file checking mechanisms that were removed as part of the
reworking of the cache in containers#1792.  If we have a 'COPY mytar.tar.xz' command
in the Containerfile and the tar file was regenerated after an intial build,
secondary 'buildah bud' commands would use the cache from the first build
for the tar file instead of regenerating it.

The historyMatches() function (https://github.com/containers/buildah/blob/master/imagebuildah/executor.go#L287)
does not seem to pick up this type of change and I wasn't able to quickly
rework that to do so.  We may want to revert back to this check for now,
then tweak the historyMatches to work properly in a follow up PR.

Regardless, I've added a test as part of this PR to catch this issue
going forward.

Fixes:  containers#1906

Signed-off-by: TomSweeneyRedHat <[email protected]>
TomSweeneyRedHat added a commit to TomSweeneyRedHat/buildah that referenced this issue Oct 16, 2019
@TomSweeneyRedHat
Re-Add file check to cacheing algorithm
600c658 
Add back the file checking mechanisms that were removed as part of the
reworking of the cache in containers#1792.  If we have a 'COPY mytar.tar.xz' command
in the Containerfile and the tar file was regenerated after an intial build,
secondary 'buildah bud' commands would use the cache from the first build
for the tar file instead of regenerating it.

The historyMatches() function (https://github.com/containers/buildah/blob/master/imagebuildah/executor.go#L287)
does not seem to pick up this type of change and I wasn't able to quickly
rework that to do so.  We may want to revert back to this check for now,
then tweak the historyMatches to work properly in a follow up PR.

Regardless, I've added a test as part of this PR to catch this issue
going forward.

Fixes:  containers#1906

Signed-off-by: TomSweeneyRedHat <[email protected]>
TomSweeneyRedHat added a commit to TomSweeneyRedHat/buildah that referenced this issue Oct 16, 2019
@TomSweeneyRedHat
Re-Add file check to cacheing algorithm
e5a4e47 
Try 2 after I accidentally nuked my sandbox for containers#1925

Add back the file checking mechanisms that were removed as part of the
reworking of the cache in containers#1792.  If we have a 'COPY mytar.tar.xz' command
in the Containerfile and the tar file was regenerated after an intial build,
secondary 'buildah bud' commands would use the cache from the first build
for the tar file instead of regenerating it.

The historyMatches() function (https://github.com/containers/buildah/blob/master/imagebuildah/executor.go#L287)
does not seem to pick up this type of change and I wasn't able to quickly
rework that to do so.  We may want to revert back to this check for now,
then tweak the historyMatches to work properly in a follow up PR.

Regardless, I've added a test as part of this PR to catch this issue
going forward.

FWIW, with the exception of the test code, all of the code is a reversion to pre-existing code.

Fixes:  containers#1906

Signed-off-by: TomSweeneyRedHat <[email protected]>
@TomSweeneyRedHat TomSweeneyRedHat mentioned this issue Oct 16, 2019
Closed
@TomSweeneyRedHat
Copy link
Member

@edsantiago do you have cycles to look at this and test?

nalind pushed a commit to nalind/buildah that referenced this issue Oct 30, 2019
@TomSweeneyRedHat @nalind
Re-Add file check to cacheing algorithm
bb90f81 
Regardless, I've added a test as part of this PR to catch this issue
going forward.

Fixes:  containers#1906

Signed-off-by: TomSweeneyRedHat <[email protected]>
@nalind nalind mentioned this issue Oct 30, 2019
Closed
@slaskawi slaskawi mentioned this issue Nov 25, 2019
Merged
4 tasks
@rhatdan
Copy link
Member

rhatdan commented Aug 4, 2020

I believe this is fixed in current buildah, reopen if I am mistaken

@rhatdan rhatdan closed this as completed Aug 4, 2020
@seperman
Copy link

seperman commented Nov 4, 2021

I suddenly started having this issue with podman build

Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

@rhatdan
Copy link
Member

rhatdan commented Nov 4, 2021

Could you try with podman 3.4

@seperman
Copy link

seperman commented Nov 6, 2021

Ok I'm waiting for 3.4 to appear on https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman
It it takes a few more days, I will go down the pass of building podman locally.
For now I have to delete the previous images when building new images to avoid using caches.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@TomSweeneyRedHat TomSweeneyRedHat

Labels
from Podman This issue was either first reported on the Podman issue list or when running 'podman build' locked - please file new issue/PR
Projects
None yet
Milestone
No milestone
5 participants
@shaicoleman @edsantiago @rhatdan @seperman @TomSweeneyRedHat