Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman build appears to incorrectly reuse layers #2544

Closed
miabbott opened this issue Mar 5, 2019 · 15 comments
Closed

podman build appears to incorrectly reuse layers #2544

miabbott opened this issue Mar 5, 2019 · 15 comments
Assignees
@TomSweeneyRedHat @umohnani8
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@miabbott
Copy link
Contributor

miabbott commented Mar 5, 2019 •
edited
Loading

/kind bug

We have a tests for Red Hat Enterprise Linux CoreOS (RHCOS), where we create small container images on the fly in order to test certain pieces of functionality. We essentially use a simple shell script that looks like this:

$ cat build.sh 
#!/usr/bin/env bash
set -xeou pipefail

tmpdir=$(mktemp -d)
cd $tmpdir
echo -e "FROM scratch\nCOPY . /" > Dockerfile
b=$(which $@)
libs=$(sudo ldd $b | grep -o /lib'[^ ]*' | sort -u)
sudo rsync -av --relative --copy-links $b $libs ./
sudo podman build -t localhost/$1 .

When we run it in succession, the second container image that is created appears to reuse the layer from the first container image that corresponds to the COPY command:

$ rpm -q podman
podman-1.0.1-2.git921f98f.el7.x86_64

$ ./build.sh echo                                                                                                                                                                                                                                                       
++ mktemp -d
+ tmpdir=/tmp/tmp.YFDAjZd54Z
+ cd /tmp/tmp.YFDAjZd54Z
+ echo -e 'FROM scratch\nCOPY . /'
++ which echo
+ b=/bin/echo
++ sort -u
++ grep -o '/lib[^ ]*'
++ sudo ldd /bin/echo
+ libs='/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6'
+ sudo rsync -av --relative --copy-links /bin/echo /lib64/ld-linux-x86-64.so.2 /lib64/libc.so.6 ./
sending incremental file list
/bin/
/bin/echo
/lib64/
/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6

sent 2,349,029 bytes  received 85 bytes  4,698,228.00 bytes/sec
total size is 2,348,208  speedup is 1.00
+ sudo podman build -t localhost/echo .
STEP 1: FROM scratch
STEP 2: COPY . /
--> db03e87b7a7e0c10a4ceadda81576d93e8c63b6b5969fb11b6792fefda911940
STEP 3: COMMIT localhost/echo
--> 3d483e1d778ea9113c327e6dd8c8041bdf982e8a4167f6a10d3cbbd0440dd89d

$ sudo podman images
REPOSITORY       TAG      IMAGE ID       CREATED          SIZE
localhost/echo   latest   3d483e1d778e   20 seconds ago   2.35 MB
<none>           <none>   db03e87b7a7e   20 seconds ago   2.35 MB

$ sudo podman run --rm localhost/echo echo "hello"
hello

$ ./build.sh ping sh
++ mktemp -d
+ tmpdir=/tmp/tmp.ifC08LBEkj
+ cd /tmp/tmp.ifC08LBEkj
+ echo -e 'FROM scratch\nCOPY . /'
++ which ping sh
+ b='/bin/ping
/bin/sh'
++ sort -u
++ grep -o '/lib[^ ]*'
++ sudo ldd /bin/ping /bin/sh
+ libs='/lib64/ld-linux-x86-64.so.2
/lib64/libattr.so.1
/lib64/libcap.so.2
/lib64/libcrypto.so.10
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libidn.so.11
/lib64/libm.so.6
/lib64/libresolv.so.2
/lib64/libtinfo.so.5
/lib64/libz.so.1'
+ sudo rsync -av --relative --copy-links /bin/ping /bin/sh /lib64/ld-linux-x86-64.so.2 /lib64/libattr.so.1 /lib64/libcap.so.2 /lib64/libcrypto.so.10 /lib64/libc.so.6 /lib64/libdl.so.2 /lib64/libidn.so.11 /lib64/libm.so.6 /lib64/libresolv.so.2 /lib64/libtinfo.so.5 /lib64/libz.so.1 ./
sending incremental file list
/bin/
/bin/ping
/bin/sh
/lib64/
/lib64/ld-linux-x86-64.so.2
/lib64/libattr.so.1
/lib64/libc.so.6
/lib64/libcap.so.2
/lib64/libcrypto.so.10
/lib64/libdl.so.2
/lib64/libidn.so.11
/lib64/libm.so.6
/lib64/libresolv.so.2
/lib64/libtinfo.so.5
/lib64/libz.so.1

sent 7,637,301 bytes  received 275 bytes  15,275,152.00 bytes/sec
total size is 7,634,664  speedup is 1.00
+ sudo podman build -t localhost/ping .
STEP 1: FROM scratch
STEP 2: COPY . /
--> Using cache db03e87b7a7e0c10a4ceadda81576d93e8c63b6b5969fb11b6792fefda911940
STEP 3: COMMIT localhost/ping
--> 9f01f1a72a1b51e18b5b9071088db0cdb0d479465b9e8b356617b676e227cf6b

$ sudo podman images
REPOSITORY       TAG      IMAGE ID       CREATED              SIZE
localhost/ping   latest   9f01f1a72a1b   12 seconds ago       1.71 kB
localhost/echo   latest   3d483e1d778e   About a minute ago   2.35 MB
<none>           <none>   db03e87b7a7e   About a minute ago   2.35 MB

$ sudo podman run --rm localhost/ping sh -c 'ping -c 1 1.1.1.1'
container create failed: container_linux.go:336: starting container process caused "exec: \"sh\": executable file not found in $PATH"
: internal libpod error

As shown above, the second image (localhost/ping):

  • reuses the cached layer
  • is much smaller than expected
  • does not work properly.

Now, if I were to use the same script and replace podman with docker, I would get usable images. This was done on a Fedora AH host:

$ rpm -q docker
docker-1.13.1-65.git1185cfd.fc29.x86_64

$ cat build.sh
#!/usr/bin/env bash  
set -xeou pipefail  
                        
tmpdir=$(mktemp -d)
cd $tmpdir                                                                                                                                                                                                                                                                                echo -e "FROM scratch\nCOPY . /" > Dockerfile
b=$(which $@)                
libs=$(sudo ldd $b | grep -o /lib'[^ ]*' | sort -u)
sudo rsync -av --relative --copy-links $b $libs ./
sudo docker build -t localhost/$1 .

$ rpm -q docker
docker-1.13.1-65.git1185cfd.fc29.x86_64

$ ./build.sh echo
++ mktemp -d       
+ tmpdir=/tmp/tmp.0gfi9Crt8m
+ cd /tmp/tmp.0gfi9Crt8m
+ echo -e 'FROM scratch\nCOPY . /'
++ which echo       
+ b=/usr/bin/echo       
++ sort -u      
++ grep -o '/lib[^ ]*'
++ sudo ldd /usr/bin/echo
+ libs='/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6'
+ sudo rsync -av --relative --copy-links /usr/bin/echo /lib64/ld-linux-x86-64.so.2 /lib64/libc.so.6 ./
sending incremental file list                                     
/lib64/                                  
/lib64/ld-linux-x86-64.so.2            
/lib64/libc.so.6                               
/usr/                  
/usr/bin/
/usr/bin/echo      
                  
sent 3,062,186 bytes  received 89 bytes  6,124,550.00 bytes/sec
total size is 3,061,160  speedup is 1.00
+ sudo docker build -t localhost/echo .             
Sending build context to Docker daemon 3.067 MB                                     
Step 1/2 : FROM scratch                                                                
 --->                                                                                  
Step 2/2 : COPY . /                                                                            
 ---> d965034406ee                          
Removing intermediate container 9e8705b68ad6         
Successfully built d965034406ee
 
$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
localhost/echo      latest              d965034406ee        8 seconds ago       3.06 MB

$ sudo docker run --rm localhost/echo echo "hello"
hello

$ ./build.sh ping sh
++ mktemp -d
+ tmpdir=/tmp/tmp.pNaoBRaHnM
+ cd /tmp/tmp.pNaoBRaHnM
+ echo -e 'FROM scratch\nCOPY . /'
++ which ping sh
+ b='/usr/bin/ping
/usr/bin/sh'
++ sort -u
++ grep -o '/lib[^ ]*'
++ sudo ldd /usr/bin/ping /usr/bin/sh
+ libs='/lib64/ld-linux-x86-64.so.2
/lib64/libcap.so.2
/lib64/libcrypto.so.1.1
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libidn2.so.0
/lib64/libm.so.6
/lib64/libpthread.so.0
/lib64/libresolv.so.2
/lib64/libtinfo.so.6
/lib64/libunistring.so.2
/lib64/libz.so.1'
+ sudo rsync -av --relative --copy-links /usr/bin/ping /usr/bin/sh /lib64/ld-linux-x86-64.so.2 /lib64/libcap.so.2 /lib64/libcrypto.so.1.1 /lib64/libc.so.6 /lib64/libdl.so.2 /lib64/libidn2.so.0 /lib64/libm.so.6 /lib64/libpthread.so.0 /lib64/libresolv.so.2 /lib64/libtinfo.so.6 /lib64
/libunistring.so.2 /lib64/libz.so.1 ./
sending incremental file list
/lib64/
/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6
/lib64/libcap.so.2
/lib64/libcrypto.so.1.1
/lib64/libdl.so.2
/lib64/libidn2.so.0
/lib64/libm.so.6
/lib64/libpthread.so.0
/lib64/libresolv.so.2
/lib64/libtinfo.so.6
/lib64/libunistring.so.2
/lib64/libz.so.1
/usr/
/usr/bin/
/usr/bin/ping
/usr/bin/sh

sent 12,195,305 bytes  received 298 bytes  24,391,206.00 bytes/sec
total size is 12,191,464  speedup is 1.00
+ sudo docker build -t localhost/ping .
Sending build context to Docker daemon 12.21 MB
Step 1/2 : FROM scratch
 --->
Step 2/2 : COPY . /
 ---> e8caaffebfb5
Removing intermediate container 4c68363c7012
Successfully built e8caaffebfb5

$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
localhost/ping      latest              e8caaffebfb5        3 seconds ago       12.2 MB
localhost/echo      latest              d965034406ee        34 seconds ago      3.06 MB

$ sudo docker run --rm localhost/ping sh -c 'ping -c 1 1.1.1.1'
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=51 time=11.7 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.653/11.653/11.653/0.000 ms

This looks like a break in compatibility with docker. (Also, if I repeat this test with podman 0.12.1.2, it behaves the same as docker, so it also looks like this is a regression)

Now, if I use podman build --layers=false, I can avoid the reuse problem that I am seeing. However, it feels like I should not have to specify this extra parameter to get my images functional.
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 5, 2019
@rhatdan
Copy link
Member

rhatdan commented Mar 5, 2019

@umohnani8 @TomSweeneyRedHat Any ideas?

@TomSweeneyRedHat
Copy link
Member

No thoughts off the top of my head. I'll try to give it a run or three to see if I can spot anything. @umohnani8 any thoughts from you?

@umohnani8
Copy link
Member

I think the cache check to see if what is being copied in has changed or not isn't working as expected. It is seeing that there is no change hence using the cached layer. I can look into it.

@rhatdan
Copy link
Member

rhatdan commented Apr 13, 2019

@TomSweeneyRedHat @umohnani8 @nalind @miabbott Is this still broken?

@TomSweeneyRedHat
Copy link
Member

Yep, just tested, still broken. @nalind's latest layer changes didn't get this one fixed unfortunately.

@TomSweeneyRedHat
Copy link
Member

May be related to containers/buildah#1391

@danstiner danstiner mentioned this issue Apr 14, 2019
Closed
@baude
Copy link
Member

baude commented May 29, 2019

@TomSweeneyRedHat @miabbott is this still an issue?

@rhatdan
Copy link
Member

rhatdan commented May 29, 2019

I think this is fixed, Reopen if this is still an issue.

@rhatdan rhatdan closed this as completed May 29, 2019
@miabbott
Copy link
Contributor Author

@baude yeah, still looks busted to me

$ rpm -q buildah podman 
buildah-1.8.2-1.gite23314b.fc30.x86_64
podman-1.2.0-2.git3bd528e.fc30.x86_64

$ cat libpod2544.sh 
#!/usr/bin/env bash
set -xeou pipefail

tmpdir=$(mktemp -d)
cd $tmpdir
echo -e "FROM scratch\nCOPY . /" > Dockerfile
b=$(which $@)
libs=$(sudo ldd $b | grep -o /lib'[^ ]*' | sort -u)
sudo rsync -av --relative --copy-links $b $libs ./
sudo podman build -t localhost/$1 .

$ ./libpod2544.sh echo
++ mktemp -d
+ tmpdir=/tmp/tmp.4M4NCkW7Py
+ cd /tmp/tmp.4M4NCkW7Py
+ echo -e 'FROM scratch\nCOPY . /'
++ which echo
+ b=/usr/bin/echo
++ sudo ldd /usr/bin/echo
++ grep -o '/lib[^ ]*'
++ sort -u
[sudo] password for miabbott: 
+ libs='/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6'
+ sudo rsync -av --relative --copy-links /usr/bin/echo /lib64/ld-linux-x86-64.so.2 /lib64/libc.so.6 ./
sending incremental file list
/lib64/
/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6
/usr/
/usr/bin/
/usr/bin/echo

sent 6,023,538 bytes  received 89 bytes  12,047,254.00 bytes/sec
total size is 6,021,792  speedup is 1.00
+ sudo podman build -t localhost/echo .
STEP 1: FROM scratch
STEP 2: COPY . /
--> 3ff9c4e0abc71f7774491b9d34de59e8da3592a102b643ed1ff0f9c75ac65a85
STEP 3: COMMIT localhost/echo
--> 3ff9c4e0abc71f7774491b9d34de59e8da3592a102b643ed1ff0f9c75ac65a85

 sudo podman images
REPOSITORY                                                                       TAG         IMAGE ID       CREATED          SIZE
localhost/echo                                                                   latest      3ff9c4e0abc7   18 seconds ago   6.03 MB
...

$ sudo podman run --rm localhost/echo echo "hello"                                
hello        

$ ./libpod2544.sh ping sh                                                         
++ mktemp -d             
+ tmpdir=/tmp/tmp.ZlWyueTWLP
+ cd /tmp/tmp.ZlWyueTWLP
+ echo -e 'FROM scratch\nCOPY . /'
++ which ping sh                   
+ b='/usr/bin/ping
/usr/bin/sh'                                                                                              
++ sudo ldd /usr/bin/ping /usr/bin/sh
++ grep -o '/lib[^ ]*'
++ sort -u                 
+ libs='/lib64/ld-linux-x86-64.so.2
/lib64/libcap.so.2
/lib64/libcrypto.so.1.1
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libidn2.so.0                                                                                       
/lib64/libm.so.6                        
/lib64/libpthread.so.0                 
/lib64/libresolv.so.2
/lib64/libtinfo.so.6
/lib64/libunistring.so.2                                                                                  
/lib64/libz.so.1'            
+ sudo rsync -av --relative --copy-links /usr/bin/ping /usr/bin/sh /lib64/ld-linux-x86-64.so.2 /lib64/libcap.so.2 /lib64/libcrypto.so.1.1 /lib64/libc.so.6 /lib64/libdl.so.2 /lib64/libidn2.so.0 /lib64/libm.so.6 /li
b64/libpthread.so.0 /lib64/libresolv.so.2 /lib64/libtinfo.so.6 /lib64/libunistring.so.2 /lib64/libz.so.1 ./
sending incremental file list                                                                                                                                                                                        
/lib64/                                                                                                                                                                                                              
/lib64/ld-linux-x86-64.so.2                                                                                                                                                                                          
/lib64/libc.so.6                                                                                                                                                                                                     
/lib64/libcap.so.2                                                                                                                                                                                                   
/lib64/libcrypto.so.1.1                                                                                                                                                                                              
/lib64/libdl.so.2                                                                                                                                                                                                    
/lib64/libidn2.so.0                                                                                                                                                                                                  
/lib64/libm.so.6                                                                                                                                                                                                     
/lib64/libpthread.so.0                                                                                                                                                                                               
/lib64/libresolv.so.2                                                                                                                                                                                                
/lib64/libtinfo.so.6                                                                                                                                                                                                 
/lib64/libunistring.so.2                                                                                                                                                                                             
/lib64/libz.so.1                                                                                                                                                                                                     
/usr/                                                                                                                                                                                                                
/usr/bin/
/usr/bin/ping
/usr/bin/sh

sent 19,922,817 bytes  received 298 bytes  39,846,230.00 bytes/sec
total size is 19,917,096  speedup is 1.00
+ sudo podman build -t localhost/ping .
STEP 1: FROM scratch
STEP 2: COPY . /
--> Using cache 3ff9c4e0abc71f7774491b9d34de59e8da3592a102b643ed1ff0f9c75ac65a85
STEP 3: COMMIT localhost/ping
--> 3ff9c4e0abc71f7774491b9d34de59e8da3592a102b643ed1ff0f9c75ac65a85

$ sudo podman images        
REPOSITORY                                                                       TAG         IMAGE ID       CREATED          SIZE
localhost/echo                                                                   latest      3ff9c4e0abc7   45 seconds ago   6.03 MB
localhost/ping                                                                   latest      3ff9c4e0abc7   45 seconds ago   6.03 MB
...

$ sudo podman run --rm localhost/ping sh -c 'ping -c 1 1.1.1.1'                                                                                                                             
Error: container create failed: time="2019-05-29T11:39:37-04:00" level=error msg="container_linux.go:346: starting container process caused \"exec: \\\"sh\\\": executable file not found in $PATH\"\n"              
container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH"                                                                                                         
: internal libpod error

@miabbott
Copy link
Contributor Author

I don't have permissions to re-open this, but I think it is still a problem

@mheon mheon reopened this May 29, 2019
@danstiner
Copy link

@miabbott Could you re-test with podman 1.3.1 or newer? I was encountering a very similar issue and verified my fix is in that version (containers/buildah#1511)

@miabbott
Copy link
Contributor Author

@danstiner Yup, latest podman works! Thanks for pointing me in the right direction.

$ sudo podman images
REPOSITORY       TAG      IMAGE ID       CREATED              SIZE
localhost/ping   latest   ca1440b415b9   About a minute ago   12.2 MB
localhost/echo   latest   96c73c811c65   2 minutes ago        3.07 MB

$ sudo podman run --rm localhost/echo echo "hello"
hello

$ sudo podman run --rm localhost/ping sh -c 'ping -c 1 1.1.1.1'
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=51 time=29.8 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 29.813/29.813/29.813/0.000 ms

@danstiner
Copy link

No problem, glad it worked :)

@shaicoleman
Copy link

shaicoleman commented Sep 2, 2019 •
edited
Loading

I'm experiencing this issue with podman 1.5.0 on Ubuntu 18.04 (1.5.0-2~ubuntu18.04~ppa5), ext4 filesystem

I have a line in the Dockerfile, e.g.
COPY --chown=myuser:myuser config myapp/config/

And it doesn't seem to recognise when files have changed, and it reuses the layer. Works fine with Docker

Build command line: sudo podman build --rm --build-arg RAILS_ENV=development --build-arg NODE_ENV=development --file rails/Dockerfile -t myimage ..

When changing the current directory to one level up, it doesn't use the cache at all, even for commands that are exactly the same, and don't use COPY.
sudo podman build --rm --build-arg RAILS_ENV=development --build-arg NODE_ENV=development --file docker/rails/Dockerfile -t myimage .

@rhatdan
Copy link
Member

rhatdan commented Sep 2, 2019

@shaicoleman Please open a new issue, if you believe this is a new problem.

@shaicoleman shaicoleman mentioned this issue Sep 2, 2019
Closed
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@TomSweeneyRedHat TomSweeneyRedHat

@umohnani8 umohnani8

Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Correctly detect changes in rootdir
9 participants
@danstiner @shaicoleman @rhatdan @mheon @baude @miabbott @TomSweeneyRedHat @umohnani8 @openshift-ci-robot