Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[build] Experimental flag squash is removed with BuildKit causes image size to 2-4x larger #17319

Open
sheiun-xu opened this issue Nov 28, 2023 · 4 comments
Open

[build] Experimental flag squash is removed with BuildKit causes image size to 2-4x larger #17319

sheiun-xu opened this issue Nov 28, 2023 · 4 comments
Assignees
@saiarcot895
Labels
MSFT Triaged this issue has been triaged

Comments

@sheiun-xu
Copy link

sheiun-xu commented Nov 28, 2023 •
edited
Loading

Description

After docker version updated from 20 to 24 by commit e46be54 (#15652) the --squash experimental flag is no longer available in docker build.

The --squash flag is still invoked in slave.mk

And this would cause the docker-* images won't be squashed anymore, this will result in the final built image being 2-4x larger than the original.

The following warning message will be printed in any target/docker-*.gz.log

WARNING: experimental flag squash is removed with BuildKit. You should squash inside build using a multi-stage Dockerfile for efficiency.

Steps to reproduce the issue:

  1. make init
  2. make configure
  3. make target/docker-base-bullseye.gz
  4. docker load -i target/docker-base-bullseye.gz
  5. docker history docker-base-bullseye

Describe the results you received:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d378eeb0ad23        4 days ago          RUN /bin/sh -c rm /cache.tgz # buildkit         0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   3.13MB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY etc/supervisor/containercfgd.conf /etc/…   231B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY etc/supervisor/supervisord.conf /etc/su…   1.2kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   1.46MB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY root/.vimrc /root/.vimrc # buildkit        39B                 buildkit.dockerfile.v0
<missing>           4 days ago          COPY etc/rsyslog.d/* /etc/rsyslog.d/ # build…   303B                buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   453kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   15.2MB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY debs//socat_1.7.4.1-3_arm64.deb debs//l…   4.52MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   77.6MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   339kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   4.69MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   1.36MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   163kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   4.03MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   453kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   13.7MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   3.04MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   271MB               buildkit.dockerfile.v0
<missing>           4 days ago          COPY apt-multiple-retries /etc/apt/apt.conf.…   107B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY no-check-valid-until /etc/apt/apt.conf.…   178B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY no_install_recommend_suggest /etc/apt/a…   177B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY sources.list.arm64 /etc/apt/sources.lis…   1.74kB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY dpkg_01_drop /etc/dpkg/dpkg.cfg.d/01_dr…   998B                buildkit.dockerfile.v0
<missing>           4 days ago          ENV DEBIAN_FRONTEND=noninteractive              0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   1.19MB              buildkit.dockerfile.v0
<missing>           4 days ago          ENV DISTRO=bullseye                             0B                  buildkit.dockerfile.v0
<missing>           4 days ago          ENV IMAGENAME=docker-base-bullseye              0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   232kB               buildkit.dockerfile.v0
<missing>           4 days ago          COPY vcache/ /sonic/target/vcache/docker-bas…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          COPY buildinfo /usr/local/share/buildinfo # …   26.5kB              buildkit.dockerfile.v0
<missing>           4 days ago          ARG SONIC_VERSION_CONTROL_COMPONENTS            0B                  buildkit.dockerfile.v0
<missing>           4 days ago          ARG SONIC_VERSION_CACHE                         0B                  buildkit.dockerfile.v0
<missing>           3 years ago         /bin/sh -c #(nop) ADD file:27bc8fdb3559dece7…   4.45MB              
<missing>           3 years ago         /bin/sh -c #(nop)  ENV ARCH=arm64 UBUNTU_SUI…   0B                  
<missing>           3 years ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           3 years ago         /bin/sh -c #(nop) ADD file:66cac2b0328037785…   115MB

Describe the results you expected:

This result is from the old version almost every layer is 0B and the latest layer is merged.

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
581f61cc09ca        About an hour ago   RUN /bin/sh -c rm /cache.tgz # buildkit         0B                  buildkit.dockerfile.v0
<missing>           About an hour ago                                                   145MB               merge sha256:dc34bff1633b292662fd50f90050dc8b906f6d8f209b4efbfda225972a4f8d67 to sha256:873181a41b991c9878c60e441108002df9a23ef0bef25ca2f3494ad399d2132d
<missing>           About an hour ago   /bin/sh -c #(nop)  LABEL com.azure.sonic.man…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c post…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c post…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:0cc107d9583abacc…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:07707f5a5ad9f4ca…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c ln /…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:72a3f85df6ef23ab…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:f9f37473474b73fa…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c dpkg…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY multi:67d474349da9908…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c mkdi…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:1519c061413d88ea…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:064709ca31490f58…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:2ef539619e61ab99…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:2347580ff6afa417…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:aafbed21b2425f61…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ENV DEBIAN_FRONTEND=nonin…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c whil…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c find…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pre_…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ENV DISTRO=bullseye          0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ENV IMAGENAME=docker-base…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c dpkg…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY dir:1f0c9c3fcb4591a86…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY dir:3a644c568267c17e2…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ARG SONIC_VERSION_CONTROL…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ARG SONIC_VERSION_CACHE      0B                  
<missing>           7 days ago          /bin/sh -c #(nop)  CMD ["bash"]                 0B                  
<missing>           7 days ago          /bin/sh -c #(nop) ADD file:71543995e4d314b0c…   124MB

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@stepanblyschak
Copy link
Collaborator

stepanblyschak commented Nov 28, 2023 •
edited
Loading

The increase of the size comes from the fact that intermediate layers contain temporary content used to build/install packages which then gets usually removed by apt-get purge, clean commands, etc.
As a short term fix we could use 3rd party tools like - https://github.com/goldmann/docker-squash to squash an image.
In the long run we could really leverage multi staged builds.

@saiarcot895
Copy link
Contributor

In rules/config, do you have SONIC_USE_DOCKER_BUILDKIT enabled? I believe if it's not enabled, then BuildKit should get explicitly disabled because of export DOCKER_BUILDKIT=0 in slave.mk.

@sheiun-xu
Copy link
Author

In rules/config, do you have SONIC_USE_DOCKER_BUILDKIT enabled? I believe if it's not enabled, then BuildKit should get explicitly disabled because of export DOCKER_BUILDKIT=0 in slave.mk.

Both tried, and the layers are not squashed although we specify SONIC_USE_DOCKER_BUILDKIT=y.

In originally (docker-ce<=20) the docker build is combined with docker-ce-cli and the --squash works.
And now after docker-ce upgraded to 23.0.0 the docker build will point to docker-buildx-plugin (alias of docker buildx build).

From Docker Engine version 23.0.0, Buildx is distributed in a separate package: docker-buildx-plugin. In earlier versions, Buildx was included in the docker-ce-cli package.

An as mentioned in

The --squash flag is already deprecated from buildx but still work in original docker build, but after upgrade the docker>=23 the buildx will replace orignal docker build and the squash broken.

@prgeor prgeor added Triaged this issue has been triaged MSFT labels Dec 6, 2023
@k-v1
Copy link
Contributor

k-v1 commented Feb 12, 2024

@sheiun-xu
This bug was fixed for master branch here: #14405
Do you have this fix in your branch?
But ofc we'll need to do something when we switch to the docker buildkit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@saiarcot895 saiarcot895

Labels
MSFT Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@saiarcot895 @stepanblyschak @prgeor @sheiun-xu @k-v1