Some debian package manager tweaks

[Rajpratik71]

By default, Ubuntu or Debian based "apt" or "apt-get" system installs recommended but not suggested packages .

By passing "--no-install-recommends" option, the user lets apt-get know not to consider recommended packages as a dependency to install.

This results in smaller downloads and installation of packages .

Refer to blog at Ubuntu Blog .

[k8s-ci-robot]

Welcome @Rajpratik71!

It looks like this is your first PR to kubernetes/kops 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kops has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @Rajpratik71. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Rajpratik71
To complete the pull request process, please assign mikesplain
You can assign the PR to them by writing /assign @mikesplain in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

/ok-to-test

[Rajpratik71]

almost every test passed , one unrelated test "pull-kops-e2e-kubernetes-aws" related to storage failed with error "FailedMount: Unable to attach or mount volumes: ".

Here , i mentioned unrelated because i think my commit doesn't contain any changes related to this.

[rifelpet]

Looking at the protokube-builder image for example, this is the difference in packages being installed:

before:

The following NEW packages will be installed:
  bash bash-completion binutils ca-certificates cpp cpp-6 curl gcc gcc-6 git git-man krb5-locales less libasan3 libatomic1 libbsd0 libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libcurl3 libcurl3-gnutls libedit2 liberror-perl libexpat1 libffi6
  libgcc-6-dev libgdbm3 libgmp10 libgnutls30 libgomp1 libgpm2 libgssapi-krb5-2 libhogweed4 libidn2-0 libisl15 libitm1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common liblsan0 libmpc3 libmpfr4 libmpx2
  libncurses5 libnghttp2-14 libp11-kit0 libperl5.24 libpopt0 libpsl5 libquadmath0 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1 libssl1.0.2 libssl1.1 libtasn1-6 libtsan0 libubsan0 libunistring0 libx11-6 libx11-data
  libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 linux-libc-dev make manpages manpages-dev netbase openssh-client openssl patch perl perl-modules-5.24 publicsuffix rename rsync xauth
The following packages will be upgraded:
  libc6 perl-base

after:

The following NEW packages will be installed:
  bash binutils cpp cpp-6 curl gcc gcc-6 git git-man libasan3 libatomic1 libcc1-0 libcilkrts5 libcurl3 libcurl3-gnutls liberror-perl libexpat1 libffi6 libgcc-6-dev libgdbm3 libgmp10 libgnutls30 libgomp1 libgssapi-krb5-2 libhogweed4 libidn2-0
  libisl15 libitm1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common liblsan0 libmpc3 libmpfr4 libmpx2 libnghttp2-14 libp11-kit0 libperl5.24 libpsl5 libquadmath0 librtmp1 libsasl2-2 libsasl2-modules-db libssh2-1
  libssl1.0.2 libtasn1-6 libtsan0 libubsan0 libunistring0 make perl perl-modules-5.24
The following packages will be upgraded:
  perl-base

leaving a difference of:

bash-completion ca-certificates krb5-locales less libbsd0 libc-dev-bin libc6-dev libedit2 libgpm2 libncurses5 libpopt0 libsasl2-modules libssl1.1 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 linux-libc-dev manpages manpages-dev netbase openssh-client openssl patch publicsuffix rename rsync xauth

While I agree we should definitely use --no-install-recommends, we may need to make sure that we weren't relying on any packages installed implicitly and instead add them to the apt-get install command. Most of those seem okay to remove but something like ca-certificates may actually be needed. We should probably test this bit more thoroughly.

[rifelpet]

and yes the e2e aws test is definitely unrelated, it can be flaky sometimes.

Thanks for doing this!

/retest

[Rajpratik71]

Looking at the protokube-builder image for example, this is the difference in packages being installed:

before:

The following NEW packages will be installed:
  bash bash-completion binutils ca-certificates cpp cpp-6 curl gcc gcc-6 git git-man krb5-locales less libasan3 libatomic1 libbsd0 libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libcurl3 libcurl3-gnutls libedit2 liberror-perl libexpat1 libffi6
  libgcc-6-dev libgdbm3 libgmp10 libgnutls30 libgomp1 libgpm2 libgssapi-krb5-2 libhogweed4 libidn2-0 libisl15 libitm1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common liblsan0 libmpc3 libmpfr4 libmpx2
  libncurses5 libnghttp2-14 libp11-kit0 libperl5.24 libpopt0 libpsl5 libquadmath0 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1 libssl1.0.2 libssl1.1 libtasn1-6 libtsan0 libubsan0 libunistring0 libx11-6 libx11-data
  libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 linux-libc-dev make manpages manpages-dev netbase openssh-client openssl patch perl perl-modules-5.24 publicsuffix rename rsync xauth
The following packages will be upgraded:
  libc6 perl-base

after:

The following NEW packages will be installed:
  bash binutils cpp cpp-6 curl gcc gcc-6 git git-man libasan3 libatomic1 libcc1-0 libcilkrts5 libcurl3 libcurl3-gnutls liberror-perl libexpat1 libffi6 libgcc-6-dev libgdbm3 libgmp10 libgnutls30 libgomp1 libgssapi-krb5-2 libhogweed4 libidn2-0
  libisl15 libitm1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common liblsan0 libmpc3 libmpfr4 libmpx2 libnghttp2-14 libp11-kit0 libperl5.24 libpsl5 libquadmath0 librtmp1 libsasl2-2 libsasl2-modules-db libssh2-1
  libssl1.0.2 libtasn1-6 libtsan0 libubsan0 libunistring0 make perl perl-modules-5.24
The following packages will be upgraded:
  perl-base

leaving a difference of:

bash-completion ca-certificates krb5-locales less libbsd0 libc-dev-bin libc6-dev libedit2 libgpm2 libncurses5 libpopt0 libsasl2-modules libssl1.1 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 linux-libc-dev manpages manpages-dev netbase openssh-client openssl patch publicsuffix rename rsync xauth

While I agree we should definitely use --no-install-recommends, we may need to make sure that we weren't relying on any packages installed implicitly and instead add them to the apt-get install command. Most of those seem okay to remove but something like ca-certificates may actually be needed. We should probably test this bit more thoroughly.

I agree with the point of "ca-certificates" is needed as besides this i saw many others CI builds failing in the absence of "ca-certificates" , as wget throws certificate error.

For that here , i proposing to add "ca-certificates" in the every possible script and dockerfile and also like to add "apt-utils" package as in ci build log . it taking a lot of time while configuring package after downloading .

[mikesplain]

Thanks for doing this @Rajpratik71! This is looking pretty good, once you feel this is ready to merge, could you make sure you merge things, I think this could likely all be a single commit or two depending on how you'd like to logically break up your changes? Thanks!

[Rajpratik71]

Thanks for doing this @Rajpratik71! This is looking pretty good, once you feel this is ready to merge, could you make sure you merge things, I think this could likely all be a single commit or two depending on how you'd like to logically break up your changes? Thanks!

As requested , created a clean PR #8687 , including all the changes of this PR in two commit . Feel free to merge which one you want

[hakman]

I did a quick test to see how this helps with hooks/nvidia-bootstrap/image/Dockerfile:

• without fix - 153MB
• with fix - 152MB

To get some meaningful results the images should be based on a *-slim image and remove /var/lib/apt/lists/:

• jessie-slim - 94MB
• stretch-slim - 75MB
• buster-slim - 81MB

FROM debian:jessie-slim

RUN \
  apt-get update && \
  apt-get -yq install curl jq && \
  rm -rf /var/lib/apt/lists/*

ADD run.sh /run.sh

CMD [ "/bin/bash", "/run.sh" ]

[Rajpratik71]

I did a quick test to see how this helps with hooks/nvidia-bootstrap/image/Dockerfile:

• without fix - 153MB
• with fix - 152MB

To get some meaningful results the images should be based on a *-slim image and remove /var/lib/apt/lists/:

• jessie-slim - 94MB
• stretch-slim - 75MB
• buster-slim - 81MB

FROM debian:jessie-slim

RUN \
  apt-get update && \
  apt-get -yq install curl jq && \
  rm -rf /var/lib/apt/lists/*

ADD run.sh /run.sh

CMD [ "/bin/bash", "/run.sh" ]

This "--no-install-recommends" is , included in one of the best practices of Docker . Talking about difference in size of images , it vary from one images with few packages to some other images with many layers and a lot of other packages are installed as per requirements .

Switching to "*-slim" version of Docker good at first sight but for larger projects , we have to check packages dependencies manually if not present in image and not satisfied automatically with package managers.

[hakman]

@Rajpratik71 I agree that installing less packages is a good thing in general. Usually it's done for reducing the image size. In this case I would at least add the rm -rf /var/lib/apt/lists/*, also a best practice when installing packages in Docker images.

We can explore using the slim images when we have some more time to test.

[Rajpratik71]

@Rajpratik71 I agree that installing less packages is a good thing in general. Usually it's done for reducing the image size. In this case I would at least add the rm -rf /var/lib/apt/lists/*, also a best practice when installing packages in Docker images.

We can explore using the slim images when we have some more time to test.

Ya , these are the two more ways we can further optimize and test them when possible.

[Rajpratik71]

@rifelpet @mikesplain requesting review

[rifelpet]

Can this be closed as a duplicate of #8687?

[Rajpratik71]

yes