From 7e0ff4e4f7ca5fcfc687aa893b26842cbbf22860 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 21 Jul 2021 19:27:24 -0300 Subject: [PATCH 01/49] Add blog post about new docker images Signed-off-by: Uilian Ries --- .../2021-07-21-Modern-docker-images.markdown | 338 ++++++++++++++++++ 1 file changed, 338 insertions(+) create mode 100644 _posts/2021-07-21-Modern-docker-images.markdown diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown new file mode 100644 index 00000000..764469a7 --- /dev/null +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -0,0 +1,338 @@ +--- +layout: post +comments: false +title: "Modern Docker images: How I Learned to Stop Worrying and Love the Old images" +meta_description: "Conan Docker Tools: New Docker images which should become the official ones" +--- + + +This is the continuation of the initial [post](https://blog.conan.io/2020/06/17/Conan-Docker-Images.html) about new +Docker images for the Conan Center. +Preferably read the first post to understand the problem, the motivation and the proposed +solution for Conan Docker Tools. In this post we will continue the proposed implementation, +highlight the problems encountered during development and the result of this long journey that +should improve in many aspects the way we use Docker images for Conan. + +## The importance of the community + +Right after the presentation of the first proposal of the previous post, we had a great and +important feedback from the community involving Conan. This shows how important these Docker images +are not only for the Conan Center, generating official packages for general distribution, but also +for people who own their personal projects and use the images to validate and build their Conan +recipes, as well as the companies they build packages through sources. + +After long feedback and new formulations, we understood that it was necessary to make changes to +the initial concept, we realized that we could not modify the existing images for better +maintenance, given the risk of breaking the behavior between users, so we started from scratch +again, formulating a more balanced version between Conan Center and users. Unfortunately we can't +keep an old version long enough to support older compilers, but we believe this is the price of +progress, we need to sacrifice older versions for better maintenance for newer versions. + +That said, it is clear how important the involvement of the entire community has been and how much +we take into account its manifestation. We are grateful to the great tribe that surrounds Conan and +makes it more and more complete. + +## Conan Docker Images: Revisited + +The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some weak points to consider: + +* Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to +build +GCC +* APT packages that were requirements for GCC and Clang were still present, further inflating the final image +* The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This +affects the reproducibility requirement. +* Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was +necessary to update the PPA address and build the images again +* The continuous integration service used, although it was considerably fast, it was not possible to customize and +prioritize the build, if necessary. + +Noticing the listed issues, we took a more radical solution, even though it took more time to implement, but resulted +in something better in terms of maintainability and practicality. + +Therefore, we decided to abandon PR #204 and start again from scratch, considering the items listed above. + +## A new plan + +Before we start implementing new recipes with their proper corrections, we first need to understand what the objective +behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have +seen before, in a few years we have reached over 40 recipes. Also, one of the points discussed is whether there will be +rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. + +For current and already available images on hub.docker, these will be kept but no longer supported (new versions will +not be introduced). Your recipes and images will remain available, as their immediate removal would result in +catastrophic failure for many users. They will eventually be removed, but at a distant date and to be defined. + +As for the new images, these will be adopted as official and widely promoted for use by all users. The transition +should take place in a few months after release, because it will be necessary to rebuild the packages that already +exist in the Conan Center and replace them, this is to ensure full compatibility between packages. As for rotation and +maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and +maintenance load for old images and packages that are not always used by the community. Therefore, the following rule +will be adopted: +Clang will be supported in the 3 newest versions. Older versions will be available for download but will not receive +updates. +GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. + +One of the problems we would like to solve is the compiler used and its libraries, we always wanted to be independent +of the Linux distribution. In the initial pull request, we built the GCC from sources, while the Clang uses +prebuilt. So, we chose to build both from sources in order to have more control over the compiler used. + +The library ``libstdc++`` is distributed along with the GCC project. We chose to use a single version of the library, +which +was neither the newest, to allow older distributions to use, but also not so old, so that new features can be consumed +by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10. + +Ubuntu 16.04 Xenial LTS is still the base used, its support will be until April 2024. After that date, we will need to +update the images to a newer version of the distribution, in addition to rebuilding all available official packages. + +So forward thinking is having fewer images, but better support, without the drastic breakage and incompatibility issues + +## From blueprint to prototype + +During prototyping, we realized that we could divide the process of building a Docker image into 3 phases: +The base, where all common packages are installed to all images, such as Python, git, svn, etc, in addition to the +non-root user configuration. +An image where only the compiler is built. In this container can be installed packages referring to the compiler build +only, which will not be present in the final image, for example, Ninja, which is used for LLVM. +Finally, we need to merge the base to the produced compiler into a single image, without adding extra packages, but +still reusable between each compiler version. + +For the case of the base image, this one is still quite modular, just changing the variables file to update the package +to be installed. The complete recipe can be obtained +[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/base/Dockerfile), but let's look +at a few pieces: + +{% highlight docker %} + +ARG DISTRO_VERSION +FROM ubuntu:${DISTRO_VERSION} + +ENV PYENV_ROOT=/opt/pyenv \ + PATH=/opt/pyenv/shims:${PATH} + +ARG CMAKE_VERSION +ARG CMAKE_VERSION_FULL +ARG PYTHON_VERSION +ARG CONAN_VERSION + +{% endhighlight %} + +Now, both the distribution version and the installed packages are configurable in terms of version used. Previously, a +script was used to update all 42 recipes as needed! + +{% highlight docker %} + +RUN printf '/usr/local/lib64\n' >> /etc/ld.so.conf.d/20local-lib.conf \ + && printf '/usr/local/lib\n' > /etc/ld.so .conf.d/20local-lib.conf \ + ... + +{% endhighlight %} + +In order to not be affected by system packages or Conan packages that invoke ``apt-get``, the compiler and its +artifacts are installed in ``/usr/local``. However, this is not enough to prioritize the order ``libstdc++`` used, for +that we need to update ``ldconfig`` with the local directories. Until then, this was not necessary in the previous +images, as everything was either consumed directly from the system, or installed directly in /usr. + +These are the main features of the base image, which is used in all final images. + +## Building GCC + +Now let's look at the GCC build image, the full recipe can be found +[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/gcc/Dockerfile), but let's +highlight a few points: + +{% highlight docker %} + +RUN cd gcc-${GCC_VERSION} \ + && ./configure --build=x86_64-linux-gnu --disable-bootstrap --disable-multilib ... + +{% endhighlight %} + +No matter the version, GCC continues to use the same lines for its build. +Some factors were configured in this version used: +* The multilib support was discarded as we are only interested in producing packages with 64-bit support. +* Fortran support has been added, thus producing ``gfortran`` together in the image. Currently the Conan package for +``gfortran`` is totally broken and has a complex dependencies chain to fix. +* Bootstrap has been disabled to reduce build time to just 20 minutes. + +Once GCC 10 is built, its libstdc++ generated is copied to an Artifactory instance and this is downloaded directly to +each final image. Actually, that was the original intent, but as we will see, that was not possible. While we were +developing the new recipes, GCC 11 was released and along with it, the version was ``libstdc++`` already ``.29``, which +made usage incompatible. In this case, we are left with the following dilemma: + +* Using the same libstdc++ version, except for GCC 11. + * Conan Center becomes homogeneous, everything can be used without compiler restriction and distribution + * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library +that is not yet available in official PPA. All build requirements would be broken. + * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of +risks. +* Each image uses the corresponding version of libstdc++ provided by the compiler. + * Better than the current scenario, where it is dependent on PPA and we have no control over it. + * All images still use the same version of glibc, another advantage over the current scenario. + * We will need to take care of the build requirements, as they will only be compatible with later versions. + * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc +version is still the same version for everyone. + +Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the +compiler**. + +The last part of the image uses the concept of Docker +[multistage-build](https://docs.docker.com/develop/develop-images/multistage-build/), a technique that avoids creating +separate recipes and images to take advantage of common parts. + + +{% highlight docker %} + +FROM ${DOCKER_USERNAME}/base-${DISTRO}:${DOCKER_TAG} as deploy + +ARG GCC_VERSION +ARG LIBSTDCPP_PATCH_VERSION + +COPY --from=builder /tmp/install /tmp/install + +RUN sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ + && sudo cp -a /tmp/install/lib/gcc/x86_64-linux-gnu/${GCC_VERSION} /usr/lib/gcc/x86_64-linux-gnu/ \ + && sudo cp -a /tmp/install/include/* /usr/local/include/ \ + && sudo cp -a /tmp/install/lib64/ /usr/local/ \ + && sudo cp -a /tmp/install/libexec/ /usr/local/ \ + && sudo cp -a /tmp/install/lib/* /usr/local/lib/ \ + && sudo cp -a /tmp/install/bin/* /usr/local/bin/ \ + && sudo rm -rf /tmp/install \ + && sudo update-alternatives --install /usr/local/bin/cc cc /usr/local/bin/gcc 100 \ + && sudo update-alternatives --install /usr/local/bin/cpp cpp /usr/local/bin/g++ 100 \ + && sudo update-alternatives --install /usr/local/bin/c++ c++ /usr/local/bin/g++ 100 \ + && sudo rm /etc/ld.so.cache \ + && sudo ldconfig -C /etc/ld.so.cache \ + && conan config init --force + +{% endhighlight %} + +In this section, the base image used is the same one we created before, through a caching mechanism, this drastically +reduces the final image build time. +Also, all artifacts generated from GCC are now copied to their respective locations. Finally, the compiler becomes the +default in the image and the libraries are listed and cached. And here's the icing on the cake, the same recipe works +from GCC 5 to the latest version, just modifying some arguments. The maintenance has been drastically simplified +compared to current Conan Docker Tools. + +## Conan meets the Wyvern + +For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles +and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and +compilation hours. To see the full recipe, it is available +[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). + +As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct +dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` +and using ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` would only +be used for Conan packages, not as a Clang requirement. However, we had some situations and the need for some actions +that will be listed here: + +* The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if +necessary. +* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The +compiler is pre-built and distributed by the official LLVM PPA. +* From version to version, options are added or removed, reflecting the evolution of project features and legacy +deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options +do not work in subsequent versions or which option should be used to specify the preferred library. +* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for +each attempt, a long wait was needed to get the result. +* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. As a solution, the +project supports a configuration file, where ``libc++`` can be specified by default. However, this behavior changes +between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for +all versions. +* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already +internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the +image with Clang and installs Conan's libunwind package? A big mess when linking, is the answer. Clang tries to link +the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original +LLVM ``libunwind`` to ``libllvm-unwind``. + +With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of +discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version +9, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux +environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. + +Now that the difficulties faced are clear, let's go a step further and detail the Clang deployment step. + +{% highlight docker %} + +FROM ${DOCKER_USERNAME}/gcc${LIBSTDCPP_MAJOR_VERSION}-${DISTRO}:${DOCKER_TAG} as libstdcpp + +FROM ${DOCKER_USERNAME}/base-${DISTRO}:${DOCKER_TAG} as deploy + +ARG LIBSTDCPP_VERSION +ARG LIBSTDCPP_PATCH_VERSION + +ARG DOCKER_USERNAME +ARG DOCKER_TAG +ARG DISTRO + +COPY --from=builder /tmp/install /tmp/clang +COPY --from=libstdcpp /usr/local /tmp/gcc + +RUN sudo mv /tmp/gcc/lib64 /usr/local/ \ + && sudo ln -s -f /usr/local/lib64/libstdc++.so.6.0.${LIBSTDCPP_PATCH_VERSION} /usr/local/lib64/libstdc++.so.6 \ + && sudo ln -s -f /usr/local/lib64/libstdc++.so.6 /usr/local/lib64/libstdc++.so \ + && sudo cp -a /tmp/gcc/include/* /usr/local/include/ \ + && sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ + && sudo cp -a /tmp/gcc/lib/gcc/x86_64-linux-gnu/${LIBSTDCPP_VERSION} /usr/lib/gcc/x86_64-linux-gnu/ \ + && sudo cp -a /tmp/gcc/lib/* /usr/local/lib/ \ + ... + +{% endhighlight %} + +Similar to what was done with GCC, in Clang we also use the same base image and copy the artifacts generated by the +compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This +is a necessity of the possible configurations supported by Conan. + +## Tests and more tests + +To ensure that the images produced met our requirements, we needed to add new tests that cover in addition to what was +already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions +of installed binaries and user permissions. The content of the new tests can be seen +[here](https://github.com/conan-io/conan-docker-tools/tree/feature/single-image/modern/tests). + +We introduced greater modularization in the tests, dividing the steps into separate scripts, to serve each compiler +closer. With the support for Fortran, it was necessary to adapt a test that covered it. Furthermore, many applications +that are now bundled with Conan are no longer part of the base image and this has also been validated. + +If you want to test locally a produced Docker image, you can easily run: + +{% highlight bash %} + +$ cd modern && pytest tests --image conanio/gcc10-ubuntu16.04 --service deploy -vv --user 0:0 + +{% endhighlight %} + +## The CI service change + +Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, +this did not give us the full power to prioritize the build in the queue, or customize the host, or customize the build +lines to use Docker-in-Docker if necessary. + +With that in mind, we started using Jenkins from the Conan Center to also build the new Docker images. The big +advantage in this is the use of features for cache in Docker. Previously, a single job took up to 2h if it was on other +services, without the use of caching. Now, using Jenkins and Docker ``--cache-from``, updating a package, from base +image to final image, takes just 4 minutes per job. The Jenkinsfile file used can be viewed +[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/.ci/xenial.jenkinsfile). + +Although the file looks complicated at first glance, it is still possible to use docker-compose to build an image from +scratch. As an example, let's use Clang 12: + +{% highlight bash %} + +$ cd modern && docker-compose build clang12 + +{% endhighlight %} + +The produced image will be named as ``conanio/clang12-ubuntu16.04:1.38.0``, where ``1.38.0`` is the image tag and +version of Conan installed. But it's totally configurable by the .env file. + +In the case of legacy images, they will continue to be built in Azure when needed, we have no intention of moving them +to Jenkins due to effort and maintenance. + +## Finals words and feedback + +We invite everyone who uses Conan Docker Tools images to use this new formulation, which should become official soon. +These new images are the result of a long journey, where we learned from our mistakes and listened to the community to +reach what we have today. We believe improvements should be added continuously to keep the CDT progressing, so please +let us your feedback on issue [#205](https://github.com/conan-io/conan-docker-tools/issues/205). From 8c8b8eaa8e9a1a2cda7bb81374022de1e75edf7f Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Thu, 22 Jul 2021 15:32:01 -0300 Subject: [PATCH 02/49] Code review suggestion Signed-off-by: Uilian Ries --- .../2021-07-21-Modern-docker-images.markdown | 37 ++++++++++++------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 764469a7..d52d2765 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -1,8 +1,10 @@ --- layout: post comments: false -title: "Modern Docker images: How I Learned to Stop Worrying and Love the Old images" -meta_description: "Conan Docker Tools: New Docker images which should become the official ones" +title: "Modern Docker Images to build C/C++ projects for ConanCenter: How I Learned to Stop Worrying and Love the Old images" +meta_description: "The New Docker images which should become the official to build all C/C++ package on Conan Center Index +and distribute under Conan Center. They should be adopted by default for all users. Fixed glibc and libstdc++ compatibility +problems and unified all Docker recipes using multistage-build on Jenkins" --- @@ -52,7 +54,7 @@ in something better in terms of maintainability and practicality. Therefore, we decided to abandon PR #204 and start again from scratch, considering the items listed above. -## A new plan +## A new plan: Creating images with same base Before we start implementing new recipes with their proper corrections, we first need to understand what the objective behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have @@ -69,9 +71,9 @@ exist in the Conan Center and replace them, this is to ensure full compatibility maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule will be adopted: -Clang will be supported in the 3 newest versions. Older versions will be available for download but will not receive +* Clang will be supported in the 3 newest versions. Older versions will be available for download but will not receive updates. -GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. +* GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. One of the problems we would like to solve is the compiler used and its libraries, we always wanted to be independent of the Linux distribution. In the initial pull request, we built the GCC from sources, while the Clang uses @@ -85,16 +87,23 @@ by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same dis Ubuntu 16.04 Xenial LTS is still the base used, its support will be until April 2024. After that date, we will need to update the images to a newer version of the distribution, in addition to rebuilding all available official packages. -So forward thinking is having fewer images, but better support, without the drastic breakage and incompatibility issues +So forward thinking is having fewer images, but better support, without the drastic breakage and incompatibility issues. +To summarize the plan: -## From blueprint to prototype +* Ubuntu 16.04 LTS as base Docker image +* Build Clang and GCC from source +* Use libstdc++.so.0.6.28 for all new Docker images +* Use glibc 2.23 for all new Docker images +* Rotate old compiler versions to keep better support in the future + +## From blueprint to prototype: Writing the new Docker recipes During prototyping, we realized that we could divide the process of building a Docker image into 3 phases: -The base, where all common packages are installed to all images, such as Python, git, svn, etc, in addition to the +* The base, where all common packages are installed to all images, such as Python, git, svn, etc, in addition to the non-root user configuration. -An image where only the compiler is built. In this container can be installed packages referring to the compiler build +* An image where only the compiler is built. In this container can be installed packages referring to the compiler build only, which will not be present in the final image, for example, Ninja, which is used for LLVM. -Finally, we need to merge the base to the produced compiler into a single image, without adding extra packages, but +* Finally, we need to merge the base to the produced compiler into a single image, without adding extra packages, but still reusable between each compiler version. For the case of the base image, this one is still quite modular, just changing the variables file to update the package @@ -135,7 +144,7 @@ images, as everything was either consumed directly from the system, or installed These are the main features of the base image, which is used in all final images. -## Building GCC +## Building GCC from source Now let's look at the GCC build image, the full recipe can be found [here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/gcc/Dockerfile), but let's @@ -214,7 +223,7 @@ default in the image and the libraries are listed and cached. And here's the ici from GCC 5 to the latest version, just modifying some arguments. The maintenance has been drastically simplified compared to current Conan Docker Tools. -## Conan meets the Wyvern +## Conan meets the Wyvern: Building Clang C/C++ compiler from source For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and @@ -284,7 +293,7 @@ Similar to what was done with GCC, in Clang we also use the same base image and compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan. -## Tests and more tests +## Tests and more tests: A CI pipeline to test Docker images To ensure that the images produced met our requirements, we needed to add new tests that cover in addition to what was already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions @@ -303,7 +312,7 @@ $ cd modern && pytest tests --image conanio/gcc10-ubuntu16.04 --service deploy - {% endhighlight %} -## The CI service change +## The CI service change: From Travis to Azure and the Jenkins arrival Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, this did not give us the full power to prioritize the build in the queue, or customize the host, or customize the build From b164d6a2e389d5264aed5bd8f707ff55e70a70d0 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:20:21 -0300 Subject: [PATCH 03/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index d52d2765..949178dc 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -39,8 +39,7 @@ makes it more and more complete. The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some weak points to consider: * Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to -build -GCC +build GCC. * APT packages that were requirements for GCC and Clang were still present, further inflating the final image * The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This affects the reproducibility requirement. From f15f9eaff0fc6c4de84b186d5f8aeb63a329cd2d Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:20:28 -0300 Subject: [PATCH 04/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 949178dc..e84f7fd9 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -40,7 +40,7 @@ The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) * Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to build GCC. -* APT packages that were requirements for GCC and Clang were still present, further inflating the final image +* APT packages that were requirements for GCC and Clang were still present, further inflating the final image. * The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This affects the reproducibility requirement. * Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was From 6b929814837776c4a6fdd4e62936cbebfce47a7c Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:20:35 -0300 Subject: [PATCH 05/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index e84f7fd9..fb887aa1 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -53,7 +53,7 @@ in something better in terms of maintainability and practicality. Therefore, we decided to abandon PR #204 and start again from scratch, considering the items listed above. -## A new plan: Creating images with same base +## A new plan: Using the same base image Before we start implementing new recipes with their proper corrections, we first need to understand what the objective behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have From 9c5c514b2f680500d84a8a8ef37baa2e2cf4ba31 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:20:44 -0300 Subject: [PATCH 06/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index fb887aa1..a56561a8 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -55,7 +55,7 @@ Therefore, we decided to abandon PR #204 and start again from scratch, consideri ## A new plan: Using the same base image -Before we start implementing new recipes with their proper corrections, we first need to understand what the objective +Before we start implementing new dockerfiles with their proper corrections, we first need to understand what the objective behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have seen before, in a few years we have reached over 40 recipes. Also, one of the points discussed is whether there will be rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. From 98a537d4132f5120a7c1a73c8b6784d1a8ac51a8 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:20:56 -0300 Subject: [PATCH 07/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index a56561a8..19134003 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -57,7 +57,7 @@ Therefore, we decided to abandon PR #204 and start again from scratch, consideri Before we start implementing new dockerfiles with their proper corrections, we first need to understand what the objective behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have -seen before, in a few years we have reached over 40 recipes. Also, one of the points discussed is whether there will be +seen before, in a few years with the current approach we are already maintaining over 40 different dockerfiles. Also, one of the points discussed is whether there will be rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. For current and already available images on hub.docker, these will be kept but no longer supported (new versions will From bf8e6c8c191b6d2a1d9f44adf892753cb488a9e7 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:21:06 -0300 Subject: [PATCH 08/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 19134003..a02d8c72 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -139,7 +139,7 @@ RUN printf '/usr/local/lib64\n' >> /etc/ld.so.conf.d/20local-lib.conf \ In order to not be affected by system packages or Conan packages that invoke ``apt-get``, the compiler and its artifacts are installed in ``/usr/local``. However, this is not enough to prioritize the order ``libstdc++`` used, for that we need to update ``ldconfig`` with the local directories. Until then, this was not necessary in the previous -images, as everything was either consumed directly from the system, or installed directly in /usr. +images, as everything was either consumed directly from the system, or installed directly in ``/usr``. These are the main features of the base image, which is used in all final images. From 80a1e58f96a082c74d6977684efcce267c8b8b60 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:21:26 -0300 Subject: [PATCH 09/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index a02d8c72..04a95fdf 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -65,7 +65,7 @@ not be introduced). Your recipes and images will remain available, as their imme catastrophic failure for many users. They will eventually be removed, but at a distant date and to be defined. As for the new images, these will be adopted as official and widely promoted for use by all users. The transition -should take place in a few months after release, because it will be necessary to rebuild the packages that already +to the new images in Conan Center should take place in a few months after release, because it will be necessary to rebuild the packages that already exist in the Conan Center and replace them, this is to ensure full compatibility between packages. As for rotation and maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule From 525368c2196449c909406d52587eb125020a43a4 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:22:21 -0300 Subject: [PATCH 10/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 04a95fdf..f9096a56 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -256,7 +256,7 @@ LLVM ``libunwind`` to ``libllvm-unwind``. With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version -9, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux +10, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. Now that the difficulties faced are clear, let's go a step further and detail the Clang deployment step. From 01b00b4019d217517912f6cfbf2a5f9909293486 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:22:33 -0300 Subject: [PATCH 11/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index f9096a56..b712a1e3 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -317,7 +317,7 @@ Since the beginning of the project, Conan Docker Tools has always used CI servic this did not give us the full power to prioritize the build in the queue, or customize the host, or customize the build lines to use Docker-in-Docker if necessary. -With that in mind, we started using Jenkins from the Conan Center to also build the new Docker images. The big +With that in mind, we started using Jenkins to also build the new Docker images. The big advantage in this is the use of features for cache in Docker. Previously, a single job took up to 2h if it was on other services, without the use of caching. Now, using Jenkins and Docker ``--cache-from``, updating a package, from base image to final image, takes just 4 minutes per job. The Jenkinsfile file used can be viewed From 66b5b59fd85b34b98a5c34e0dca88a9d79994984 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:22:48 -0300 Subject: [PATCH 12/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index b712a1e3..35687ab3 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -328,7 +328,10 @@ scratch. As an example, let's use Clang 12: {% highlight bash %} -$ cd modern && docker-compose build clang12 +$ cd modern +$ docker-compose build base +$ docker-compose build clang12-builder +$ docker-compose build clang12 {% endhighlight %} From 2b0c640dc4f6b32bf5356f94bb02dae6a3802ea1 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:47:57 -0300 Subject: [PATCH 13/49] Apply review consideration Signed-off-by: Uilian Ries --- _posts/2021-07-21-Modern-docker-images.markdown | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 35687ab3..0efb31fb 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -70,8 +70,7 @@ exist in the Conan Center and replace them, this is to ensure full compatibility maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule will be adopted: -* Clang will be supported in the 3 newest versions. Older versions will be available for download but will not receive -updates. +* Clang will be supported from 10.0 to the newest version. We will update according new releases be available. * GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. One of the problems we would like to solve is the compiler used and its libraries, we always wanted to be independent @@ -81,7 +80,8 @@ prebuilt. So, we chose to build both from sources in order to have more control The library ``libstdc++`` is distributed along with the GCC project. We chose to use a single version of the library, which was neither the newest, to allow older distributions to use, but also not so old, so that new features can be consumed -by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10. +by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also +is the default version in Ubuntu 20.04 LTS (Focal). Ubuntu 16.04 Xenial LTS is still the base used, its support will be until April 2024. After that date, we will need to update the images to a newer version of the distribution, in addition to rebuilding all available official packages. @@ -93,7 +93,8 @@ To summarize the plan: * Build Clang and GCC from source * Use libstdc++.so.0.6.28 for all new Docker images * Use glibc 2.23 for all new Docker images -* Rotate old compiler versions to keep better support in the future +* Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers. + ## From blueprint to prototype: Writing the new Docker recipes @@ -328,7 +329,7 @@ scratch. As an example, let's use Clang 12: {% highlight bash %} -$ cd modern +$ cd modern $ docker-compose build base $ docker-compose build clang12-builder $ docker-compose build clang12 From d64f66f7d93fc491b6c2e931b12707e70173d7f2 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 15:54:06 -0300 Subject: [PATCH 14/49] Apply review consideration Signed-off-by: Uilian Ries --- _posts/2021-07-21-Modern-docker-images.markdown | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 0efb31fb..9c09bafa 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -70,12 +70,15 @@ exist in the Conan Center and replace them, this is to ensure full compatibility maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule will be adopted: -* Clang will be supported from 10.0 to the newest version. We will update according new releases be available. +* Clang will be supported from 10.0 to the newest version. * GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. +* For both compilers we will keep updating all new compiler versions and Conan client version, according new releases. One of the problems we would like to solve is the compiler used and its libraries, we always wanted to be independent of the Linux distribution. In the initial pull request, we built the GCC from sources, while the Clang uses prebuilt. So, we chose to build both from sources in order to have more control over the compiler used. +Also, packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible +different distros. The library ``libstdc++`` is distributed along with the GCC project. We chose to use a single version of the library, which From 5c95cd6dfa40b3a49c1fa18a2decc36c7abc2c8f Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 16:23:03 -0300 Subject: [PATCH 15/49] Add Parts 1 and 2 Signed-off-by: Uilian Ries --- .../2021-07-21-Modern-docker-images.markdown | 38 ++++++++++++++----- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 9c09bafa..fa16be86 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -15,6 +15,13 @@ solution for Conan Docker Tools. In this post we will continue the proposed impl highlight the problems encountered during development and the result of this long journey that should improve in many aspects the way we use Docker images for Conan. +Also, this blog is a bit longer than usual, and contains all explanation about the decisions and mistakes +committed, plus the technical part with Dockerfiles in details. To attend all kind of readers, we divided +this blog post in 2 parts: + +* Part 1: The long journey, ideas, motivation, mistakes and challenges +* Part 2: Under the hood of Dockerfiles and technical details + ## The importance of the community Right after the presentation of the first proposal of the previous post, we had a great and @@ -34,7 +41,13 @@ That said, it is clear how important the involvement of the entire community has we take into account its manifestation. We are grateful to the great tribe that surrounds Conan and makes it more and more complete. -## Conan Docker Images: Revisited + +## Part 1: The long journey, ideas, motivation, mistakes and challenges + +On this section we will describe how we revisited the initial proposal, found problems, proposed solutions and started a +new entire idea. If you are more interested reading only the more technical part, start from the Part 2. + +### Conan Docker Images: Revisited The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some weak points to consider: @@ -53,7 +66,7 @@ in something better in terms of maintainability and practicality. Therefore, we decided to abandon PR #204 and start again from scratch, considering the items listed above. -## A new plan: Using the same base image +### A new plan: Using the same base image Before we start implementing new dockerfiles with their proper corrections, we first need to understand what the objective behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have @@ -73,6 +86,9 @@ will be adopted: * Clang will be supported from 10.0 to the newest version. * GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. * For both compilers we will keep updating all new compiler versions and Conan client version, according new releases. +* The multilib support was discarded as we are only interested in producing packages with 64-bit support. +* Fortran support has been added, thus producing ``gfortran`` together in the image. Currently the Conan package for +``gfortran`` is totally broken and has a complex dependencies chain to fix. One of the problems we would like to solve is the compiler used and its libraries, we always wanted to be independent of the Linux distribution. In the initial pull request, we built the GCC from sources, while the Clang uses @@ -99,7 +115,7 @@ To summarize the plan: * Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers. -## From blueprint to prototype: Writing the new Docker recipes +### From blueprint to prototype: Writing the new Docker recipes During prototyping, we realized that we could divide the process of building a Docker image into 3 phases: * The base, where all common packages are installed to all images, such as Python, git, svn, etc, in addition to the @@ -147,7 +163,12 @@ images, as everything was either consumed directly from the system, or installed These are the main features of the base image, which is used in all final images. -## Building GCC from source +## Part 2: Under the hood of Dockerfiles and technical details + +Here we will be more focused on the final product, Dockerfiles, tests and CI. If you are interested to read about +our decisions, read the Part 1 first. + +### Building GCC from source Now let's look at the GCC build image, the full recipe can be found [here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/gcc/Dockerfile), but let's @@ -162,9 +183,6 @@ RUN cd gcc-${GCC_VERSION} \ No matter the version, GCC continues to use the same lines for its build. Some factors were configured in this version used: -* The multilib support was discarded as we are only interested in producing packages with 64-bit support. -* Fortran support has been added, thus producing ``gfortran`` together in the image. Currently the Conan package for -``gfortran`` is totally broken and has a complex dependencies chain to fix. * Bootstrap has been disabled to reduce build time to just 20 minutes. Once GCC 10 is built, its libstdc++ generated is copied to an Artifactory instance and this is downloaded directly to @@ -226,7 +244,7 @@ default in the image and the libraries are listed and cached. And here's the ici from GCC 5 to the latest version, just modifying some arguments. The maintenance has been drastically simplified compared to current Conan Docker Tools. -## Conan meets the Wyvern: Building Clang C/C++ compiler from source +### Conan meets the Wyvern: Building Clang C/C++ compiler from source For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and @@ -296,7 +314,7 @@ Similar to what was done with GCC, in Clang we also use the same base image and compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan. -## Tests and more tests: A CI pipeline to test Docker images +### Tests and more tests: A CI pipeline to test Docker images To ensure that the images produced met our requirements, we needed to add new tests that cover in addition to what was already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions @@ -315,7 +333,7 @@ $ cd modern && pytest tests --image conanio/gcc10-ubuntu16.04 --service deploy - {% endhighlight %} -## The CI service change: From Travis to Azure and the Jenkins arrival +### The CI service change: From Travis to Azure and the Jenkins arrival Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, this did not give us the full power to prioritize the build in the queue, or customize the host, or customize the build From 870c3f871964666902fe12d4fff1f8d08c04a8c1 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 2 Aug 2021 16:45:03 -0300 Subject: [PATCH 16/49] Add pilcrow to parts Signed-off-by: Uilian Ries --- _posts/2021-07-21-Modern-docker-images.markdown | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index fa16be86..98ef8fac 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -19,8 +19,8 @@ Also, this blog is a bit longer than usual, and contains all explanation about t committed, plus the technical part with Dockerfiles in details. To attend all kind of readers, we divided this blog post in 2 parts: -* Part 1: The long journey, ideas, motivation, mistakes and challenges -* Part 2: Under the hood of Dockerfiles and technical details +* Part 1: The long journey, ideas, motivation, mistakes and challenges +* Part 2: Under the hood of Dockerfiles and technical details ## The importance of the community @@ -41,7 +41,7 @@ That said, it is clear how important the involvement of the entire community has we take into account its manifestation. We are grateful to the great tribe that surrounds Conan and makes it more and more complete. - +

## Part 1: The long journey, ideas, motivation, mistakes and challenges On this section we will describe how we revisited the initial proposal, found problems, proposed solutions and started a @@ -163,6 +163,7 @@ images, as everything was either consumed directly from the system, or installed These are the main features of the base image, which is used in all final images. +

## Part 2: Under the hood of Dockerfiles and technical details Here we will be more focused on the final product, Dockerfiles, tests and CI. If you are interested to read about From b17ded7e1470ebcb96db2661193bb75bf0735a88 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Thu, 5 Aug 2021 18:34:48 -0300 Subject: [PATCH 17/49] Include package build example Signed-off-by: Uilian Ries --- .../2021-07-21-Modern-docker-images.markdown | 172 ++++++++++++------ 1 file changed, 115 insertions(+), 57 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 98ef8fac..9b4eafea 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -100,7 +100,27 @@ The library ``libstdc++`` is distributed along with the GCC project. We chose to which was neither the newest, to allow older distributions to use, but also not so old, so that new features can be consumed by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also -is the default version in Ubuntu 20.04 LTS (Focal). +is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 is built, its ``libstdc++`` generated is copied to an +Artifactory instance and this is downloaded directly to +each final image. Actually, that was the original intent, but as we will see, that was not possible. While we were +developing the new recipes, GCC 11 was released and along with it, the version was ``libstdc++`` already ``.29``, which +made usage incompatible. In this case, we are left with the following dilemma: + +* Using the same libstdc++ version, except for GCC 11. + * Conan Center becomes homogeneous, everything can be used without compiler restriction and distribution + * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library +that is not yet available in official PPA. All build requirements would be broken. + * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of +risks. +* Each image uses the corresponding version of libstdc++ provided by the compiler. + * Better than the current scenario, where it is dependent on PPA and we have no control over it. + * All images still use the same version of glibc, another advantage over the current scenario. + * We will need to take care of the build requirements, as they will only be compatible with later versions. + * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc +version is still the same version for everyone. + +Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the +compiler**. Ubuntu 16.04 Xenial LTS is still the base used, its support will be until April 2024. After that date, we will need to update the images to a newer version of the distribution, in addition to rebuilding all available official packages. @@ -110,7 +130,7 @@ To summarize the plan: * Ubuntu 16.04 LTS as base Docker image * Build Clang and GCC from source -* Use libstdc++.so.0.6.28 for all new Docker images +* Use libstdc++ provided by the compiler * Use glibc 2.23 for all new Docker images * Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers. @@ -163,6 +183,44 @@ images, as everything was either consumed directly from the system, or installed These are the main features of the base image, which is used in all final images. +For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles +and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and +compilation hours. To see the full recipe, it is available +[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). + + +### Training the Dragon: Clang Dockefile + +As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct +dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` +and using ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` would only +be used for Conan packages, not as a Clang requirement. However, we had some situations and the need for some actions +that will be listed here: + +* The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if +necessary. +* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The +compiler is pre-built and distributed by the official LLVM PPA. +* From version to version, options are added or removed, reflecting the evolution of project features and legacy +deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options +do not work in subsequent versions or which option should be used to specify the preferred library. +* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for +each attempt, a long wait was needed to get the result. +* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. As a solution, the +project supports a configuration file, where ``libc++`` can be specified by default. However, this behavior changes +between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for +all versions. +* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already +internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the +image with Clang and installs Conan's libunwind package? A big mess when linking, is the answer. Clang tries to link +the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original +LLVM ``libunwind`` to ``libllvm-unwind``. + +With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of +discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version +10, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux +environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. +

## Part 2: Under the hood of Dockerfiles and technical details @@ -185,27 +243,8 @@ RUN cd gcc-${GCC_VERSION} \ No matter the version, GCC continues to use the same lines for its build. Some factors were configured in this version used: * Bootstrap has been disabled to reduce build time to just 20 minutes. +* Fortran is enabled, but it barely increase the building time and final -Once GCC 10 is built, its libstdc++ generated is copied to an Artifactory instance and this is downloaded directly to -each final image. Actually, that was the original intent, but as we will see, that was not possible. While we were -developing the new recipes, GCC 11 was released and along with it, the version was ``libstdc++`` already ``.29``, which -made usage incompatible. In this case, we are left with the following dilemma: - -* Using the same libstdc++ version, except for GCC 11. - * Conan Center becomes homogeneous, everything can be used without compiler restriction and distribution - * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library -that is not yet available in official PPA. All build requirements would be broken. - * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of -risks. -* Each image uses the corresponding version of libstdc++ provided by the compiler. - * Better than the current scenario, where it is dependent on PPA and we have no control over it. - * All images still use the same version of glibc, another advantage over the current scenario. - * We will need to take care of the build requirements, as they will only be compatible with later versions. - * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc -version is still the same version for everyone. - -Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the -compiler**. The last part of the image uses the concept of Docker [multistage-build](https://docs.docker.com/develop/develop-images/multistage-build/), a technique that avoids creating @@ -247,42 +286,10 @@ compared to current Conan Docker Tools. ### Conan meets the Wyvern: Building Clang C/C++ compiler from source -For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles -and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and -compilation hours. To see the full recipe, it is available +To see the full recipe, it is available [here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). -As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct -dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` -and using ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` would only -be used for Conan packages, not as a Clang requirement. However, we had some situations and the need for some actions -that will be listed here: - -* The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if -necessary. -* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The -compiler is pre-built and distributed by the official LLVM PPA. -* From version to version, options are added or removed, reflecting the evolution of project features and legacy -deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options -do not work in subsequent versions or which option should be used to specify the preferred library. -* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for -each attempt, a long wait was needed to get the result. -* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. As a solution, the -project supports a configuration file, where ``libc++`` can be specified by default. However, this behavior changes -between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for -all versions. -* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already -internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the -image with Clang and installs Conan's libunwind package? A big mess when linking, is the answer. Clang tries to link -the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original -LLVM ``libunwind`` to ``libllvm-unwind``. - -With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of -discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version -10, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux -environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. - -Now that the difficulties faced are clear, let's go a step further and detail the Clang deployment step. +Let's go a step further and detail the Clang deployment step. {% highlight docker %} @@ -315,6 +322,27 @@ Similar to what was done with GCC, in Clang we also use the same base image and compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan. +Besides that, Clang requires some interesting CMake definitions: +* LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build +* LLVM_USE_LINKER: We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time + +{% highlight docker %} + +... +&& ninja unwind \ +&& ninja cxxabi \ +&& cp lib/libc++abi* /usr/lib/ \ +&& ninja cxx \ +&& ninja clang \ + +{% endhighlight %} + +If we run `ninja` command alone, it builds more projects than we want configured as enabled, so we +build one by one. +Also, `libcxx` has a limitation when building using `libc++abi`, it searches on system library folder, +not the internal folders first. + + ### Tests and more tests: A CI pipeline to test Docker images To ensure that the images produced met our requirements, we needed to add new tests that cover in addition to what was @@ -358,12 +386,42 @@ $ docker-compose build clang12 {% endhighlight %} -The produced image will be named as ``conanio/clang12-ubuntu16.04:1.38.0``, where ``1.38.0`` is the image tag and +The produced image will be named as ``conanio/clang12-ubuntu16.04:1.39.0``, where ``1.39.0`` is the image tag and version of Conan installed. But it's totally configurable by the .env file. In the case of legacy images, they will continue to be built in Azure when needed, we have no intention of moving them to Jenkins due to effort and maintenance. +### How to build a Conan package with new Docker images + +After building our new images, we are ready to build our Conan packages. Let's take Boost as our example. + +{% highlight shell %} + +$ docker run --rm -ti -v ${HOME}/.conan/data:/home/conan/.conan/data conanio/gcc10-ubuntu16.04:1.39.0 +conan@148a77cfbc33:~$ conan install boost/1.76.0@ --build +conan@148a77cfbc33:~$ exit + +{% endhighlight %} + +Here, we start a temporary Docker container with interactive support. Also, we share our Conan cache data as volume. +After starting, we build Boost 1.76.0 and its dependencies from source. All packages will be built and installed to +the shared volume, so we can use it after closing the container. To finish and remove the container, we just need to +exit. + +{% highlight shell %} + +$ docker run -d -t -v ${HOME}/.conan/data:/home/conan/.conan/data --name conan_container conanio/gcc10 +$ docker exec conan_container conan install boost/1.76.0@ --build +$ docker stop conan_container +$ docker rm conan_container + +{% endhighlight %} + +Similar execution, same result. Instead of creating a temporary Docker container, we executed it on background. +All container commands are passed by `conan exec` command. Also, we need to stop and remove manually after finishing. + + ## Finals words and feedback We invite everyone who uses Conan Docker Tools images to use this new formulation, which should become official soon. From 22faeb4caf022e36f79457d957ce781f7d3bee0d Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:31:30 -0300 Subject: [PATCH 18/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 9b4eafea..514b2210 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -96,7 +96,7 @@ prebuilt. So, we chose to build both from sources in order to have more control Also, packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible different distros. -The library ``libstdc++`` is distributed along with the GCC project. We chose to use a single version of the library, +The library ``libstdc++`` is distributed along with the GCC project. The intention was to use a single version of the library, which was neither the newest, to allow older distributions to use, but also not so old, so that new features can be consumed by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also From d548f4e3b1614daa98f500ae60fb938bd2822b6b Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:31:44 -0300 Subject: [PATCH 19/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 514b2210..412f91f5 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -100,9 +100,9 @@ The library ``libstdc++`` is distributed along with the GCC project. The intenti which was neither the newest, to allow older distributions to use, but also not so old, so that new features can be consumed by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also -is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 is built, its ``libstdc++`` generated is copied to an -Artifactory instance and this is downloaded directly to -each final image. Actually, that was the original intent, but as we will see, that was not possible. While we were +is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built it should be possible to copy +this library to the rest of the images. Actually, that was the original intent, but as we will see, that was +not possible. While we were developing the new recipes, GCC 11 was released and along with it, the version was ``libstdc++`` already ``.29``, which made usage incompatible. In this case, we are left with the following dilemma: From 126c6e9a898baf05602fe757cc33480e1c431b8f Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:31:52 -0300 Subject: [PATCH 20/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 412f91f5..c616b1ca 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -107,7 +107,7 @@ developing the new recipes, GCC 11 was released and along with it, the version w made usage incompatible. In this case, we are left with the following dilemma: * Using the same libstdc++ version, except for GCC 11. - * Conan Center becomes homogeneous, everything can be used without compiler restriction and distribution + * Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same ``libstdc++`` version, which guarantees that all can run in any image. * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library that is not yet available in official PPA. All build requirements would be broken. * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of From 6ff47dccbe7279bcb15d090b2e4d7be93d1d9619 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:32:01 -0300 Subject: [PATCH 21/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index c616b1ca..c0b24c45 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -103,8 +103,9 @@ by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same dis is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built it should be possible to copy this library to the rest of the images. Actually, that was the original intent, but as we will see, that was not possible. While we were -developing the new recipes, GCC 11 was released and along with it, the version was ``libstdc++`` already ``.29``, which -made usage incompatible. In this case, we are left with the following dilemma: +developing the new recipes, GCC 11 was released and, with it, a new ``libstdc++`` version (``6.0.29``). +It was not possible to use the previous version with this new compiler. We were left with the following +dilemma: * Using the same libstdc++ version, except for GCC 11. * Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same ``libstdc++`` version, which guarantees that all can run in any image. From 568312aa20da935aa323af23fff5af861c84884b Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:32:09 -0300 Subject: [PATCH 22/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index c0b24c45..36c21fca 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -110,7 +110,7 @@ dilemma: * Using the same libstdc++ version, except for GCC 11. * Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same ``libstdc++`` version, which guarantees that all can run in any image. * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library -that is not yet available in official PPA. All build requirements would be broken. +that is not yet available in the official PPA. All executables built inside ConanCenter won't work in the users' machines. * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of risks. * Each image uses the corresponding version of libstdc++ provided by the compiler. From f858f24a38ef39d6cc547397efb137d2d85f5b29 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:32:16 -0300 Subject: [PATCH 23/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 36c21fca..0ee882ff 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -116,7 +116,7 @@ risks. * Each image uses the corresponding version of libstdc++ provided by the compiler. * Better than the current scenario, where it is dependent on PPA and we have no control over it. * All images still use the same version of glibc, another advantage over the current scenario. - * We will need to take care of the build requirements, as they will only be compatible with later versions. + * We will need to take care of the executables, as they will only be compatible with later versions (as they are now). * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc version is still the same version for everyone. From 1813a304304796d64ca57cd7a8018eb9c4755e3a Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:32:23 -0300 Subject: [PATCH 24/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 0ee882ff..27799cd6 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -118,7 +118,7 @@ risks. * All images still use the same version of glibc, another advantage over the current scenario. * We will need to take care of the executables, as they will only be compatible with later versions (as they are now). * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc -version is still the same version for everyone. +version is the same version for all the packages. Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the compiler**. From 3dfa193057641a0d62a0e9b28abeb327574cef76 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 12:32:45 -0300 Subject: [PATCH 25/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 27799cd6..36185b9c 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -190,7 +190,7 @@ compilation hours. To see the full recipe, it is available [here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). -### Training the Dragon: Clang Dockefile +### Training the Dragon: Building Clang from sources As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` From 36a2d492abfedc150eb3e7f42ad666e33e6a1387 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Fri, 6 Aug 2021 16:17:01 -0300 Subject: [PATCH 26/49] Clang and libstdcplusplus Signed-off-by: Uilian Ries --- .../2021-07-21-Modern-docker-images.markdown | 80 ++++++++++--------- 1 file changed, 41 insertions(+), 39 deletions(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 36185b9c..3f4c5204 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -123,6 +123,10 @@ version is the same version for all the packages. Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the compiler**. +As Clang also supports ``libstdc++``, we choose ``libstdc++.so.6.0.28`` to be its default version. As commented before, +that version has some advantages due its age and compatibility. To copy the library with Clang, first we need to build +GCC 10, then we can mount its Docker container and copy the ``libstdc++.so.6.0.28`` to the Clang image. + Ubuntu 16.04 Xenial LTS is still the base used, its support will be until April 2024. After that date, we will need to update the images to a newer version of the distribution, in addition to rebuilding all available official packages. @@ -135,6 +139,43 @@ To summarize the plan: * Use glibc 2.23 for all new Docker images * Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers. +### Training the Dragon: Building Clang from sources + +As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct +dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` +and using ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` would only +be used for Conan packages, not as a Clang requirement. However, we had some situations and the need for some actions +that will be listed here: + +* The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if +necessary. +* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The +compiler is pre-built and distributed by the official LLVM PPA. +* From version to version, options are added or removed, reflecting the evolution of project features and legacy +deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options +do not work in subsequent versions or which option should be used to specify the preferred library. +* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for +each attempt, a long wait was needed to get the result. +* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. As a solution, the +project supports a configuration file, where ``libc++`` can be specified by default. However, this behavior changes +between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for +all versions. +* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already +internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the +image with Clang and installs Conan's libunwind package? A big mess when linking, is the answer. Clang tries to link +the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original +LLVM ``libunwind`` to ``libllvm-unwind``. + +With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of +discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version +10, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux +environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. + +

+## Part 2: Under the hood of Dockerfiles and technical details + +Here we will be more focused on the final product, Dockerfiles, tests and CI. If you are interested to read about +our decisions, read the Part 1 first. ### From blueprint to prototype: Writing the new Docker recipes @@ -189,45 +230,6 @@ and challenges that made us change our mind. Here we will share a little bit of compilation hours. To see the full recipe, it is available [here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). - -### Training the Dragon: Building Clang from sources - -As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct -dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` -and using ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` would only -be used for Conan packages, not as a Clang requirement. However, we had some situations and the need for some actions -that will be listed here: - -* The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if -necessary. -* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The -compiler is pre-built and distributed by the official LLVM PPA. -* From version to version, options are added or removed, reflecting the evolution of project features and legacy -deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options -do not work in subsequent versions or which option should be used to specify the preferred library. -* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for -each attempt, a long wait was needed to get the result. -* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. As a solution, the -project supports a configuration file, where ``libc++`` can be specified by default. However, this behavior changes -between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for -all versions. -* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already -internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the -image with Clang and installs Conan's libunwind package? A big mess when linking, is the answer. Clang tries to link -the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original -LLVM ``libunwind`` to ``libllvm-unwind``. - -With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of -discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version -10, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux -environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. - -

-## Part 2: Under the hood of Dockerfiles and technical details - -Here we will be more focused on the final product, Dockerfiles, tests and CI. If you are interested to read about -our decisions, read the Part 1 first. - ### Building GCC from source Now let's look at the GCC build image, the full recipe can be found From b37d47183e75494b4864601edca65c5bd0e99219 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 9 Aug 2021 15:06:27 -0300 Subject: [PATCH 27/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Daniel --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 3f4c5204..3fd04c52 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -422,7 +422,7 @@ $ docker rm conan_container {% endhighlight %} Similar execution, same result. Instead of creating a temporary Docker container, we executed it on background. -All container commands are passed by `conan exec` command. Also, we need to stop and remove manually after finishing. +All the Conan commands are passed via `docker exec` command. Also, we need to stop and remove manually after finishing. ## Finals words and feedback From b485734c5f76b870d1897fe90540b7336efc099f Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 9 Aug 2021 15:06:36 -0300 Subject: [PATCH 28/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Daniel --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index 3fd04c52..cbbf00d4 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -375,7 +375,7 @@ With that in mind, we started using Jenkins to also build the new Docker images. advantage in this is the use of features for cache in Docker. Previously, a single job took up to 2h if it was on other services, without the use of caching. Now, using Jenkins and Docker ``--cache-from``, updating a package, from base image to final image, takes just 4 minutes per job. The Jenkinsfile file used can be viewed -[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/.ci/xenial.jenkinsfile). +[here](https://github.com/conan-io/conan-docker-tools/blob/master/.ci/xenial.jenkinsfile). Although the file looks complicated at first glance, it is still possible to use docker-compose to build an image from scratch. As an example, let's use Clang 12: From ffccb782d81c02d934666d26d27f19fea9f1dad2 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 9 Aug 2021 15:06:45 -0300 Subject: [PATCH 29/49] Update _posts/2021-07-21-Modern-docker-images.markdown Co-authored-by: Daniel --- _posts/2021-07-21-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index cbbf00d4..acccd5aa 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -351,7 +351,7 @@ not the internal folders first. To ensure that the images produced met our requirements, we needed to add new tests that cover in addition to what was already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions of installed binaries and user permissions. The content of the new tests can be seen -[here](https://github.com/conan-io/conan-docker-tools/tree/feature/single-image/modern/tests). +[here](https://github.com/conan-io/conan-docker-tools/tree/master/modern/tests). We introduced greater modularization in the tests, dividing the steps into separate scripts, to serve each compiler closer. With the support for Fortran, it was necessary to adapt a test that covered it. Furthermore, many applications From b596af8c2254a80889f6f6043d903b388be736e0 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 9 Aug 2021 15:08:07 -0300 Subject: [PATCH 30/49] Add SEO review Signed-off-by: Uilian Ries --- _posts/2021-07-21-Modern-docker-images.markdown | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-07-21-Modern-docker-images.markdown index acccd5aa..e8e54d98 100644 --- a/_posts/2021-07-21-Modern-docker-images.markdown +++ b/_posts/2021-07-21-Modern-docker-images.markdown @@ -2,6 +2,8 @@ layout: post comments: false title: "Modern Docker Images to build C/C++ projects for ConanCenter: How I Learned to Stop Worrying and Love the Old images" +meta_title: "Modern Docker Images to build C/C++ projects for ConanCenter" +description: "The New Docker images should become the default while building all C/C++ package on Conan Center Index. Fixed glibc and libstdc++ compatibility problems and unified all Docker recipes using multistage-build on Jenkins" meta_description: "The New Docker images which should become the official to build all C/C++ package on Conan Center Index and distribute under Conan Center. They should be adopted by default for all users. Fixed glibc and libstdc++ compatibility problems and unified all Docker recipes using multistage-build on Jenkins" From 45063b488cfdab1d0a78154e8f102e7587281cf9 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Mon, 9 Aug 2021 15:10:22 -0300 Subject: [PATCH 31/49] Update post date Signed-off-by: Uilian Ries --- ...r-images.markdown => 2021-08-09-Modern-docker-images.markdown} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename _posts/{2021-07-21-Modern-docker-images.markdown => 2021-08-09-Modern-docker-images.markdown} (100%) diff --git a/_posts/2021-07-21-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown similarity index 100% rename from _posts/2021-07-21-Modern-docker-images.markdown rename to _posts/2021-08-09-Modern-docker-images.markdown From 612c8567bc4d82d75f1827d5a61a6a0596a04975 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Tue, 10 Aug 2021 18:46:40 -0300 Subject: [PATCH 32/49] Apply Lori's grammar review Signed-off-by: Uilian Ries --- .../2021-08-09-Modern-docker-images.markdown | 221 ++++++------------ 1 file changed, 71 insertions(+), 150 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index e8e54d98..1d84d59b 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -10,104 +10,60 @@ problems and unified all Docker recipes using multistage-build on Jenkins" --- -This is the continuation of the initial [post](https://blog.conan.io/2020/06/17/Conan-Docker-Images.html) about new -Docker images for the Conan Center. -Preferably read the first post to understand the problem, the motivation and the proposed -solution for Conan Docker Tools. In this post we will continue the proposed implementation, -highlight the problems encountered during development and the result of this long journey that -should improve in many aspects the way we use Docker images for Conan. +This is the continuation of the initial post about new [Docker images for the Conan Center](https://blog.conan.io/2020/06/17/Conan-Docker-Images.html) that will help you understand the problem, the motivation and the proposed solution for Conan Docker Tools. In this post we will continue the proposed implementation, highlight the problems encountered during development,and show how this solution improves many aspects of the way we use Docker images for Conan. +Also, this blog is a bit longer than usual and contains explanations about the decisions made, mistakes committed, and details the technical part of working with Dockerfiles. e divided this blog post in 2 parts: -Also, this blog is a bit longer than usual, and contains all explanation about the decisions and mistakes -committed, plus the technical part with Dockerfiles in details. To attend all kind of readers, we divided -this blog post in 2 parts: * Part 1: The long journey, ideas, motivation, mistakes and challenges -* Part 2: Under the hood of Dockerfiles and technical details +* Part 2: Technical details: Under the hood of Dockerfiles and recipes -## The importance of the community +## The Importance of the Community -Right after the presentation of the first proposal of the previous post, we had a great and -important feedback from the community involving Conan. This shows how important these Docker images -are not only for the Conan Center, generating official packages for general distribution, but also -for people who own their personal projects and use the images to validate and build their Conan -recipes, as well as the companies they build packages through sources. +We want to thank the community for all of the great and important feedback we received involving Conan and Docker images. Your feedback lets us know how important these Docker images are not only for the Conan Center by generating official packages for general distribution, but also for people who own their personal projects and use the images to validate and build their Conan recipes, as well as the companies they build packages through sources. -After long feedback and new formulations, we understood that it was necessary to make changes to -the initial concept, we realized that we could not modify the existing images for better -maintenance, given the risk of breaking the behavior between users, so we started from scratch -again, formulating a more balanced version between Conan Center and users. Unfortunately we can't -keep an old version long enough to support older compilers, but we believe this is the price of -progress, we need to sacrifice older versions for better maintenance for newer versions. +After long feedback and new formulations we understood that it was necessary to make changes to the initial concept. We realized that we could not modify the existing images for better maintenance given the risk of breaking the behavior between users, so we started from scratch again, formulating a more balanced version between Conan Center and users. We can't keep an old version long enough to support older compilers, this is the price of progress. To keep moving forward we need to remove older versions for better maintenance for newer versions. -That said, it is clear how important the involvement of the entire community has been and how much -we take into account its manifestation. We are grateful to the great tribe that surrounds Conan and -makes it more and more complete. +We are grateful to the great tribe that surrounds Conan and helps make it more and more complete. We would not be where we are without your support, feedback, and trust in our ability to continue to improve Conan.

-## Part 1: The long journey, ideas, motivation, mistakes and challenges +## Part 1: The Long Journey: Ideas, Motivation, Mistakes and Challenges -On this section we will describe how we revisited the initial proposal, found problems, proposed solutions and started a -new entire idea. If you are more interested reading only the more technical part, start from the Part 2. +In this section we will describe how we revisited the initial proposal, found problems, proposed solutions and embarked on an entirely new idea. If you are more interested in the technical part of our journey please skip ahead to Part 2. ### Conan Docker Images: Revisited The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some weak points to consider: -* Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to -build GCC. +* Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to build GCC. * APT packages that were requirements for GCC and Clang were still present, further inflating the final image. -* The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This -affects the reproducibility requirement. -* Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was -necessary to update the PPA address and build the images again -* The continuous integration service used, although it was considerably fast, it was not possible to customize and -prioritize the build, if necessary. - -Noticing the listed issues, we took a more radical solution, even though it took more time to implement, but resulted -in something better in terms of maintainability and practicality. - -Therefore, we decided to abandon PR #204 and start again from scratch, considering the items listed above. - -### A new plan: Using the same base image - -Before we start implementing new dockerfiles with their proper corrections, we first need to understand what the objective -behind it all is. We would like to understand what the expectation is for these new images 2 years from now. As we have -seen before, in a few years with the current approach we are already maintaining over 40 different dockerfiles. Also, one of the points discussed is whether there will be -rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. - -For current and already available images on hub.docker, these will be kept but no longer supported (new versions will -not be introduced). Your recipes and images will remain available, as their immediate removal would result in -catastrophic failure for many users. They will eventually be removed, but at a distant date and to be defined. - -As for the new images, these will be adopted as official and widely promoted for use by all users. The transition -to the new images in Conan Center should take place in a few months after release, because it will be necessary to rebuild the packages that already -exist in the Conan Center and replace them, this is to ensure full compatibility between packages. As for rotation and -maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and -maintenance load for old images and packages that are not always used by the community. Therefore, the following rule -will be adopted: +* The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This affects the reproducibility requirement. +* Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was necessary to update the PPA address and build the images again +* The continuous integration service used, although it was considerably fast, it was not possible to customize and prioritize the build, if necessary. + +Noticing the listed issues, we tried a radical solution. This solution took more time to implement but resulted in something better in terms of maintainability and practicality. + +This being the case we decided to abandon PR #204 and start again from scratch, considering the items listed above. + +### A New Plan: Using the Same Base Image + +We first need to understand the objective behind the project before we start implementing new dockerfiles with their proper corrections. We want to understand what the expectation is for these new images 2 years from now. With the current approach we are already maintaining over 40 different dockerfiles which can be problematic. We also want to address one of the points discussed previously: will there be rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. + +#### The Plan + +For current and already available images on hub.docker these will be kept but no longer supported (new versions will not be introduced). Your recipes and images will remain available, as their immediate removal would result in catastrophic failure for many users. They will eventually be removed, but at a distant and yet to be defined date. + +As for the new images, these will be adopted as official and widely promoted for use by all users. The transition to the new images in Conan Center should take place in a few months after release because it will be necessary to rebuild the packages that already exist in the Conan Center and replace them to ensure full compatibility between packages. As for rotation and maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule will be adopted: + * Clang will be supported from 10.0 to the newest version. * GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. -* For both compilers we will keep updating all new compiler versions and Conan client version, according new releases. +* For both compilers we will keep updating all new compiler versions and Conan client version, according to new releases. * The multilib support was discarded as we are only interested in producing packages with 64-bit support. * Fortran support has been added, thus producing ``gfortran`` together in the image. Currently the Conan package for ``gfortran`` is totally broken and has a complex dependencies chain to fix. -One of the problems we would like to solve is the compiler used and its libraries, we always wanted to be independent -of the Linux distribution. In the initial pull request, we built the GCC from sources, while the Clang uses -prebuilt. So, we chose to build both from sources in order to have more control over the compiler used. -Also, packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible -different distros. - -The library ``libstdc++`` is distributed along with the GCC project. The intention was to use a single version of the library, -which -was neither the newest, to allow older distributions to use, but also not so old, so that new features can be consumed -by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also -is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built it should be possible to copy -this library to the rest of the images. Actually, that was the original intent, but as we will see, that was -not possible. While we were -developing the new recipes, GCC 11 was released and, with it, a new ``libstdc++`` version (``6.0.29``). -It was not possible to use the previous version with this new compiler. We were left with the following -dilemma: +We always wanted to be independent of the Linux distribution so one of the problems we would like to solve is the compiler used and its libraries . In the initial pull request, we built the GCC from sources, while the Clang uses prebuilt. To solve this problem we chose to build both from sources in order to have more control over the compiler used. Packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible different distros. + +The library ``libstdc++`` is distributed along with the GCC project. The intention was to use a single recent version of the library. This decision would allow older distributions to continue to use the library while also allowing new features to be consumed by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built we presumed it would be possible to copy this library to the rest of the images. That was the original intent but as we soon learned it was not possible. While we were developing the new recipes GCC 11 was released and with it a new ``libstdc++`` version (``6.0.29``). It was not possible to use the previous version with this new compiler. We were left with the following dilemma: * Using the same libstdc++ version, except for GCC 11. * Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same ``libstdc++`` version, which guarantees that all can run in any image. @@ -122,18 +78,15 @@ risks. * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc version is the same version for all the packages. -Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the -compiler**. +Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the compiler**. -As Clang also supports ``libstdc++``, we choose ``libstdc++.so.6.0.28`` to be its default version. As commented before, -that version has some advantages due its age and compatibility. To copy the library with Clang, first we need to build -GCC 10, then we can mount its Docker container and copy the ``libstdc++.so.6.0.28`` to the Clang image. +As Clang also supports ``libstdc++`` we choose ``libstdc++.so.6.0.28`` to be its default version. As commented before, that version has some advantages due its age and compatibility. To copy the library with Clang, first we needed to build GCC 10, then we could mount its Docker container and copy the ``libstdc++.so.6.0.28`` to the Clang image. -Ubuntu 16.04 Xenial LTS is still the base used, its support will be until April 2024. After that date, we will need to -update the images to a newer version of the distribution, in addition to rebuilding all available official packages. +Ubuntu 16.04 Xenial LTS is still the base used and it will be supported until April 2024. After that date we will need to update the images to a newer version of the distribution in addition to rebuilding all available official packages. -So forward thinking is having fewer images, but better support, without the drastic breakage and incompatibility issues. -To summarize the plan: +To summarize: forward thinking is having fewer images, but better support without the drastic breakage and incompatibility issues. + +The Revised Plan: * Ubuntu 16.04 LTS as base Docker image * Build Clang and GCC from source @@ -141,13 +94,11 @@ To summarize the plan: * Use glibc 2.23 for all new Docker images * Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers. -### Training the Dragon: Building Clang from sources +### Training the Dragon: Building Clang from Sources + +We want to use only one version of ``libstdc++`` so we found a way to build the Clang without the direct dependency on GCC. By building the Clang with another Clang already installed it avoided ``libgcc_s``, ``libstdc++`` and used ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` will only be used for Conan packages - not as a Clang requirement. -As we would like to use only one version of libstdc++, we chose to find a way to build the Clang without the direct -dependency on GCC, building the Clang with another Clang already installed, thus avoiding ``libgcc_s``, ``libstdc++`` -and using ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` would only -be used for Conan packages, not as a Clang requirement. However, we had some situations and the need for some actions -that will be listed here: +Challenges & Action Items: * The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if necessary. @@ -168,30 +119,23 @@ image with Clang and installs Conan's libunwind package? A big mess when linking the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original LLVM ``libunwind`` to ``libllvm-unwind``. -With all the advents and limitations, it became quite difficult to maintain from Clang 6 to 12. After a lot of -discussions and advices from some of the LLVM maintainers, we decided to limit Clang support to starting from version -10, because it is not necessary to apply as many modifications, including the configuration file. Also, in the Linux -environment, Clang is not the primary compiler, so we believe its use is always tied to newer versions. +It became quite difficult to maintain from Clang 6 to 12 with the discovery of these issues and limitations.After a lot of discussions and advice from some of the LLVM maintainers we decided to limit Clang support to starting from version 10 because it is not necessary to apply as many modifications including the configuration file. Also, in the Linux environment Clang is not the primary compiler so we believe its use is always tied to newer versions.

-## Part 2: Under the hood of Dockerfiles and technical details +## Part 2: Under the Hood of Dockerfiles and Technical Details -Here we will be more focused on the final product, Dockerfiles, tests and CI. If you are interested to read about -our decisions, read the Part 1 first. +Here we will be more focused on the final product, Dockerfiles, tests and CI. -### From blueprint to prototype: Writing the new Docker recipes +### From Blueprint to Prototype: Writing the New Docker Recipes -During prototyping, we realized that we could divide the process of building a Docker image into 3 phases: -* The base, where all common packages are installed to all images, such as Python, git, svn, etc, in addition to the -non-root user configuration. -* An image where only the compiler is built. In this container can be installed packages referring to the compiler build -only, which will not be present in the final image, for example, Ninja, which is used for LLVM. -* Finally, we need to merge the base to the produced compiler into a single image, without adding extra packages, but -still reusable between each compiler version. +During prototyping we realized that we could divide the process of building a Docker image into 3 phases: -For the case of the base image, this one is still quite modular, just changing the variables file to update the package -to be installed. The complete recipe can be obtained -[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/base/Dockerfile), but let's look +* Phase 1:The base where all common packages are installed to all images such as Python, git, svn, etc, in addition to the non-root user configuration. +* Phase 2: An image where only the compiler is built. In this container can be installed packages referring to the compiler build only, which will not be present in the final image, for example, Ninja, which is used for LLVM. +* Phase 3: Merge the base to the produced compiler into a single image without adding extra packages but still reusable between each compiler version. + +For the case of the base image, this one is still quite modular, just changing the variables file to update the package to be installed. The complete recipe can be obtained +[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/base/Dockerfile), but let's look at a few pieces: {% highlight docker %} @@ -230,12 +174,12 @@ These are the main features of the base image, which is used in all final images For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and compilation hours. To see the full recipe, it is available -[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). +[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/clang/Dockerfile). -### Building GCC from source +### Building GCC from Source Now let's look at the GCC build image, the full recipe can be found -[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/gcc/Dockerfile), but let's +[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/gcc/Dockerfile), but let's highlight a few points: {% highlight docker %} @@ -247,10 +191,10 @@ RUN cd gcc-${GCC_VERSION} \ No matter the version, GCC continues to use the same lines for its build. Some factors were configured in this version used: + * Bootstrap has been disabled to reduce build time to just 20 minutes. * Fortran is enabled, but it barely increase the building time and final - The last part of the image uses the concept of Docker [multistage-build](https://docs.docker.com/develop/develop-images/multistage-build/), a technique that avoids creating separate recipes and images to take advantage of common parts. @@ -282,17 +226,12 @@ RUN sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ {% endhighlight %} -In this section, the base image used is the same one we created before, through a caching mechanism, this drastically -reduces the final image build time. -Also, all artifacts generated from GCC are now copied to their respective locations. Finally, the compiler becomes the -default in the image and the libraries are listed and cached. And here's the icing on the cake, the same recipe works -from GCC 5 to the latest version, just modifying some arguments. The maintenance has been drastically simplified -compared to current Conan Docker Tools. +In this section the base image used is the same one we created before through a caching mechanism that drastically reduces the final image build time. All artifacts generated from GCC are now copied to their respective locations. Finally, the compiler becomes the default in the image and the libraries are listed and cached. And here's the icing on the cake, the same recipe works from GCC 5 to the latest version. You just need to modify some arguments. The maintenance has been drastically simplified compared to current Conan Docker Tools. -### Conan meets the Wyvern: Building Clang C/C++ compiler from source +### Conan Meets the Wyvern: Building Clang C/C++ Compiler from Source To see the full recipe, it is available -[here](https://github.com/conan-io/conan-docker-tools/blob/feature/single-image/modern/clang/Dockerfile). +[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/clang/Dockerfile). Let's go a step further and detail the Clang deployment step. @@ -327,7 +266,7 @@ Similar to what was done with GCC, in Clang we also use the same base image and compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan. -Besides that, Clang requires some interesting CMake definitions: +In addition Clang requires some interesting CMake definitions: * LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build * LLVM_USE_LINKER: We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time @@ -344,20 +283,15 @@ Besides that, Clang requires some interesting CMake definitions: If we run `ninja` command alone, it builds more projects than we want configured as enabled, so we build one by one. -Also, `libcxx` has a limitation when building using `libc++abi`, it searches on system library folder, +Also, `libcxx` has a limitation when building using `libc++abi`, it searches on the system library folder, not the internal folders first. ### Tests and more tests: A CI pipeline to test Docker images -To ensure that the images produced met our requirements, we needed to add new tests that cover in addition to what was -already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions -of installed binaries and user permissions. The content of the new tests can be seen -[here](https://github.com/conan-io/conan-docker-tools/tree/master/modern/tests). +To ensure that the images produced met our requirements we needed to add new tests that cover in addition to what was already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions of installed binaries, and user permissions. The content of the new tests can be seen [here](https://github.com/conan-io/conan-docker-tools/tree/master/modern/tests). -We introduced greater modularization in the tests, dividing the steps into separate scripts, to serve each compiler -closer. With the support for Fortran, it was necessary to adapt a test that covered it. Furthermore, many applications -that are now bundled with Conan are no longer part of the base image and this has also been validated. +We introduced greater modularization in the tests, dividing the steps into separate scripts to serve each compiler closer. With the support for Fortran it was necessary to adapt a test that covered it. Furthermore, many applications that are now bundled with Conan are no longer part of the base image and this has also been validated. If you want to test locally a produced Docker image, you can easily run: @@ -367,20 +301,14 @@ $ cd modern && pytest tests --image conanio/gcc10-ubuntu16.04 --service deploy - {% endhighlight %} -### The CI service change: From Travis to Azure and the Jenkins arrival +### The CI Service Change: From Travis to Azure and the Jenkins Arrival -Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, -this did not give us the full power to prioritize the build in the queue, or customize the host, or customize the build -lines to use Docker-in-Docker if necessary. +Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, this did not give us the full power to prioritize the build in the queue, customize the host, or customize the build lines to use Docker-in-Docker if necessary. -With that in mind, we started using Jenkins to also build the new Docker images. The big -advantage in this is the use of features for cache in Docker. Previously, a single job took up to 2h if it was on other -services, without the use of caching. Now, using Jenkins and Docker ``--cache-from``, updating a package, from base -image to final image, takes just 4 minutes per job. The Jenkinsfile file used can be viewed +With that in mind we started using Jenkins to also build the new Docker images. The big advantage in this is the use of features for cache in Docker. Previously a single job took up to 2 hours if it was on other services without the use of caching. Now using Jenkins and Docker ``--cache-from``, updating a package from base image to final image takes just 4 minutes per job. The Jenkinsfile file used can be viewed [here](https://github.com/conan-io/conan-docker-tools/blob/master/.ci/xenial.jenkinsfile). -Although the file looks complicated at first glance, it is still possible to use docker-compose to build an image from -scratch. As an example, let's use Clang 12: +Although the file looks complicated at first glance, it is still possible to use docker-compose to build an image from scratch. As an example, let's use Clang 12: {% highlight bash %} @@ -409,10 +337,7 @@ conan@148a77cfbc33:~$ exit {% endhighlight %} -Here, we start a temporary Docker container with interactive support. Also, we share our Conan cache data as volume. -After starting, we build Boost 1.76.0 and its dependencies from source. All packages will be built and installed to -the shared volume, so we can use it after closing the container. To finish and remove the container, we just need to -exit. +Here, we start a temporary Docker container with interactive support. Also, we share our Conan cache data as volume. After starting, we build Boost 1.76.0 and its dependencies from source. All packages will be built and installed to the shared volume, so we can use it after closing the container. To finish and remove the container, we just need to exit. {% highlight shell %} @@ -423,13 +348,9 @@ $ docker rm conan_container {% endhighlight %} -Similar execution, same result. Instead of creating a temporary Docker container, we executed it on background. -All the Conan commands are passed via `docker exec` command. Also, we need to stop and remove manually after finishing. +Similar execution, same result. Instead of creating a temporary Docker container, we executed it in the background. All container commands are passed by ``conan exec`` command. Also, we need to stop and remove manually after finishing. ## Finals words and feedback -We invite everyone who uses Conan Docker Tools images to use this new formulation, which should become official soon. -These new images are the result of a long journey, where we learned from our mistakes and listened to the community to -reach what we have today. We believe improvements should be added continuously to keep the CDT progressing, so please -let us your feedback on issue [#205](https://github.com/conan-io/conan-docker-tools/issues/205). +We invite everyone who uses Conan Docker Tools images to use this new formulation, which should become official soon. These new images are the result of a long journey where we learned from our mistakes and listened to the community to achieve what we have today. We believe improvements should be added continuously to keep the CDT progressing so please let us your feedback on issue [#205](https://github.com/conan-io/conan-docker-tools/issues/205). From a9d0de08ecff6e0d16807fe5f15e2920a07f0cc1 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Tue, 10 Aug 2021 19:00:06 -0300 Subject: [PATCH 33/49] Format markdown file Signed-off-by: Uilian Ries --- .../2021-08-09-Modern-docker-images.markdown | 426 +++++++++++------- 1 file changed, 260 insertions(+), 166 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 1d84d59b..47266be6 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -3,88 +3,157 @@ layout: post comments: false title: "Modern Docker Images to build C/C++ projects for ConanCenter: How I Learned to Stop Worrying and Love the Old images" meta_title: "Modern Docker Images to build C/C++ projects for ConanCenter" -description: "The New Docker images should become the default while building all C/C++ package on Conan Center Index. Fixed glibc and libstdc++ compatibility problems and unified all Docker recipes using multistage-build on Jenkins" -meta_description: "The New Docker images which should become the official to build all C/C++ package on Conan Center Index -and distribute under Conan Center. They should be adopted by default for all users. Fixed glibc and libstdc++ compatibility -problems and unified all Docker recipes using multistage-build on Jenkins" +description: "The New Docker images should become the default while building all C/C++ package on Conan Center Index. Fixed glibc and libstdc++ +compatibility problems and unified all Docker recipes using multistage-build on Jenkins" +meta_description: "The New Docker images which should become the official to build all C/C++ +package on Conan Center Index and distribute under Conan Center. They should be adopted by +default for all users. Fixed glibc and libstdc++ compatibility problems and unified all Docker +recipes using multistage-build on Jenkins" --- -This is the continuation of the initial post about new [Docker images for the Conan Center](https://blog.conan.io/2020/06/17/Conan-Docker-Images.html) that will help you understand the problem, the motivation and the proposed solution for Conan Docker Tools. In this post we will continue the proposed implementation, highlight the problems encountered during development,and show how this solution improves many aspects of the way we use Docker images for Conan. -Also, this blog is a bit longer than usual and contains explanations about the decisions made, mistakes committed, and details the technical part of working with Dockerfiles. e divided this blog post in 2 parts: - +This is the continuation of the initial post about new +[Docker images for the Conan Center](https://blog.conan.io/2020/06/17/Conan-Docker-Images.html) +that will help you understand +the problem, the motivation and the proposed solution for Conan Docker Tools. In this post we will +continue the proposed implementation, highlight the problems encountered during development,and +show how this solution improves many aspects of the way we use Docker images for Conan. Also, +this blog is a bit longer than usual and contains explanations about the decisions made, mistakes +committed, and details the technical part of working with Dockerfiles. e divided this blog post +in 2 parts: * Part 1: The long journey, ideas, motivation, mistakes and challenges * Part 2: Technical details: Under the hood of Dockerfiles and recipes ## The Importance of the Community -We want to thank the community for all of the great and important feedback we received involving Conan and Docker images. Your feedback lets us know how important these Docker images are not only for the Conan Center by generating official packages for general distribution, but also for people who own their personal projects and use the images to validate and build their Conan recipes, as well as the companies they build packages through sources. +We want to thank the community for all of the great and important feedback we received involving +Conan and Docker images. Your feedback lets us know how important these Docker images are not +only for the Conan Center by generating official packages for general distribution, but also +for people who own their personal projects and use the images to validate and build their Conan +recipes, as well as the companies they build packages through sources. -After long feedback and new formulations we understood that it was necessary to make changes to the initial concept. We realized that we could not modify the existing images for better maintenance given the risk of breaking the behavior between users, so we started from scratch again, formulating a more balanced version between Conan Center and users. We can't keep an old version long enough to support older compilers, this is the price of progress. To keep moving forward we need to remove older versions for better maintenance for newer versions. +After long feedback and new formulations we understood that it was necessary to make changes +to the initial concept. We realized that we could not modify the existing images for better +maintenance given the risk of breaking the behavior between users, so we started from scratch +again, formulating a more balanced version between Conan Center and users. We can't keep an old +version long enough to support older compilers, this is the price of progress. To keep moving +forward we need to remove older versions for better maintenance for newer versions. -We are grateful to the great tribe that surrounds Conan and helps make it more and more complete. We would not be where we are without your support, feedback, and trust in our ability to continue to improve Conan. +We are grateful to the great tribe that surrounds Conan and helps make it more and more +complete. We would not be where we are without your support, feedback, and trust in our ability +to continue to improve Conan.

## Part 1: The Long Journey: Ideas, Motivation, Mistakes and Challenges -In this section we will describe how we revisited the initial proposal, found problems, proposed solutions and embarked on an entirely new idea. If you are more interested in the technical part of our journey please skip ahead to Part 2. +In this section we will describe how we revisited the initial proposal, found problems, proposed +solutions and embarked on an entirely new idea. If you are more interested in the technical +part of our journey please skip ahead to Part 2. ### Conan Docker Images: Revisited -The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some weak points to consider: +The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some +weak points to consider: -* Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to build GCC. -* APT packages that were requirements for GCC and Clang were still present, further inflating the final image. -* The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This affects the reproducibility requirement. -* Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was necessary to update the PPA address and build the images again -* The continuous integration service used, although it was considerably fast, it was not possible to customize and prioritize the build, if necessary. +* Installing Clang via LLVM's APT repository does not guarantee full compatibility with +``libstdc++`` version used to build GCC. * APT packages that were requirements for GCC and +Clang were still present, further inflating the final image. * The packages provided through +Ubuntu do not have older versions available, in case a new version comes along. This affects +the reproducibility requirement. * Older compilers were always an issue in terms of maintenance +once they arrived in the EOL (end-of-life) state. It was necessary to update the PPA address and +build the images again * The continuous integration service used, although it was considerably +fast, it was not possible to customize and prioritize the build, if necessary. -Noticing the listed issues, we tried a radical solution. This solution took more time to implement but resulted in something better in terms of maintainability and practicality. +Noticing the listed issues, we tried a radical solution. This solution took more time to +implement but resulted in something better in terms of maintainability and practicality. -This being the case we decided to abandon PR #204 and start again from scratch, considering the items listed above. +This being the case we decided to abandon PR #204 and start again from scratch, considering +the items listed above. ### A New Plan: Using the Same Base Image -We first need to understand the objective behind the project before we start implementing new dockerfiles with their proper corrections. We want to understand what the expectation is for these new images 2 years from now. With the current approach we are already maintaining over 40 different dockerfiles which can be problematic. We also want to address one of the points discussed previously: will there be rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. +We first need to understand the objective behind the project before we start implementing new +dockerfiles with their proper corrections. We want to understand what the expectation is for +these new images 2 years from now. With the current approach we are already maintaining over +40 different dockerfiles which can be problematic. We also want to address one of the points +discussed previously: will there be rotation to avoid the accumulation of old images and +their restrictions in terms of maintenance. #### The Plan -For current and already available images on hub.docker these will be kept but no longer supported (new versions will not be introduced). Your recipes and images will remain available, as their immediate removal would result in catastrophic failure for many users. They will eventually be removed, but at a distant and yet to be defined date. +For current and already available images on hub.docker these will be kept but no longer supported +(new versions will not be introduced). Your recipes and images will remain available, as their +immediate removal would result in catastrophic failure for many users. They will eventually be +removed, but at a distant and yet to be defined date. -As for the new images, these will be adopted as official and widely promoted for use by all users. The transition to the new images in Conan Center should take place in a few months after release because it will be necessary to rebuild the packages that already exist in the Conan Center and replace them to ensure full compatibility between packages. As for rotation and maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a large build, effort and maintenance load for old images and packages that are not always used by the community. Therefore, the following rule will be adopted: +As for the new images, these will be adopted as official and widely promoted for use by all +users. The transition to the new images in Conan Center should take place in a few months after +release because it will be necessary to rebuild the packages that already exist in the Conan +Center and replace them to ensure full compatibility between packages. As for rotation and +maintenance, we believe it is necessary to rotate supported compilers over time, to avoid a +large build, effort and maintenance load for old images and packages that are not always used +by the community. Therefore, the following rule will be adopted: * Clang will be supported from 10.0 to the newest version. * GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. -* For both compilers we will keep updating all new compiler versions and Conan client version, according to new releases. -* The multilib support was discarded as we are only interested in producing packages with 64-bit support. -* Fortran support has been added, thus producing ``gfortran`` together in the image. Currently the Conan package for -``gfortran`` is totally broken and has a complex dependencies chain to fix. - -We always wanted to be independent of the Linux distribution so one of the problems we would like to solve is the compiler used and its libraries . In the initial pull request, we built the GCC from sources, while the Clang uses prebuilt. To solve this problem we chose to build both from sources in order to have more control over the compiler used. Packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible different distros. - -The library ``libstdc++`` is distributed along with the GCC project. The intention was to use a single recent version of the library. This decision would allow older distributions to continue to use the library while also allowing new features to be consumed by newer compilers. The version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also is the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built we presumed it would be possible to copy this library to the rest of the images. That was the original intent but as we soon learned it was not possible. While we were developing the new recipes GCC 11 was released and with it a new ``libstdc++`` version (``6.0.29``). It was not possible to use the previous version with this new compiler. We were left with the following dilemma: +* For both compilers we will keep updating all new compiler versions and Conan client version, +according to new releases. +* The multilib support was discarded as we are only interested in producing packages with +64-bit support. +* Fortran support has been added, thus producing ``gfortran`` together in +the image. Currently the Conan package for ``gfortran`` is totally broken and has a complex +dependencies chain to fix. + +We always wanted to be independent of the Linux distribution so one of the problems we would like +to solve is the compiler used and its libraries . In the initial pull request, we built the GCC +from sources, while the Clang uses prebuilt. To solve this problem we chose to build both from +sources in order to have more control over the compiler used. Packages generated using these images +(packages in ConanCenter) should work out-of-the-box in as many as possible different distros. + +The library ``libstdc++`` is distributed along with the GCC project. The intention was to use a +single recent version of the library. This decision would allow older distributions to continue +to use the library while also allowing new features to be consumed by newer compilers. The +version chosen was ``libstdc++.so.6.0.28``, the same distributed with GCC 9 and 10, but also is +the default version in Ubuntu 20.04 LTS (Focal). Once GCC 10 was built we presumed it would be +possible to copy this library to the rest of the images. That was the original intent but as we +soon learned it was not possible. While we were developing the new recipes GCC 11 was released +and with it a new ``libstdc++`` version (``6.0.29``). It was not possible to use the previous +version with this new compiler. We were left with the following dilemma: * Using the same libstdc++ version, except for GCC 11. - * Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same ``libstdc++`` version, which guarantees that all can run in any image. - * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library -that is not yet available in the official PPA. All executables built inside ConanCenter won't work in the users' machines. - * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of -risks. +* Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked +using the same ``libstdc++`` version, which guarantees that all can run in any image. +* Binaries can hardly be used outside of Conan Center because they need the newest version +of libstdc++ library that is not yet available in the official PPA. All executables built inside +ConanCenter won't work in the users' machines. +* A possible solution would be to statically link libstdc++ in all binaries, but this +solution has a number of risks. * Each image uses the corresponding version of libstdc++ provided by the compiler. - * Better than the current scenario, where it is dependent on PPA and we have no control over it. - * All images still use the same version of glibc, another advantage over the current scenario. - * We will need to take care of the executables, as they will only be compatible with later versions (as they are now). - * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc +* Better than the current scenario, where it is dependent on PPA and we have no control +over it. +* All images still use the same version of glibc, another advantage over the +current scenario. +* We will need to take care of the executables, as they will only be +compatible with later versions (as they are now). +* Better for users, than the current scenario. +The requirements related to libstdc++ are the same, but the glibc version is the same version for all the packages. -Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version available together with the compiler**. +Given the conditions and risks, we chose to go the second way: **Use the libstdc++ version +available together with the compiler**. -As Clang also supports ``libstdc++`` we choose ``libstdc++.so.6.0.28`` to be its default version. As commented before, that version has some advantages due its age and compatibility. To copy the library with Clang, first we needed to build GCC 10, then we could mount its Docker container and copy the ``libstdc++.so.6.0.28`` to the Clang image. +As Clang also supports ``libstdc++`` we choose ``libstdc++.so.6.0.28`` to be its default +version. As commented before, that version has some advantages due its age and compatibility. To +copy the library with Clang, first we needed to build GCC 10, then we could mount its Docker +container and copy the ``libstdc++.so.6.0.28`` to the Clang image. -Ubuntu 16.04 Xenial LTS is still the base used and it will be supported until April 2024. After that date we will need to update the images to a newer version of the distribution in addition to rebuilding all available official packages. +Ubuntu 16.04 Xenial LTS is still the base used and it will be supported until April 2024. After +that date we will need to update the images to a newer version of the distribution in addition +to rebuilding all available official packages. -To summarize: forward thinking is having fewer images, but better support without the drastic breakage and incompatibility issues. +To summarize: forward thinking is having fewer images, but better support without the drastic +breakage and incompatibility issues. The Revised Plan: @@ -92,34 +161,45 @@ The Revised Plan: * Build Clang and GCC from source * Use libstdc++ provided by the compiler * Use glibc 2.23 for all new Docker images -* Images for old compilers will be built as long as their build script is compatible with the one for the newer compilers. +* Images for old compilers will be built as long as their build script is compatible with the +one for the newer compilers. ### Training the Dragon: Building Clang from Sources -We want to use only one version of ``libstdc++`` so we found a way to build the Clang without the direct dependency on GCC. By building the Clang with another Clang already installed it avoided ``libgcc_s``, ``libstdc++`` and used ``libc++``, ``libc++-abi``, ``libunwind``, ``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` will only be used for Conan packages - not as a Clang requirement. +We want to use only one version of ``libstdc++`` so we found a way to build the Clang without +the direct dependency on GCC. By building the Clang with another Clang already installed +it avoided ``libgcc_s``, ``libstdc++`` and used ``libc++``, ``libc++-abi``, ``libunwind``, +``compiler-rt`` and ``ldd`` instead. The ``libstdc++`` will only be used for Conan packages - +not as a Clang requirement. Challenges & Action Items: -* The LLVM project uses CMake support, which facilitates the configuration of its construction, even customization if -necessary. -* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen Ubuntu version. The -compiler is pre-built and distributed by the official LLVM PPA. -* From version to version, options are added or removed, reflecting the evolution of project features and legacy -deprecation. With these changes, it was inevitable to study the CMake files of each version to understand which options -do not work in subsequent versions or which option should be used to specify the preferred library. -* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending on the host. So, for -each attempt, a long wait was needed to get the result. -* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. As a solution, the -project supports a configuration file, where ``libc++`` can be specified by default. However, this behavior changes -between versions 6, 7 and 8, requiring different standards and making it difficult to use the same Docker recipe for -all versions. -* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already -internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the -image with Clang and installs Conan's libunwind package? A big mess when linking, is the answer. Clang tries to link -the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original -LLVM ``libunwind`` to ``libllvm-unwind``. - -It became quite difficult to maintain from Clang 6 to 12 with the discovery of these issues and limitations.After a lot of discussions and advice from some of the LLVM maintainers we decided to limit Clang support to starting from version 10 because it is not necessary to apply as many modifications including the configuration file. Also, in the Linux environment Clang is not the primary compiler so we believe its use is always tied to newer versions. +* The LLVM project uses CMake support, which facilitates the configuration of its construction, +even customization if necessary. +* We chose to use Clang 10 as a builder, as it is current and still compatible with the chosen +Ubuntu version. The compiler is pre-built and distributed by the official LLVM PPA. +* From version to version, options are added or removed, reflecting the evolution of project +features and legacy deprecation. With these changes, it was inevitable to study the CMake files +of each version to understand which options do not work in subsequent versions or which option +should be used to specify the preferred library. +* Unlike GCC, LLVM has a huge range of parameters and a longer build time, around 1h depending +on the host. So, for each attempt, a long wait was needed to get the result. +* Until Clang 9 release, ``libc++`` was not automatically added to be linked when using Clang. +As a solution, the project supports a configuration file, where ``libc++`` can be specified by +default. However, this behavior changes between versions 6, 7 and 8, requiring different standards +and making it difficult to use the same Docker recipe for all versions. +* With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. +It is already internalized in LLVM, but used as a dynamic library only. So a question arises, what +happens if a project uses the image with Clang and installs Conan's libunwind package? A big mess +when linking, is the answer. Clang tries to link the version distributed by the Conan package, +resulting in several errors. As a workaround, we renamed the original LLVM +``libunwind`` to ``libllvm-unwind``. + +It became quite difficult to maintain from Clang 6 to 12 with the discovery of these issues and +limitations.After a lot of discussions and advice from some of the LLVM maintainers we decided +to limit Clang support to starting from version 10 because it is not necessary to apply as +many modifications including the configuration file. Also, in the Linux environment Clang is +not the primary compiler so we believe its use is always tied to newer versions.

## Part 2: Under the Hood of Dockerfiles and Technical Details @@ -128,59 +208,61 @@ Here we will be more focused on the final product, Dockerfiles, tests and CI. ### From Blueprint to Prototype: Writing the New Docker Recipes -During prototyping we realized that we could divide the process of building a Docker image into 3 phases: +During prototyping we realized that we could divide the process of building a Docker image into +3 phases: -* Phase 1:The base where all common packages are installed to all images such as Python, git, svn, etc, in addition to the non-root user configuration. -* Phase 2: An image where only the compiler is built. In this container can be installed packages referring to the compiler build only, which will not be present in the final image, for example, Ninja, which is used for LLVM. -* Phase 3: Merge the base to the produced compiler into a single image without adding extra packages but still reusable between each compiler version. +* Phase 1:The base where all common packages are installed to all images such as Python, git, +svn, etc, in addition to the non-root user configuration. +* Phase 2: An image where only the +compiler is built. In this container can be installed packages referring to the compiler build +only, which will not be present in the final image, for example, Ninja, which is used for LLVM. +* Phase 3: Merge the base to the produced compiler into a single image without adding extra +packages but still reusable between each compiler version. -For the case of the base image, this one is still quite modular, just changing the variables file to update the package to be installed. The complete recipe can be obtained -[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/base/Dockerfile), but let's look -at a few pieces: +For the case of the base image, this one is still quite modular, just changing the +variables file to update the package to be installed. The complete recipe can be obtained +[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/base/Dockerfile), +but let's look at a few pieces: {% highlight docker %} -ARG DISTRO_VERSION -FROM ubuntu:${DISTRO_VERSION} +ARG DISTRO_VERSION FROM ubuntu:${DISTRO_VERSION} ENV PYENV_ROOT=/opt/pyenv \ PATH=/opt/pyenv/shims:${PATH} -ARG CMAKE_VERSION -ARG CMAKE_VERSION_FULL -ARG PYTHON_VERSION -ARG CONAN_VERSION +ARG CMAKE_VERSION ARG CMAKE_VERSION_FULL ARG PYTHON_VERSION ARG CONAN_VERSION {% endhighlight %} -Now, both the distribution version and the installed packages are configurable in terms of version used. Previously, a -script was used to update all 42 recipes as needed! +Now, both the distribution version and the installed packages are configurable in terms of +version used. Previously, a script was used to update all 42 recipes as needed! {% highlight docker %} RUN printf '/usr/local/lib64\n' >> /etc/ld.so.conf.d/20local-lib.conf \ - && printf '/usr/local/lib\n' > /etc/ld.so .conf.d/20local-lib.conf \ - ... + && printf '/usr/local/lib\n' > /etc/ld.so .conf.d/20local-lib.conf \ ... {% endhighlight %} -In order to not be affected by system packages or Conan packages that invoke ``apt-get``, the compiler and its -artifacts are installed in ``/usr/local``. However, this is not enough to prioritize the order ``libstdc++`` used, for -that we need to update ``ldconfig`` with the local directories. Until then, this was not necessary in the previous -images, as everything was either consumed directly from the system, or installed directly in ``/usr``. +In order to not be affected by system packages or Conan packages that invoke ``apt-get``, +the compiler and its artifacts are installed in ``/usr/local``. However, this is not enough +to prioritize the order ``libstdc++`` used, for that we need to update ``ldconfig`` with the +local directories. Until then, this was not necessary in the previous images, as everything +was either consumed directly from the system, or installed directly in ``/usr``. These are the main features of the base image, which is used in all final images. -For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles -and challenges that made us change our mind. Here we will share a little bit of this long journey of CMake files and -compilation hours. To see the full recipe, it is available +For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a +series of obstacles and challenges that made us change our mind. Here we will share a little bit +of this long journey of CMake files and compilation hours. To see the full recipe, it is available [here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/clang/Dockerfile). ### Building GCC from Source Now let's look at the GCC build image, the full recipe can be found -[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/gcc/Dockerfile), but let's -highlight a few points: +[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/gcc/Dockerfile), +but let's highlight a few points: {% highlight docker %} @@ -189,44 +271,44 @@ RUN cd gcc-${GCC_VERSION} \ {% endhighlight %} -No matter the version, GCC continues to use the same lines for its build. -Some factors were configured in this version used: +No matter the version, GCC continues to use the same lines for its build. Some factors were +configured in this version used: -* Bootstrap has been disabled to reduce build time to just 20 minutes. -* Fortran is enabled, but it barely increase the building time and final +* Bootstrap has been disabled to reduce build time to just 20 minutes. * Fortran is enabled, +but it barely increase the building time and final The last part of the image uses the concept of Docker -[multistage-build](https://docs.docker.com/develop/develop-images/multistage-build/), a technique that avoids creating -separate recipes and images to take advantage of common parts. +[multistage-build](https://docs.docker.com/develop/develop-images/multistage-build/), a technique +that avoids creating separate recipes and images to take advantage of common parts. {% highlight docker %} FROM ${DOCKER_USERNAME}/base-${DISTRO}:${DOCKER_TAG} as deploy -ARG GCC_VERSION -ARG LIBSTDCPP_PATCH_VERSION +ARG GCC_VERSION ARG LIBSTDCPP_PATCH_VERSION COPY --from=builder /tmp/install /tmp/install RUN sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ - && sudo cp -a /tmp/install/lib/gcc/x86_64-linux-gnu/${GCC_VERSION} /usr/lib/gcc/x86_64-linux-gnu/ \ - && sudo cp -a /tmp/install/include/* /usr/local/include/ \ - && sudo cp -a /tmp/install/lib64/ /usr/local/ \ - && sudo cp -a /tmp/install/libexec/ /usr/local/ \ - && sudo cp -a /tmp/install/lib/* /usr/local/lib/ \ - && sudo cp -a /tmp/install/bin/* /usr/local/bin/ \ - && sudo rm -rf /tmp/install \ - && sudo update-alternatives --install /usr/local/bin/cc cc /usr/local/bin/gcc 100 \ - && sudo update-alternatives --install /usr/local/bin/cpp cpp /usr/local/bin/g++ 100 \ - && sudo update-alternatives --install /usr/local/bin/c++ c++ /usr/local/bin/g++ 100 \ - && sudo rm /etc/ld.so.cache \ - && sudo ldconfig -C /etc/ld.so.cache \ - && conan config init --force + && sudo cp -a /tmp/install/lib/gcc/x86_64-linux-gnu/${GCC_VERSION} + /usr/lib/gcc/x86_64-linux-gnu/ \ && sudo cp -a /tmp/install/include/* /usr/local/include/ \ && + sudo cp -a /tmp/install/lib64/ /usr/local/ \ && sudo cp -a /tmp/install/libexec/ /usr/local/ + \ && sudo cp -a /tmp/install/lib/* /usr/local/lib/ \ && sudo cp -a /tmp/install/bin/* + /usr/local/bin/ \ && sudo rm -rf /tmp/install \ && sudo update-alternatives --install + /usr/local/bin/cc cc /usr/local/bin/gcc 100 \ && sudo update-alternatives --install + /usr/local/bin/cpp cpp /usr/local/bin/g++ 100 \ && sudo update-alternatives --install + /usr/local/bin/c++ c++ /usr/local/bin/g++ 100 \ && sudo rm /etc/ld.so.cache \ && sudo + ldconfig -C /etc/ld.so.cache \ && conan config init --force {% endhighlight %} -In this section the base image used is the same one we created before through a caching mechanism that drastically reduces the final image build time. All artifacts generated from GCC are now copied to their respective locations. Finally, the compiler becomes the default in the image and the libraries are listed and cached. And here's the icing on the cake, the same recipe works from GCC 5 to the latest version. You just need to modify some arguments. The maintenance has been drastically simplified compared to current Conan Docker Tools. +In this section the base image used is the same one we created before through a caching mechanism +that drastically reduces the final image build time. All artifacts generated from GCC are now +copied to their respective locations. Finally, the compiler becomes the default in the image and +the libraries are listed and cached. And here's the icing on the cake, the same recipe works +from GCC 5 to the latest version. You just need to modify some arguments. The maintenance has +been drastically simplified compared to current Conan Docker Tools. ### Conan Meets the Wyvern: Building Clang C/C++ Compiler from Source @@ -241,57 +323,55 @@ FROM ${DOCKER_USERNAME}/gcc${LIBSTDCPP_MAJOR_VERSION}-${DISTRO}:${DOCKER_TAG} as FROM ${DOCKER_USERNAME}/base-${DISTRO}:${DOCKER_TAG} as deploy -ARG LIBSTDCPP_VERSION -ARG LIBSTDCPP_PATCH_VERSION +ARG LIBSTDCPP_VERSION ARG LIBSTDCPP_PATCH_VERSION -ARG DOCKER_USERNAME -ARG DOCKER_TAG -ARG DISTRO +ARG DOCKER_USERNAME ARG DOCKER_TAG ARG DISTRO -COPY --from=builder /tmp/install /tmp/clang -COPY --from=libstdcpp /usr/local /tmp/gcc +COPY --from=builder /tmp/install /tmp/clang COPY --from=libstdcpp /usr/local /tmp/gcc RUN sudo mv /tmp/gcc/lib64 /usr/local/ \ - && sudo ln -s -f /usr/local/lib64/libstdc++.so.6.0.${LIBSTDCPP_PATCH_VERSION} /usr/local/lib64/libstdc++.so.6 \ - && sudo ln -s -f /usr/local/lib64/libstdc++.so.6 /usr/local/lib64/libstdc++.so \ - && sudo cp -a /tmp/gcc/include/* /usr/local/include/ \ - && sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ - && sudo cp -a /tmp/gcc/lib/gcc/x86_64-linux-gnu/${LIBSTDCPP_VERSION} /usr/lib/gcc/x86_64-linux-gnu/ \ - && sudo cp -a /tmp/gcc/lib/* /usr/local/lib/ \ - ... + && sudo ln -s -f /usr/local/lib64/libstdc++.so.6.0.${LIBSTDCPP_PATCH_VERSION} + /usr/local/lib64/libstdc++.so.6 \ && sudo ln -s -f /usr/local/lib64/libstdc++.so.6 + /usr/local/lib64/libstdc++.so \ && sudo cp -a /tmp/gcc/include/* + /usr/local/include/ \ && sudo rm -rf /usr/lib/gcc/x86_64-linux-gnu/* \ && sudo cp -a + /tmp/gcc/lib/gcc/x86_64-linux-gnu/${LIBSTDCPP_VERSION} /usr/lib/gcc/x86_64-linux-gnu/ \ && + sudo cp -a /tmp/gcc/lib/* /usr/local/lib/ \ ... {% endhighlight %} -Similar to what was done with GCC, in Clang we also use the same base image and copy the artifacts generated by the -compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This -is a necessity of the possible configurations supported by Conan. +Similar to what was done with GCC, in Clang we also use the same base image and copy the +artifacts generated by the compiler to the ``/usr/local`` directory. However, the ``libstdc++`` +library was extracted from the GCC 10 image. This is a necessity of the possible configurations +supported by Conan. -In addition Clang requires some interesting CMake definitions: -* LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build -* LLVM_USE_LINKER: We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time +In addition Clang requires some interesting CMake definitions: * LLVM_ENABLE_PROJECTS: Only +enable what we want, otherwise we will have tons of binaries and hours of build * LLVM_USE_LINKER: +We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time {% highlight docker %} -... -&& ninja unwind \ -&& ninja cxxabi \ -&& cp lib/libc++abi* /usr/lib/ \ -&& ninja cxx \ -&& ninja clang \ +... && ninja unwind \ && ninja cxxabi \ && cp lib/libc++abi* /usr/lib/ \ && ninja cxx \ && +ninja clang \ {% endhighlight %} -If we run `ninja` command alone, it builds more projects than we want configured as enabled, so we -build one by one. -Also, `libcxx` has a limitation when building using `libc++abi`, it searches on the system library folder, -not the internal folders first. +If we run `ninja` command alone, it builds more projects than we want configured as enabled, +so we build one by one. Also, `libcxx` has a limitation when building using `libc++abi`, +it searches on the system library folder, not the internal folders first. ### Tests and more tests: A CI pipeline to test Docker images -To ensure that the images produced met our requirements we needed to add new tests that cover in addition to what was already tested in Conan Docker Tools. Until then, a single script was used which validated a series of builds, versions of installed binaries, and user permissions. The content of the new tests can be seen [here](https://github.com/conan-io/conan-docker-tools/tree/master/modern/tests). +To ensure that the images produced met our requirements we needed to add new tests +that cover in addition to what was already tested in Conan Docker Tools. Until +then, a single script was used which validated a series of builds, versions of +installed binaries, and user permissions. The content of the new tests can be seen +[here](https://github.com/conan-io/conan-docker-tools/tree/master/modern/tests). -We introduced greater modularization in the tests, dividing the steps into separate scripts to serve each compiler closer. With the support for Fortran it was necessary to adapt a test that covered it. Furthermore, many applications that are now bundled with Conan are no longer part of the base image and this has also been validated. +We introduced greater modularization in the tests, dividing the steps into separate scripts to +serve each compiler closer. With the support for Fortran it was necessary to adapt a test that +covered it. Furthermore, many applications that are now bundled with Conan are no longer part +of the base image and this has also been validated. If you want to test locally a produced Docker image, you can easily run: @@ -303,54 +383,68 @@ $ cd modern && pytest tests --image conanio/gcc10-ubuntu16.04 --service deploy - ### The CI Service Change: From Travis to Azure and the Jenkins Arrival -Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis and Azure. However, this did not give us the full power to prioritize the build in the queue, customize the host, or customize the build lines to use Docker-in-Docker if necessary. +Since the beginning of the project, Conan Docker Tools has always used CI services such as Travis +and Azure. However, this did not give us the full power to prioritize the build in the queue, +customize the host, or customize the build lines to use Docker-in-Docker if necessary. -With that in mind we started using Jenkins to also build the new Docker images. The big advantage in this is the use of features for cache in Docker. Previously a single job took up to 2 hours if it was on other services without the use of caching. Now using Jenkins and Docker ``--cache-from``, updating a package from base image to final image takes just 4 minutes per job. The Jenkinsfile file used can be viewed +With that in mind we started using Jenkins to also build the new Docker images. The +big advantage in this is the use of features for cache in Docker. Previously a single +job took up to 2 hours if it was on other services without the use of caching. Now +using Jenkins and Docker ``--cache-from``, updating a package from base image to +final image takes just 4 minutes per job. The Jenkinsfile file used can be viewed [here](https://github.com/conan-io/conan-docker-tools/blob/master/.ci/xenial.jenkinsfile). -Although the file looks complicated at first glance, it is still possible to use docker-compose to build an image from scratch. As an example, let's use Clang 12: +Although the file looks complicated at first glance, it is still possible to use docker-compose +to build an image from scratch. As an example, let's use Clang 12: {% highlight bash %} -$ cd modern -$ docker-compose build base -$ docker-compose build clang12-builder -$ docker-compose build clang12 +$ cd modern $ docker-compose build base $ docker-compose build clang12-builder $ docker-compose +build clang12 {% endhighlight %} -The produced image will be named as ``conanio/clang12-ubuntu16.04:1.39.0``, where ``1.39.0`` is the image tag and -version of Conan installed. But it's totally configurable by the .env file. +The produced image will be named as ``conanio/clang12-ubuntu16.04:1.39.0``, where ``1.39.0`` +is the image tag and version of Conan installed. But it's totally configurable by the .env file. -In the case of legacy images, they will continue to be built in Azure when needed, we have no intention of moving them -to Jenkins due to effort and maintenance. +In the case of legacy images, they will continue to be built in Azure when needed, we have no +intention of moving them to Jenkins due to effort and maintenance. ### How to build a Conan package with new Docker images -After building our new images, we are ready to build our Conan packages. Let's take Boost as our example. +After building our new images, we are ready to build our Conan packages. Let's take Boost as +our example. {% highlight shell %} -$ docker run --rm -ti -v ${HOME}/.conan/data:/home/conan/.conan/data conanio/gcc10-ubuntu16.04:1.39.0 -conan@148a77cfbc33:~$ conan install boost/1.76.0@ --build +$ docker run --rm -ti -v ${HOME}/.conan/data:/home/conan/.conan/data +conanio/gcc10-ubuntu16.04:1.39.0 conan@148a77cfbc33:~$ conan install boost/1.76.0@ --build conan@148a77cfbc33:~$ exit {% endhighlight %} -Here, we start a temporary Docker container with interactive support. Also, we share our Conan cache data as volume. After starting, we build Boost 1.76.0 and its dependencies from source. All packages will be built and installed to the shared volume, so we can use it after closing the container. To finish and remove the container, we just need to exit. +Here, we start a temporary Docker container with interactive support. Also, we share our Conan +cache data as volume. After starting, we build Boost 1.76.0 and its dependencies from source. All +packages will be built and installed to the shared volume, so we can use it after closing the +container. To finish and remove the container, we just need to exit. {% highlight shell %} -$ docker run -d -t -v ${HOME}/.conan/data:/home/conan/.conan/data --name conan_container conanio/gcc10 -$ docker exec conan_container conan install boost/1.76.0@ --build -$ docker stop conan_container -$ docker rm conan_container +$ docker run -d -t -v ${HOME}/.conan/data:/home/conan/.conan/data --name conan_container +conanio/gcc10 $ docker exec conan_container conan install boost/1.76.0@ --build $ docker stop +conan_container $ docker rm conan_container {% endhighlight %} -Similar execution, same result. Instead of creating a temporary Docker container, we executed it in the background. All container commands are passed by ``conan exec`` command. Also, we need to stop and remove manually after finishing. +Similar execution, same result. Instead of creating a temporary Docker container, we executed +it in the background. All container commands are passed by ``conan exec`` command. Also, we +need to stop and remove manually after finishing. ## Finals words and feedback -We invite everyone who uses Conan Docker Tools images to use this new formulation, which should become official soon. These new images are the result of a long journey where we learned from our mistakes and listened to the community to achieve what we have today. We believe improvements should be added continuously to keep the CDT progressing so please let us your feedback on issue [#205](https://github.com/conan-io/conan-docker-tools/issues/205). +We invite everyone who uses Conan Docker Tools images to use this new formulation, which should +become official soon. These new images are the result of a long journey where we learned from +our mistakes and listened to the community to achieve what we have today. We believe improvements +should be added continuously to keep the CDT progressing so please let us your feedback on issue +[#205](https://github.com/conan-io/conan-docker-tools/issues/205). From 351261ee1762d693eb288529f420d6f375993597 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:00:27 -0300 Subject: [PATCH 34/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 47266be6..325bae3d 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -19,7 +19,7 @@ the problem, the motivation and the proposed solution for Conan Docker Tools. In continue the proposed implementation, highlight the problems encountered during development,and show how this solution improves many aspects of the way we use Docker images for Conan. Also, this blog is a bit longer than usual and contains explanations about the decisions made, mistakes -committed, and details the technical part of working with Dockerfiles. e divided this blog post +committed, and details the technical part of working with Dockerfiles. We divided this blog post in 2 parts: * Part 1: The long journey, ideas, motivation, mistakes and challenges From 550eb9bc4a037ed7b4ed2392c77f728cba73b659 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:01:06 -0300 Subject: [PATCH 35/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 325bae3d..decfe667 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -56,14 +56,15 @@ part of our journey please skip ahead to Part 2. The pull request [#204](https://github.com/conan-io/conan-docker-tools/pull/204) showed us some weak points to consider: -* Installing Clang via LLVM's APT repository does not guarantee full compatibility with -``libstdc++`` version used to build GCC. * APT packages that were requirements for GCC and -Clang were still present, further inflating the final image. * The packages provided through -Ubuntu do not have older versions available, in case a new version comes along. This affects -the reproducibility requirement. * Older compilers were always an issue in terms of maintenance -once they arrived in the EOL (end-of-life) state. It was necessary to update the PPA address and -build the images again * The continuous integration service used, although it was considerably -fast, it was not possible to customize and prioritize the build, if necessary. + * Installing Clang via LLVM's APT repository does not guarantee full compatibility with + ``libstdc++`` version used to build GCC. + * APT packages that were requirements for GCC and Clang were still present, further inflating the final image. + * The packages provided through Ubuntu do not have older versions available, in case a new version + comes along. This affects the reproducibility requirement. + * Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) + state. It was necessary to update the PPA address and build the images again. + * The continuous integration service used, although it was considerably fast, it was not possible to + customize and prioritize the build, if necessary. Noticing the listed issues, we tried a radical solution. This solution took more time to implement but resulted in something better in terms of maintainability and practicality. From f45b72f043c857f44fd5d770df85bc20eee2e1ee Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:01:19 -0300 Subject: [PATCH 36/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index decfe667..c5c24a06 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -123,21 +123,22 @@ and with it a new ``libstdc++`` version (``6.0.29``). It was not possible to use version with this new compiler. We were left with the following dilemma: * Using the same libstdc++ version, except for GCC 11. -* Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked + * Conan Center becomes homogeneous (except GCC 11): all binaries will be built and linked using the same ``libstdc++`` version, which guarantees that all can run in any image. -* Binaries can hardly be used outside of Conan Center because they need the newest version + * Binaries can hardly be used outside of Conan Center because they need the newest version of libstdc++ library that is not yet available in the official PPA. All executables built inside ConanCenter won't work in the users' machines. -* A possible solution would be to statically link libstdc++ in all binaries, but this + * A possible solution would be to statically link libstdc++ in all binaries, but this solution has a number of risks. -* Each image uses the corresponding version of libstdc++ provided by the compiler. -* Better than the current scenario, where it is dependent on PPA and we have no control + +* Each image uses the corresponding version of libstdc++ provided by the compiler: + * Better than the current scenario, where it is dependent on PPA and we have no control over it. -* All images still use the same version of glibc, another advantage over the + * All images still use the same version of glibc, another advantage over the current scenario. -* We will need to take care of the executables, as they will only be + * We will need to take care of the executables, as they will only be compatible with later versions (as they are now). -* Better for users, than the current scenario. + * Better for users, than the current scenario. The requirements related to libstdc++ are the same, but the glibc version is the same version for all the packages. From ca97ef5d576abc7db3a837669ba8519b1cb48f78 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:01:34 -0300 Subject: [PATCH 37/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index c5c24a06..10d89371 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -193,7 +193,7 @@ and making it difficult to use the same Docker recipe for all versions. * With the removal of the GCC dependency, it was necessary to use ``libunwind`` during the build. It is already internalized in LLVM, but used as a dynamic library only. So a question arises, what happens if a project uses the image with Clang and installs Conan's libunwind package? A big mess -when linking, is the answer. Clang tries to link the version distributed by the Conan package, +when linking is the answer. Clang tries to link the version distributed by the Conan package, resulting in several errors. As a workaround, we renamed the original LLVM ``libunwind`` to ``libllvm-unwind``. From 81ee30237c582ffe68bf038652fc7f9a8c431a55 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:07:58 -0300 Subject: [PATCH 38/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 10d89371..6165aefc 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -198,7 +198,7 @@ resulting in several errors. As a workaround, we renamed the original LLVM ``libunwind`` to ``libllvm-unwind``. It became quite difficult to maintain from Clang 6 to 12 with the discovery of these issues and -limitations.After a lot of discussions and advice from some of the LLVM maintainers we decided +limitations. After a lot of discussions and advice from some of the LLVM maintainers we decided to limit Clang support to starting from version 10 because it is not necessary to apply as many modifications including the configuration file. Also, in the Linux environment Clang is not the primary compiler so we believe its use is always tied to newer versions. From 08ab8793e1f09d2d8355c7518eca994e4d8a1cf4 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:08:11 -0300 Subject: [PATCH 39/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 6165aefc..7db35b4f 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -276,7 +276,8 @@ RUN cd gcc-${GCC_VERSION} \ No matter the version, GCC continues to use the same lines for its build. Some factors were configured in this version used: -* Bootstrap has been disabled to reduce build time to just 20 minutes. * Fortran is enabled, +* Bootstrap has been disabled to reduce build time to just 20 minutes. +* Fortran is enabled, but it barely increase the building time and final The last part of the image uses the concept of Docker From 28035ee70c51c801872f00be47d7778d6434fefd Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:08:22 -0300 Subject: [PATCH 40/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 7db35b4f..a7c2790a 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -345,7 +345,7 @@ RUN sudo mv /tmp/gcc/lib64 /usr/local/ \ Similar to what was done with GCC, in Clang we also use the same base image and copy the artifacts generated by the compiler to the ``/usr/local`` directory. However, the ``libstdc++`` library was extracted from the GCC 10 image. This is a necessity of the possible configurations -supported by Conan. +supported by Conan (possible values of ``compiler.libcxx`` setting). In addition Clang requires some interesting CMake definitions: * LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build * LLVM_USE_LINKER: From 938f24ec7e0cbaa7e013f31674f50dddb000ccb3 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:08:35 -0300 Subject: [PATCH 41/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index a7c2790a..f08f3bef 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -347,9 +347,9 @@ artifacts generated by the compiler to the ``/usr/local`` directory. However, th library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan (possible values of ``compiler.libcxx`` setting). -In addition Clang requires some interesting CMake definitions: * LLVM_ENABLE_PROJECTS: Only -enable what we want, otherwise we will have tons of binaries and hours of build * LLVM_USE_LINKER: -We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time +In addition Clang requires some interesting CMake definitions: + * LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build + * LLVM_USE_LINKER: We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time {% highlight docker %} From 04b0617964867e40eccc42a410b754c1d5153f56 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:09:28 -0300 Subject: [PATCH 42/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index f08f3bef..4a3a9c09 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -408,7 +408,7 @@ build clang12 {% endhighlight %} The produced image will be named as ``conanio/clang12-ubuntu16.04:1.39.0``, where ``1.39.0`` -is the image tag and version of Conan installed. But it's totally configurable by the .env file. +is the image tag and version of Conan installed. But it's totally configurable by the ``.env`` file. In the case of legacy images, they will continue to be built in Azure when needed, we have no intention of moving them to Jenkins due to effort and maintenance. From c96011b3a5cd94e493192e7ae03e69504c172687 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:10:42 -0300 Subject: [PATCH 43/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 4a3a9c09..5fe41ab1 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -29,7 +29,7 @@ in 2 parts: We want to thank the community for all of the great and important feedback we received involving Conan and Docker images. Your feedback lets us know how important these Docker images are not -only for the Conan Center by generating official packages for general distribution, but also +only for Conan Center to generate official packages for general distribution, but also for people who own their personal projects and use the images to validate and build their Conan recipes, as well as the companies they build packages through sources. From 3e4c79e2ce228257f913f78bb7f543c96f5d65bc Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:10:51 -0300 Subject: [PATCH 44/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 5fe41ab1..f3991942 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -31,7 +31,7 @@ We want to thank the community for all of the great and important feedback we Conan and Docker images. Your feedback lets us know how important these Docker images are not only for Conan Center to generate official packages for general distribution, but also for people who own their personal projects and use the images to validate and build their Conan -recipes, as well as the companies they build packages through sources. +recipes, as well as the companies that build packages through sources. After long feedback and new formulations we understood that it was necessary to make changes to the initial concept. We realized that we could not modify the existing images for better From e429a16360c17292f4fb3f9ae1f76f4ce80a30bb Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:17:46 -0300 Subject: [PATCH 45/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index f3991942..5da13956 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -97,7 +97,7 @@ large build, effort and maintenance load for old images and packages that are no by the community. Therefore, the following rule will be adopted: * Clang will be supported from 10.0 to the newest version. -* GCC on the other hand, is widely used for the Linux environment and only version 4.x was left out. +* GCC, on the other hand, is widely used for the Linux environment and we will generate images from version 5 to the latest. * For both compilers we will keep updating all new compiler versions and Conan client version, according to new releases. * The multilib support was discarded as we are only interested in producing packages with From bf71b6b2a6be7b429e5561610ebea355d2737c4f Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:17:59 -0300 Subject: [PATCH 46/49] Update _posts/2021-08-09-Modern-docker-images.markdown Co-authored-by: Javier G. Sogo --- _posts/2021-08-09-Modern-docker-images.markdown | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 5da13956..d4259b90 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -146,9 +146,8 @@ Given the conditions and risks, we chose to go the second way: **Use the libstdc available together with the compiler**. As Clang also supports ``libstdc++`` we choose ``libstdc++.so.6.0.28`` to be its default -version. As commented before, that version has some advantages due its age and compatibility. To -copy the library with Clang, first we needed to build GCC 10, then we could mount its Docker -container and copy the ``libstdc++.so.6.0.28`` to the Clang image. +version. As commented before, that version has some advantages due to its age and compatibility. It is also +the default for the latest LTS Ubuntu version, so we really think it should satisfy most of the use-cases. Ubuntu 16.04 Xenial LTS is still the base used and it will be supported until April 2024. After that date we will need to update the images to a newer version of the distribution in addition From 44807020db864f865fc3a6d871965925ee2b7562 Mon Sep 17 00:00:00 2001 From: Uilian Ries Date: Wed, 11 Aug 2021 11:24:40 -0300 Subject: [PATCH 47/49] Remove leftover link Signed-off-by: Uilian Ries --- _posts/2021-08-09-Modern-docker-images.markdown | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index d4259b90..8d264f27 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -59,7 +59,7 @@ weak points to consider: * Installing Clang via LLVM's APT repository does not guarantee full compatibility with ``libstdc++`` version used to build GCC. * APT packages that were requirements for GCC and Clang were still present, further inflating the final image. - * The packages provided through Ubuntu do not have older versions available, in case a new version + * The packages provided through Ubuntu do not have older versions available, in case a new version comes along. This affects the reproducibility requirement. * Older compilers were always an issue in terms of maintenance once they arrived in the EOL (end-of-life) state. It was necessary to update the PPA address and build the images again. @@ -78,7 +78,7 @@ We first need to understand the objective behind the project before we start imp dockerfiles with their proper corrections. We want to understand what the expectation is for these new images 2 years from now. With the current approach we are already maintaining over 40 different dockerfiles which can be problematic. We also want to address one of the points -discussed previously: will there be rotation to avoid the accumulation of old images and +discussed previously: there will be rotation to avoid the accumulation of old images and their restrictions in terms of maintenance. #### The Plan @@ -256,8 +256,7 @@ These are the main features of the base image, which is used in all final images For the construction of Clang, we tried to make it available from version 6.0 to 12, but we had a series of obstacles and challenges that made us change our mind. Here we will share a little bit -of this long journey of CMake files and compilation hours. To see the full recipe, it is available -[here](https://github.com/conan-io/conan-docker-tools/blob/master/modern/clang/Dockerfile). +of this long journey of CMake files and compilation hours. ### Building GCC from Source @@ -346,8 +345,8 @@ artifacts generated by the compiler to the ``/usr/local`` directory. However, th library was extracted from the GCC 10 image. This is a necessity of the possible configurations supported by Conan (possible values of ``compiler.libcxx`` setting). -In addition Clang requires some interesting CMake definitions: - * LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build +In addition Clang requires some interesting CMake definitions: + * LLVM_ENABLE_PROJECTS: Only enable what we want, otherwise we will have tons of binaries and hours of build * LLVM_USE_LINKER: We enforce LLVM linker (lld). It's faster than GNU ld and reduces the total building time {% highlight docker %} From 0c77beb9f736a06b0677ce8ec588c6c60dd2b1ce Mon Sep 17 00:00:00 2001 From: "Javier G. Sogo" Date: Thu, 12 Aug 2021 09:55:29 +0200 Subject: [PATCH 48/49] Update _posts/2021-08-09-Modern-docker-images.markdown --- _posts/2021-08-09-Modern-docker-images.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 8d264f27..1e512725 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -84,7 +84,7 @@ their restrictions in terms of maintenance. #### The Plan For current and already available images on hub.docker these will be kept but no longer supported -(new versions will not be introduced). Your recipes and images will remain available, as their +(new versions will not be introduced unless it's needed for ConanCenter). Your recipes and images will remain available, as their immediate removal would result in catastrophic failure for many users. They will eventually be removed, but at a distant and yet to be defined date. From 12783d21ce2fdb39187d6a5b154c31d3984b59db Mon Sep 17 00:00:00 2001 From: "Javier G. Sogo" Date: Thu, 12 Aug 2021 09:55:59 +0200 Subject: [PATCH 49/49] Apply suggestions from code review --- .../2021-08-09-Modern-docker-images.markdown | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/_posts/2021-08-09-Modern-docker-images.markdown b/_posts/2021-08-09-Modern-docker-images.markdown index 1e512725..a5db3f90 100644 --- a/_posts/2021-08-09-Modern-docker-images.markdown +++ b/_posts/2021-08-09-Modern-docker-images.markdown @@ -107,7 +107,7 @@ the image. Currently the Conan package for ``gfortran`` is totally broken and ha dependencies chain to fix. We always wanted to be independent of the Linux distribution so one of the problems we would like -to solve is the compiler used and its libraries . In the initial pull request, we built the GCC +to solve is the compiler used and its libraries. In the initial pull request, we built the GCC from sources, while the Clang uses prebuilt. To solve this problem we chose to build both from sources in order to have more control over the compiler used. Packages generated using these images (packages in ConanCenter) should work out-of-the-box in as many as possible different distros. @@ -400,8 +400,10 @@ to build an image from scratch. As an example, let's use Clang 12: {% highlight bash %} -$ cd modern $ docker-compose build base $ docker-compose build clang12-builder $ docker-compose -build clang12 +$ cd modern +$ docker-compose build base +$ docker-compose build clang12-builder +$ docker-compose build clang12 {% endhighlight %} @@ -418,8 +420,8 @@ our example. {% highlight shell %} -$ docker run --rm -ti -v ${HOME}/.conan/data:/home/conan/.conan/data -conanio/gcc10-ubuntu16.04:1.39.0 conan@148a77cfbc33:~$ conan install boost/1.76.0@ --build +$ docker run --rm -ti -v ${HOME}/.conan/data:/home/conan/.conan/data conanio/gcc10-ubuntu16.04:1.39.0 +conan@148a77cfbc33:~$ conan install boost/1.76.0@ --build conan@148a77cfbc33:~$ exit {% endhighlight %} @@ -431,9 +433,10 @@ container. To finish and remove the container, we just need to exit. {% highlight shell %} -$ docker run -d -t -v ${HOME}/.conan/data:/home/conan/.conan/data --name conan_container -conanio/gcc10 $ docker exec conan_container conan install boost/1.76.0@ --build $ docker stop -conan_container $ docker rm conan_container +$ docker run -d -t -v ${HOME}/.conan/data:/home/conan/.conan/data --name conan_container conanio/gcc10 +$ docker exec conan_container conan install boost/1.76.0@ --build +$ docker stop conan_container +$ docker rm conan_container {% endhighlight %}