Reduce terraform deployer build verbosity #665

endorama · 2022-01-26T11:01:20Z

Reduce output from gcloud SDK installation process, by silencing the various steps.

This is needed to reduce image build log verbosity; gcloud SDK installation output is also difficult to read on terminal that do not support colours.

mtojek · 2022-01-26T11:33:52Z

internal/install/_static/Dockerfile.terraform_deployer

 RUN apk add --no-cache git openssh curl

+# Announcements for new releases: https://groups.google.com/g/google-cloud-sdk-announce


mtojek · 2022-01-26T11:34:23Z

internal/install/_static/Dockerfile.terraform_deployer

 ENV GCLOUD_SDK_VERSION 369.0.0
 ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
-RUN curl "https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-$GCLOUD_SDK_VERSION-linux-x86_64.tar.gz" > /tmp/google-cloud-sdk.tar.gz \
+RUN curl --silent "https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-$GCLOUD_SDK_VERSION-linux-x86_64.tar.gz" > /tmp/google-cloud-sdk.tar.gz \


curl -s <url> -o /tmp/google-cloud-sdk.tar.gz

mtojek · 2022-01-26T11:36:13Z

internal/install/_static/Dockerfile.terraform_deployer

-  && tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
-  && /usr/local/gcloud/google-cloud-sdk/install.sh -q --override-components="bq" \
+  && tar -C /usr/local/gcloud -xf /tmp/google-cloud-sdk.tar.gz \
+  && /usr/local/gcloud/google-cloud-sdk/install.sh -q --override-components="bq" &>/dev/null \


According to the public docs, there is --disable-prompts flag. We can give it a try and let the installer decide what's rubbish and what isn't :)

updated. Where can I look at the logs? I've not been able to find the pipeline where this log is displayed :(

Do you mean this?

[2022-01-26T14:00:31.542Z] ---> Running in f3dbb078bd82 [2022-01-26T14:00:41.632Z] Welcome to the Google Cloud SDK! [2022-01-26T14:00:41.632Z] WARNING: You appear to be running this script as root. This may cause [2022-01-26T14:00:41.632Z] the installation to be inaccessible to users other than the root user. [2022-01-26T14:00:43.561Z] usage: install.py [-h] [--usage-reporting USAGE_REPORTING] [2022-01-26T14:00:43.561Z] [--screen-reader SCREEN_READER] [--rc-path RC_PATH] [2022-01-26T14:00:43.561Z] [--command-completion COMMAND_COMPLETION] [2022-01-26T14:00:43.561Z] [--path-update PATH_UPDATE] [--disable-installation-options] [2022-01-26T14:00:43.561Z] [--override-components [OVERRIDE_COMPONENTS [OVERRIDE_COMPONENTS ...]]] [2022-01-26T14:00:43.561Z] [--additional-components ADDITIONAL_COMPONENTS [ADDITIONAL_COMPONENTS ...]] [2022-01-26T14:00:43.561Z] [--quiet] [--install-python INSTALL_PYTHON] [2022-01-26T14:00:43.561Z] install.py: error: unrecognized arguments: --disable-prompts [2022-01-26T14:00:47.792Z] Service 'terraform' failed to build: The command '/bin/sh -c curl --silent "https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-$GCLOUD_SDK_VERSION-linux-x86_64.tar.gz" > /tmp/google-cloud-sdk.tar.gz && mkdir -p /usr/local/gcloud && tar -C /usr/local/gcloud -xf /tmp/google-cloud-sdk.tar.gz && /usr/local/gcloud/google-cloud-sdk/install.sh -q --override-components="bq" --disable-prompts && rm /tmp/google-cloud-sdk.tar.gz' returned a non-zero code: 2

It looks there are different options available, sorry for linking wrong docs. Could you please adjust the command?

--disable-prompts is not available with the installation method in use, the tests for 8e505f2 broke with:

[2022-01-26T14:00:41.632Z] Welcome to the Google Cloud SDK! [2022-01-26T14:00:41.632Z] WARNING: You appear to be running this script as root. This may cause [2022-01-26T14:00:41.632Z] the installation to be inaccessible to users other than the root user. [2022-01-26T14:00:43.561Z] usage: install.py [-h] [--usage-reporting USAGE_REPORTING] [2022-01-26T14:00:43.561Z] [--screen-reader SCREEN_READER] [--rc-path RC_PATH] [2022-01-26T14:00:43.561Z] [--command-completion COMMAND_COMPLETION] [2022-01-26T14:00:43.561Z] [--path-update PATH_UPDATE] [--disable-installation-options] [2022-01-26T14:00:43.561Z] [--override-components [OVERRIDE_COMPONENTS [OVERRIDE_COMPONENTS ...]]] [2022-01-26T14:00:43.561Z] [--additional-components ADDITIONAL_COMPONENTS [ADDITIONAL_COMPONENTS ...]] [2022-01-26T14:00:43.561Z] [--quiet] [--install-python INSTALL_PYTHON] [2022-01-26T14:00:43.561Z] install.py: error: unrecognized arguments: --disable-prompts

Reverting the commit

Looks more readable, and seems to take less space (tough maybe more components need to be installed with the optional packages).

FROM ubuntu:20.04 RUN apt-get -q update && apt-get install -yq curl apt-transport-https ca-certificates gnupg && apt-get clean ADD google-cloud-sdk.list /etc/apt/sources.list.d/google-cloud-sdk.list RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - ENV GCLOUD_SDK_VERSION 369.0.0-0 RUN apt-get -q update && \ apt-get -yq install google-cloud-sdk=${GCLOUD_SDK_VERSION} && \ apt-get clean

da5b98a5dec3 About a minute ago /bin/sh -c apt-get -q update && apt-get … 620MB

eb91706519b5 5 minutes ago /bin/sh -c curl "https://dl.google.com/dl/cl… 1.05GB

Yeah, that's something I wanted originally to omit, but it looks like it may not be so easy unless we modify the service deployer image to pull it in runtime? That's probably a topic for a different discussion.

@mtojek I'm already using the versioned archive installation. install.sh triggers a python script that takes care of installing other components so I don't think can be safely skipped.

Could you please try again without calling install.sh at all? I suppose you need to unpack the archive in the correct place instead of /tmp.

I tried it but the result is not good. This is what happens when running gcloud in the docker container without running install:

/workspace # gcloud ERROR: (gcloud) Command name argument expected. ERROR: gcloud failed to load (gcloud.interactive): Problem loading gcloud.interactive: cannot import name 'Mapping' from 'collections' (/usr/local/lib/python3.10/collections/__init__.py). This usually indicates corruption in your gcloud installation or problems with your Python interpreter. [...]

PS We will definitely need a package system test to verify if it works as expected.

This is in the works!

I'm moving forward to using Ubuntu as base image.

I updated the image to use Ubuntu.

These are the dimensions for the two images:

ubuntu 978MB alpine 594MB

@jsoriano the gcloud SDK when installed from a versioned archive is a lot less that 1.05GB, as is possible to specify which components to install thus reducing the overall size.

I'm happy with both, let me know what do you think of the current change :)

Much better now and managed in a normal way :)

I will drop a few nit-picks.

elasticmachine · 2022-01-26T11:37:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-07T13:19:59.529+0000
Duration: 33 min 36 sec

Test stats 🧪

Test	Results
Failed	0
Passed	512
Skipped	0
Total	512

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

mtojek

I think we have an agreement. It's more like a final polishing of the PR.

mtojek · 2022-01-31T09:53:20Z

internal/install/_static/Dockerfile.terraform_deployer

-  && tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
-  && /usr/local/gcloud/google-cloud-sdk/install.sh -q --override-components="bq" \
-  && rm /tmp/google-cloud-sdk.tar.gz
+ENV GCLOUD_SDK_VERSION 370.0.0-0


Please move this to the top of the file (after FROM)

mtojek · 2022-01-31T09:56:40Z

internal/install/_static/Dockerfile.terraform_deployer

-  && /usr/local/gcloud/google-cloud-sdk/install.sh -q --override-components="bq" \
-  && rm /tmp/google-cloud-sdk.tar.gz
+ENV GCLOUD_SDK_VERSION 370.0.0-0
+RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \


I think you mixed all commands a bit.

Install GPG key and cloud-sdk repo

apt-get update -yq && apt-get install -yq curl apt-transport-https.... - google-cloud-sdk=${GCLOUD_SDK_VERSION} && apt-get clean

curl apt-transport-https gnupg are required to add the deb repo and install the repo key, so two steps are required unfortunately.
Would you prefer to split GPG key and repo addition from the actual package installation?

You can try something like this:

apt-get update -yq && \ apt-get install apt-transport-https gnupg && \ deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \ apt-get update -yq && \ apt-get install -yq google-cloud-sdk=${GCLOUD_SDK_VERSION} && \ apt-get clean

The aim is to consider this as an atomic piece to prevent any APT repo update from breaking the container build.

I updated it with the suggestion. Just consider that if in the future another repo will be required I think would be best to extract the dependency installation so there is no implicit dependency.

@mtojek I extracted it again, as Hashicorp has it's own repo that has the same requirements. It happened sooner than expected :D

mtojek

Something is fishy with the terraform as it doesn't start. Could you please investigate it?

jsoriano · 2022-01-31T20:22:21Z

internal/install/_static/Dockerfile.terraform_deployer

@@ -1,17 +1,14 @@
 FROM hashicorp/terraform:light as terraform


Now that we have apt maybe we can also install terraform using it :) https://www.terraform.io/downloads

And we could also pin the version.

This is unrelated to this PR in any case.

Actually it may help if this is a PATH issue or library issue :)

Added installation through deb thanks for the suggestion!

mtojek · 2022-02-02T14:35:23Z

@endorama Please pull the latest main branch.

internal/install/_static/Dockerfile.terraform_deployer

mtojek

I fetched this code locally, run it, and checked logs for the terraform service deployer:

$ docker logs 49fee7edacf7
/run.sh: 3: set: Illegal option -o pipefail

You have to update the terraform_deployer_run.sh to /bin/bash. sh most likely points to a different shell.

mtojek · 2022-02-04T10:40:09Z

internal/compose/compose.go

-			// Service is up and running and it's healthy
-			if containerDescription.State.Status == "running" && containerDescription.State.Health.Status == "healthy" {
-				continue
+			for _, containerDescription := range descriptions {


Let's extract this change around ticker to a separate PR and focus here on improving the terraform Docker image.

So, could you please remove it from this PR?

mtojek · 2022-02-04T11:20:21Z

internal/compose/compose.go

-			// Service is up and running and it's healthy
-			if containerDescription.State.Status == "running" && containerDescription.State.Health.Status == "healthy" {
-				continue
+			for _, containerDescription := range descriptions {


So, could you please remove it from this PR?

Reduce output from gcloud SDK installation process, by silencing the various steps. This is needed to reduce image build log verbosity; gcloud SDK installation output is also difficult to read on terminal that do not support colours.

This reverts commit 8e505f2. `--disable-prompts` is not available with the installation method in use, the tests for 8e505f2 broke with: ``` install.py: error: unrecognized arguments: --disable-prompts ```

mtojek · 2022-02-07T08:13:01Z

/test

mtojek · 2022-02-07T08:41:59Z

internal/install/_static/terraform_deployer_run.sh

@@ -1,7 +1,9 @@
-#!sh
+#!/usr/bin/env bash

 set -euxo pipefail


Let's drop -x to have a clean nice output in container logs. This script doesn't contain tangled logic which may cause issues.

mtojek · 2022-02-07T08:43:32Z

internal/install/_static/terraform_deployer_run.sh

@@ -1,7 +1,9 @@
-#!sh
+#!/usr/bin/env bash


The reason why the CI is failing is because of the Dockerfile ENTRYPOINT set to sh: here.

Let's quickly refactor it to /bin/bash instead and verify.

mtojek · 2022-02-07T10:58:37Z

internal/install/_static/terraform_deployer_run.sh

-set -euxo pipefail
+set -euo pipefail
+
+echo "BASH_VERSION: $BASH_VERSION"


Could you please remove this line?

mtojek

LGTM

endorama · 2022-02-08T08:53:14Z

🎉🎉 thank you!

In #665 we removed `-x` to debug an issue with script shebang, but the intention wasn't to remove it entirely from the script. This commit sets `-x` for the whole the script.

endorama requested a review from mtojek January 26, 2022 11:01

endorama self-assigned this Jan 26, 2022

mtojek reviewed Jan 26, 2022

View reviewed changes

mtojek requested a review from a team January 26, 2022 14:16

mtojek reviewed Jan 31, 2022

View reviewed changes

mtojek self-requested a review January 31, 2022 19:55

mtojek reviewed Jan 31, 2022

View reviewed changes

jsoriano reviewed Jan 31, 2022

View reviewed changes

mtojek mentioned this pull request Feb 2, 2022

Collect unsafe logs #679

Merged

mtojek reviewed Feb 3, 2022

View reviewed changes

internal/install/_static/Dockerfile.terraform_deployer Show resolved Hide resolved

mtojek self-requested a review February 4, 2022 10:23

mtojek reviewed Feb 4, 2022

View reviewed changes

mtojek suggested changes Feb 4, 2022

View reviewed changes

endorama mentioned this pull request Feb 4, 2022

Improve Docker Compose container execution #687

Merged

mtojek suggested changes Feb 4, 2022

View reviewed changes

endorama added 8 commits February 4, 2022 16:01

terraform deployer: silent gcloud SDK installation

5ee5cfd

Reduce output from gcloud SDK installation process, by silencing the various steps. This is needed to reduce image build log verbosity; gcloud SDK installation output is also difficult to read on terminal that do not support colours.

try disable-prompts flag

04a69dd

Revert "try disable-prompts flag"

40b2d3e

This reverts commit 8e505f2. `--disable-prompts` is not available with the installation method in use, the tests for 8e505f2 broke with: ``` install.py: error: unrecognized arguments: --disable-prompts ```

use ubuntu:20.04 base image

95b3fa4

install in single RUN

5f5d9c3

split apt requisites installation

ce34806

install terraform using deb

bf68394

suppress terraform colored output

cc8fa55

mtojek self-requested a review February 7, 2022 08:41

mtojek suggested changes Feb 7, 2022

View reviewed changes

mtojek reviewed Feb 7, 2022

View reviewed changes

run terraform deployer script in BASH

ca029f6

remove -x

af5bbcc

mtojek self-requested a review February 7, 2022 13:52

mtojek approved these changes Feb 7, 2022

View reviewed changes

jsoriano approved these changes Feb 7, 2022

View reviewed changes

endorama merged commit a77bd34 into elastic:main Feb 8, 2022

endorama deleted the terraform-deployer-gcp-verbosity branch February 8, 2022 08:53

This was referenced Feb 11, 2022

Use GCP ADC for auth in Terraform Deployer #662

Closed

Re-add -x to Terraform deployer script #699

Merged

endorama mentioned this pull request Feb 16, 2022

Migrate some Metricbeat modules to GCP package elastic/integrations#490

Closed

11 tasks

		RUN apk add --no-cache git openssh curl

		# Announcements for new releases: https://groups.google.com/g/google-cloud-sdk-announce

		@@ -1,17 +1,14 @@
		FROM hashicorp/terraform:light as terraform

Reduce terraform deployer build verbosity #665

Reduce terraform deployer build verbosity #665

Conversation

endorama commented Jan 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

endorama Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek Jan 31, 2022 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Jan 26, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek commented Feb 2, 2022

mtojek left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek commented Feb 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

endorama commented Feb 8, 2022

mtojek Jan 28, 2022 •

edited

Loading

endorama Jan 28, 2022 •

edited

Loading

mtojek Jan 31, 2022 •

edited

Loading

elasticmachine commented Jan 26, 2022 •

edited

Loading

mtojek left a comment •

edited

Loading