Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(docker): Prevent all possible "silent errors" during docker build #644

Merged
merged 19 commits into from
Apr 25, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix "redefinition" of global vars in function
MaxymVlasov committed Mar 14, 2024
commit 65ba59343a0faf47e761ac55e58495a9bc34b8cf
8 changes: 4 additions & 4 deletions tools/install/hcledit.sh
Original file line number Diff line number Diff line change
@@ -8,10 +8,10 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
# Unique part
#

readonly GH_ORG="minamijoyo"
readonly GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.tar.gz"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz"
readonly DISTRIBUTED_AS="tar.gz"
GH_ORG="minamijoyo"
GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe re-use DISTRIBUTED_AS?

Suggested change
GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"
DISTRIBUTED_AS="tar.gz"
GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.${DISTRIBUTED_AS}"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.${DISTRIBUTED_AS}"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, because:

  1. It will be different for binary
  2. You will know the Distribution type only when you get GH Release link, not otherwise

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It will be different for binary

The binary types just don't use the DISTRIBUTED_AS in GH_RELEASE_REGEX_LATEST and GH_RELEASE_REGEX_SPECIFIC_VERSION vars so they are not affected at all by the suggested change.

  1. You will know the Distribution type only when you get GH Release link, not otherwise

This is probably about something else.
I see the code:

GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"

and

GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}_${TARGETOS}_${TARGETARCH}"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}_${TARGETOS}_${TARGETARCH}"
DISTRIBUTED_AS="binary"

and I see no reason to not swap these two to the below:

DISTRIBUTED_AS="tar.gz"
GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.${DISTRIBUTED_AS}"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.${DISTRIBUTED_AS}"

and

DISTRIBUTED_AS="binary"
GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}_${TARGETOS}_${TARGETARCH}"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}_${TARGETOS}_${TARGETARCH}"

No change to binary types and optimization to non-binary types.

Am I missing something non-obvious?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once again:

You will know the Distribution type only when you get GH Release link, not otherwise

Imagine that you adding a new hook, let's say tfupdate.

You copy-past file, IE tfsec.sh rename it, cleanup pre-populated values and get this:

#...

GH_ORG=""
GH_RELEASE_REGEX_LATEST="https://.+?/"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?"
DISTRIBUTED_AS=""

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
  "$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"

How you should know DISTRIBUTED_AS value till you don't get any of GH_RELEASE_...?

Please, don't make it more complex than it already is.

Also, someone could distribute not .tar.gz but .tgz.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How you should know DISTRIBUTED_AS value till you don't get any of GH_RELEASE_...?

From what I see in each new script for each tool installation, the value of DISTRIBUTED_AS is known and is defined in each of these files. Along with that most of the files have the value of DISTRIBUTED_AS repeated three times. And my suggestion is to reduce it to one time.

Please, don't make it more complex than it already is.

I'm trying to simplify so that new tools are added without a need to copy&paste DISTRIBUTED_AS value three times on a three lines in a row.

image

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read once again, maybe tomorrow
#644 (comment)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There 0 simplification.

You're wrong, unless you prove opposite 🤝

Read once again, maybe tomorrow

That comment makes no sense to me. Re-write it once again, maybe tomorrow 😏

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a process of adding a new tool from scratch
Screencast from 24.04.24 21:01:35.webm

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Get URLs
  2. Replace URL parts with variables
  3. Simplify URL by replacing not matter parts by regex
  4. Set DISTRIBUTED_AS value based on URL

Also, I see that I'm already stuck to recreate that process from memory for LATEST regex, which means that this flow is already complicated.
That verbosity does not hurt anyone.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version without stuck (will add it to contributor docs)

Screencast.from.24.04.24.22.00.36.webm


common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
10 changes: 5 additions & 5 deletions tools/install/infracost.sh
Original file line number Diff line number Diff line change
@@ -7,11 +7,11 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
#
# Unique part
#
readonly GH_ORG="infracost"
readonly GH_RELEASE_REGEX_LATEST="https://.+?-${TARGETOS}-${TARGETARCH}.tar.gz"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}-${TARGETOS}-${TARGETARCH}.tar.gz"
readonly DISTRIBUTED_AS="tar.gz"
readonly UNUSUAL_TOOL_NAME_IN_PKG="${TOOL}-${TARGETOS}-${TARGETARCH}"
GH_ORG="infracost"
GH_RELEASE_REGEX_LATEST="https://.+?-${TARGETOS}-${TARGETARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}-${TARGETOS}-${TARGETARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"
UNUSUAL_TOOL_NAME_IN_PKG="${TOOL}-${TARGETOS}-${TARGETARCH}"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION" \
8 changes: 4 additions & 4 deletions tools/install/terraform-docs.sh
Original file line number Diff line number Diff line change
@@ -7,10 +7,10 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
#
# Unique part
#
readonly GH_ORG="terraform-docs"
readonly GH_RELEASE_REGEX_LATEST="https://.+?-${TARGETOS}-${TARGETARCH}.tar.gz"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}-${TARGETOS}-${TARGETARCH}.tar.gz"
readonly DISTRIBUTED_AS="tar.gz"
GH_ORG="terraform-docs"
GH_RELEASE_REGEX_LATEST="https://.+?-${TARGETOS}-${TARGETARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}-${TARGETOS}-${TARGETARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
8 changes: 4 additions & 4 deletions tools/install/terragrunt.sh
Original file line number Diff line number Diff line change
@@ -7,10 +7,10 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
#
# Unique part
#
readonly GH_ORG="gruntwork-io"
readonly GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}_${TARGETOS}_${TARGETARCH}"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}_${TARGETOS}_${TARGETARCH}"
readonly DISTRIBUTED_AS="binary"
GH_ORG="gruntwork-io"
GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}_${TARGETOS}_${TARGETARCH}"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}_${TARGETOS}_${TARGETARCH}"
DISTRIBUTED_AS="binary"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
8 changes: 4 additions & 4 deletions tools/install/terrascan.sh
Original file line number Diff line number Diff line change
@@ -16,10 +16,10 @@ fi
# Convert the first letter to Uppercase
OS="${TARGETOS^}"

readonly GH_ORG="tenable"
readonly GH_RELEASE_REGEX_LATEST="https://.+?_${OS}_${ARCH}.tar.gz"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${OS}_${ARCH}.tar.gz"
readonly DISTRIBUTED_AS="tar.gz"
GH_ORG="tenable"
GH_RELEASE_REGEX_LATEST="https://.+?_${OS}_${ARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${OS}_${ARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
8 changes: 4 additions & 4 deletions tools/install/tflint.sh
Original file line number Diff line number Diff line change
@@ -8,10 +8,10 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
# Unique part
#

readonly GH_ORG="terraform-linters"
readonly GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.zip"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?/v${VERSION}/${TOOL}_${TARGETOS}_${TARGETARCH}.zip"
readonly DISTRIBUTED_AS="zip"
GH_ORG="terraform-linters"
GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.zip"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?/v${VERSION}/${TOOL}_${TARGETOS}_${TARGETARCH}.zip"
DISTRIBUTED_AS="zip"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
8 changes: 4 additions & 4 deletions tools/install/tfsec.sh
Original file line number Diff line number Diff line change
@@ -8,10 +8,10 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
# Unique part
#

readonly GH_ORG="aquasecurity"
readonly GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}-${TARGETOS}-${TARGETARCH}"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}-${TARGETOS}-${TARGETARCH}"
readonly DISTRIBUTED_AS="binary"
GH_ORG="aquasecurity"
GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}-${TARGETOS}-${TARGETARCH}"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?v${VERSION}/${TOOL}-${TARGETOS}-${TARGETARCH}"
DISTRIBUTED_AS="binary"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
8 changes: 4 additions & 4 deletions tools/install/tfupdate.sh
Original file line number Diff line number Diff line change
@@ -7,10 +7,10 @@ readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
#
# Unique part
#
readonly GH_ORG="minamijoyo"
readonly GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.tar.gz"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz"
readonly DISTRIBUTED_AS="tar.gz"
GH_ORG="minamijoyo"
GH_RELEASE_REGEX_LATEST="https://.+?_${TARGETOS}_${TARGETARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?${VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"
8 changes: 4 additions & 4 deletions tools/install/trivy.sh
Original file line number Diff line number Diff line change
@@ -14,10 +14,10 @@ else
readonly ARCH="64bit"
fi

readonly GH_ORG="aquasecurity"
readonly GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}_.+?_${TARGETOS}-${ARCH}.tar.gz"
readonly GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?/v${VERSION}/${TOOL}_.+?_${TARGETOS}-${ARCH}.tar.gz"
readonly DISTRIBUTED_AS="tar.gz"
GH_ORG="aquasecurity"
GH_RELEASE_REGEX_LATEST="https://.+?/${TOOL}_.+?_${TARGETOS}-${ARCH}.tar.gz"
GH_RELEASE_REGEX_SPECIFIC_VERSION="https://.+?/v${VERSION}/${TOOL}_.+?_${TARGETOS}-${ARCH}.tar.gz"
DISTRIBUTED_AS="tar.gz"

common::install_from_gh_release "$GH_ORG" "$DISTRIBUTED_AS" \
"$GH_RELEASE_REGEX_LATEST" "$GH_RELEASE_REGEX_SPECIFIC_VERSION"