Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

feat: export Fetch&Download methods in the /pkg directory #1943

Merged
merged 7 commits into from
Jan 4, 2022
Merged

feat: export Fetch&Download methods in the /pkg directory #1943

merged 7 commits into from
Jan 4, 2022

Conversation

mdelapenya
Copy link
Contributor

  • chore: copy internal's version files into the exported pkg public package
  • chore: run unit tests from top-level goal
  • chore: use public package instead of the internal one
  • chore: remove old internal files for habdling downloads

What does this PR do?

This PR exposes the methods that were available in the old internal/versions.go file, using the well-known /pkg public (see https://github.com/golang-standards/project-layout/tree/master/pkg for standard layouts).

The Makefile running the unit tests has been updated to include the new pkg package.

Why is it important?

@narph needs to use similar code to download Beats binaries (from releases, CI snapshots, local paths), and we decided to expose this functionality from the E2E so that the team does not need to reinvent the wheel.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have run the Unit tests (make unit-test), and they are passing locally
  • I have run the End-2-End tests for the suite I'm working on, and they are passing locally
  • I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

  • Following standard layout for Go projects

How to test this PR locally

Related issues

@mdelapenya mdelapenya added Team:Automation Label for the Observability productivity team backport-v7.16.0 Automated backport with mergify backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify labels Dec 22, 2021
@mdelapenya mdelapenya self-assigned this Dec 22, 2021
@mdelapenya mdelapenya requested review from adam-stokes, a team and narph December 22, 2021 16:50
mdelapenya
mdelapenya commented Dec 22, 2021
View reviewed changes
pkg/downloads/versions.go
@@ -2,7 +2,7 @@
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package internal
package downloads
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adam-stokes I could not find a better name for this package. Maybe you have something better in mind

@mdelapenya
Copy link
Contributor Author

@narph let's us know if this would be enough for you to consume this.

@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 22, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-12-31T16:48:24.233+0000

  • Duration: 35 min 25 sec

  • Commit: 43f719b

Test stats 🧪

Test Results
Failed 0
Passed 216
Skipped 0
Total 216

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@mdelapenya
Copy link
Contributor Author

/test

@mdelapenya mdelapenya merged commit 63b6f42 into elastic:main Jan 4, 2022
mergify bot pushed a commit that referenced this pull request Jan 4, 2022
@mdelapenya
feat: export Fetch&Download methods in the /pkg directory (#1943)
bed60c0 
* chore: copy internal's version files into the exported pkg public package

* chore: run unit tests from top-level goal

* chore: use public package instead of the internal one

* chore: remove old internal files for habdling downloads

* fix: update missing references

(cherry picked from commit 63b6f42)
@mergify mergify bot mentioned this pull request Jan 4, 2022
Merged
mergify bot pushed a commit that referenced this pull request Jan 4, 2022
@mdelapenya
feat: export Fetch&Download methods in the /pkg directory (#1943)
e572b6b 
* chore: copy internal's version files into the exported pkg public package

* chore: run unit tests from top-level goal

* chore: use public package instead of the internal one

* chore: remove old internal files for habdling downloads

* fix: update missing references

(cherry picked from commit 63b6f42)

# Conflicts:
#	e2e/_suites/fleet/stand-alone.go
#	e2e/_suites/kubernetes-autodiscover/autodiscover_test.go
#	internal/deploy/docker.go
@mergify mergify bot mentioned this pull request Jan 4, 2022
Merged
mdelapenya added a commit that referenced this pull request Jan 4, 2022
@mergify @mdelapenya
feat: export Fetch&Download methods in the /pkg directory (#1943) (#1971
aa8e9ad 
)

* chore: copy internal's version files into the exported pkg public package

* chore: run unit tests from top-level goal

* chore: use public package instead of the internal one

* chore: remove old internal files for habdling downloads

* fix: update missing references

(cherry picked from commit 63b6f42)

Co-authored-by: Manuel de la Peña <[email protected]>
mdelapenya added a commit that referenced this pull request Jan 4, 2022
@mergify @mdelapenya
[7.16](backport #1943) feat: export Fetch&Download methods in the /pk…
73af71d 
…g directory (#1972)

* feat: export Fetch&Download methods in the /pkg directory (#1943)

* chore: copy internal's version files into the exported pkg public package

* chore: run unit tests from top-level goal

* chore: use public package instead of the internal one

* chore: remove old internal files for habdling downloads

* fix: update missing references

(cherry picked from commit 63b6f42)

# Conflicts:
#	e2e/_suites/fleet/stand-alone.go
#	e2e/_suites/kubernetes-autodiscover/autodiscover_test.go
#	internal/deploy/docker.go

* fix: manual backport of BeatsLocalPath variable

Co-authored-by: Manuel de la Peña <[email protected]>
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Jan 26, 2022
@mdelapenya
Merge branch 'main' into 1781-rename-to-aarch64
7ecd99f 
* main: (45 commits)
  feat: add CentOS 8 support (elastic#2034)
  fix: set default region for AWS cli (elastic#2053)
  chore: use Ansible's built-in replace instead of sed (elastic#2048)
  chore: split stack configuration and start into two tasks (elastic#2044)
  feat: enable SSH access to users for debugging cloud instances (elastic#2001)
  fix: use the right branch for 7.17 backports (elastic#2025)
  SLES15 enablement (elastic#2007)
  chore: bump stale agent for main (elastic#2014)
  Update `fetchBeatsBinary` to be reused in elastic-agent-poc (elastic#1984)
  chore: add resiliency when provisioning the stack (elastic#1990)
  chore: bump elastic-package to v0.32.1 (elastic#1959)
  feat: export Fetch&Download methods in the /pkg directory (elastic#1943)
  bump stack version 8.1.0-dbc834fd (elastic#1948)
  bump stack version 8.1.0-76902d39 (elastic#1946)
  chore: retire 7.15 adding 7.17 (elastic#1938)
  ci: use withAPMEnv (elastic#1917)
  Update main branch (elastic#1928)
  bump stack version 8.1.0-befff95a (elastic#1929)
  chore: properly evaluate how tests are skipped on CI when checking modified files (elastic#1924)
  bump stack version 8.1.0-60bffc32 (elastic#1921)
  ...
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Feb 28, 2022
@mdelapenya
feat: export Fetch&Download methods in the /pkg directory (elastic#1943)
056bfe5 
* chore: copy internal's version files into the exported pkg public package

* chore: run unit tests from top-level goal

* chore: use public package instead of the internal one

* chore: remove old internal files for habdling downloads

* fix: update missing references
@mdelapenya mdelapenya mentioned this pull request Feb 28, 2022
Merged
8 tasks
mdelapenya added a commit that referenced this pull request Feb 28, 2022
@mdelapenya @v1v @narph
chore: 7.17 backports (#2187)
9884dcd 
* ci: use withAPMEnv (#1917)

* feat: partial backport for (#1628)

* chore: define variable for BeatsLocalPath

* fix: update path initialisation in unit tests

* chore: pass inline env vars to commands

Co-authored-by: Victor Martinez <[email protected]>

Co-authored-by: Victor Martinez <[email protected]>

* feat: export Fetch&Download methods in the /pkg directory (#1943)

* chore: copy internal's version files into the exported pkg public package

* chore: run unit tests from top-level goal

* chore: use public package instead of the internal one

* chore: remove old internal files for habdling downloads

* fix: update missing references

* Update `fetchBeatsBinary` to be reused in elastic-agent-poc (#1984)

* update func

* fix path

* work on download

* small fix

* remove test

* add sha to google

* fix typo

* add comment

* ci: increase log rotation (#2138)

* chore: increase build timeout to 90 minutes (#2139)

* feat: support downloading project artifacts for the new bucket layout (#2172)

* feat: support for the new bucket structure

* feat: calculate bucket URLs using resolvers

* chore: support for legacy Beats

Old PRs needs to be tested and the binaries could not exist in the new layout

* chore: support looking up the bucket in up-to 3 locations

1. new project layout
2. new beats layout
3. legacy beats layout

* chore: simplify boolean logic

* fix: pass worker environment for versions to the VMs

* fix: predefine kibana version

* fix: remove extra char

* fix: remove extra char

* chore: log resolvers

* chore: use new project method when fetching artifacts

* fix: pass stack version to stack deployment

* fix: use artifact to check for ubi8 variant

* chore: improve logs in resolvers

* fix: apply variants to project resolver

Ubi8 is a valid variant for elastic-agent

* chore: use Stack version for the dependant beats

In the running-on-top-of-beats feature file, the code installs some Beats
and we do not want to use the given SHA, because elastic-agent now lives
in a separate repo, therefore they do not share commit SHAs

* fix: use fleet-ci bucket for elastic-agent and fleet-server

* chore: look up both CI buckets

* chore: make useCISnapshots configurable on downloads

* chore: background processes should not use CI snapshots

* feat: separate elastic-agent version from beats version

* chore: extract useCISnapshots to a function

* chore: use Github_Check_Repo to identify where the Git commit lives

* fix: k8s-autodiscover lives in both worlds

* fix: missing package

* fix: use snapshot version in k8s-autodiscover

* fix: missing import

Co-authored-by: Victor Martinez <[email protected]>
Co-authored-by: Mariana Dima <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@narph narph narph approved these changes

@adam-stokes adam-stokes adam-stokes approved these changes

Assignees

@mdelapenya mdelapenya

Labels
backport-v7.16.0 Automated backport with mergify backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Expose the functionality to download Beats binaries in a public package
4 participants
@mdelapenya @elasticmachine @adam-stokes @narph