Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

chore: refactor build artifacts methods (#689) backport for 6.8.x #712

Closed
wants to merge 1 commit into from

Conversation

mdelapenya
Copy link
Contributor

Backports the following commits to 6.8.x:

* feat: use local beats directory for elastic-agent installers

* docs: document BEATS_LOCAL_PATH variable

* chore: check that the local file exists

* chore: add unit tests for the BEATS_LOCAL_PATH variable

* chore: simplify logic after test coverage

* chore: enrich test name

* chore: rename test

* chore: move default values to the end of the descriptions

* chore: add a helper method to build artifact names

* chore: build artifact name only once

* chore: pass the entire filename to calculate the GCP bucket coordinates

* chore: simplify method signature, as the file name is not changed within the method

* chore: use buildArtifactName method

* chore: pass filename to helper methods to avoid unnecessary calculations

* chore: rename variable for consistency

* chore: add license headers

* chore: use lowercase comparison for artifact name

* chore: build artifact changes from Beats CI to Elastic's artifactory

* fix: support building artifact names from local

Local builds uses the same as the CI

* WIP

* fix: rename docker-ubi8 installer key

* chore: pass a fallback version when building artifact name

In the case we are using a PR , where the version is "pr-12345", then we
need to use the base version of the product

* chore: remove log

* fix: do not override artifact name in Docker installer

* feat: support consuming the docker images from snapshots or local in standalone mode

* chore: add unit tests for fetching docker images from local Beats repo

* chore: apply version to Docker installer

* chore: support loading metricbeat image from local repository

* chore: exrtact loadImage method to docker helper

* feat: support consuming CI artifacts in metricbeat

* fix: remove non-existent field from logrus log

* chore: extract download logic to helper methods

We are discarding the stale parameter, because the stale version is set
when the installer is required, so we simply download what is requested

* chore: fix typo

* chore: check agent version before setting it

* chore: use the docker client to load the image instead of the docker binary

* chore: add unit tests for check method

* chore: pass fallback version to check method

* fix: properly calculate version for PRs

* chore: move PR check logic to a method

* chore: make test independent of maintenance branch
# Conflicts:
#	e2e/_suites/fleet/fleet.go
#	e2e/_suites/fleet/ingest-manager_test.go
#	e2e/_suites/fleet/installers.go
#	e2e/_suites/fleet/services.go
#	e2e/_suites/fleet/services_test.go
@mdelapenya mdelapenya self-assigned this Feb 8, 2021
@elasticmachine
Copy link
Contributor

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #712 opened

  • Start Time: 2021-02-08T16:57:56.611+0000

  • Duration: 3 min 25 sec

  • Commit: 156d301

Test stats 🧪

Test Results
Failed 0
Passed 38
Skipped 9
Total 47

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 2

Expand to view the steps failures

Build and test
  • Took 1 min 32 sec . View more details on here
  • Description: .ci/scripts/build-test.sh
Archive the artifacts
  • Took 0 min 0 sec . View more details on here
  • Description: [2021-02-08T17:01:14.955Z] Archiving artifacts script returned exit code 2

Log output

Expand to view the last 100 lines of log output

[2021-02-08T17:01:08.898Z] PASS TestGetGCPBucketCoordinates_PullRequests (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestGetGCPBucketCoordinates_Snapshots/Fetching_snapshots_bucket_for_RPM_package (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestGetGCPBucketCoordinates_Snapshots/Fetching_snapshots_bucket_for_DEB_package (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestGetGCPBucketCoordinates_Snapshots/Fetching_snapshots_bucket_for_TAR_package_adds_OS_to_fileName_and_object (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestGetGCPBucketCoordinates_Snapshots (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestProcessBucketSearchPage_PullRequestsFound (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestProcessBucketSearchPage_PullRequestsNotFound (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestProcessBucketSearchPage_SnapshotsFound (0.00s)
[2021-02-08T17:01:08.898Z] PASS TestProcessBucketSearchPage_SnapshotsNotFound (0.00s)
[2021-02-08T17:01:08.898Z] PASS .
[2021-02-08T17:01:08.898Z] 
[2021-02-08T17:01:08.898Z] DONE 36 tests in 6.217s
[2021-02-08T17:01:08.898Z] cd _suites && gotestsum --format testname -- -count=1 -timeout=5m ./...
[2021-02-08T17:01:08.898Z] go: downloading github.com/cucumber/godog v0.11.0
[2021-02-08T17:01:08.898Z] go: downloading github.com/cucumber/messages-go/v10 v10.0.3
[2021-02-08T17:01:08.898Z] go: downloading github.com/gofrs/uuid v3.3.0+incompatible
[2021-02-08T17:01:08.898Z] go: downloading github.com/cucumber/gherkin-go/v11 v11.0.0
[2021-02-08T17:01:08.898Z] go: downloading github.com/hashicorp/go-memdb v1.3.0
[2021-02-08T17:01:08.898Z] go: downloading github.com/hashicorp/go-immutable-radix v1.3.0
[2021-02-08T17:01:08.898Z] go: downloading github.com/hashicorp/golang-lru v0.5.4
[2021-02-08T17:01:11.453Z] # github.com/elastic/e2e-testing/e2e/_suites/fleet [github.com/elastic/e2e-testing/e2e/_suites/fleet.test]
[2021-02-08T17:01:11.453Z] fleet/fleet.go:166:8: fts.Version undefined (type *FleetTestSuite has no field or method Version)
[2021-02-08T17:01:11.453Z] fleet/fleet.go:167:58: fts.Version undefined (type *FleetTestSuite has no field or method Version)
[2021-02-08T17:01:11.453Z] fleet/services.go:224:10: no new variables on left side of :=
[2021-02-08T17:01:11.453Z] fleet/services.go:290:10: no new variables on left side of :=
[2021-02-08T17:01:11.453Z] fleet/services.go:431:10: no new variables on left side of :=
[2021-02-08T17:01:11.453Z] WARN invalid TestEvent: FAIL	github.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]
[2021-02-08T17:01:11.453Z] bad output from test2json: FAIL	github.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]
[2021-02-08T17:01:12.831Z] testing: warning: no tests to run
[2021-02-08T17:01:12.831Z] PASS _suites/helm
[2021-02-08T17:01:13.090Z] testing: warning: no tests to run
[2021-02-08T17:01:13.090Z] PASS _suites/metricbeat
[2021-02-08T17:01:13.090Z] 
[2021-02-08T17:01:13.090Z] === Errors
[2021-02-08T17:01:13.090Z] fleet/fleet.go:166:8: fts.Version undefined (type *FleetTestSuite has no field or method Version)
[2021-02-08T17:01:13.090Z] fleet/fleet.go:167:58: fts.Version undefined (type *FleetTestSuite has no field or method Version)
[2021-02-08T17:01:13.090Z] fleet/services.go:224:10: no new variables on left side of :=
[2021-02-08T17:01:13.090Z] fleet/services.go:290:10: no new variables on left side of :=
[2021-02-08T17:01:13.090Z] fleet/services.go:431:10: no new variables on left side of :=
[2021-02-08T17:01:13.090Z] 
[2021-02-08T17:01:13.090Z] DONE 0 tests, 5 errors in 5.053s
[2021-02-08T17:01:13.090Z] Makefile:92: recipe for target 'unit-test' failed
[2021-02-08T17:01:13.090Z] make: *** [unit-test] Error 2
[2021-02-08T17:01:13.090Z] make: Leaving directory '/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/src/github.com/elastic/e2e-testing/e2e'
[2021-02-08T17:01:13.915Z] + export PATH=/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/.gvm/versions/go1.14.12.linux.amd64/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/src/github.com/elastic/e2e-testing/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/src/github.com/elastic/e2e-testing/.ci/scripts
[2021-02-08T17:01:13.915Z] + set +e
[2021-02-08T17:01:13.915Z] + find /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/.cache/pre-commit -type d -name scripts
[2021-02-08T17:01:13.915Z] + grep .ci/scripts
[2021-02-08T17:01:13.915Z] + searchFile=/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/.cache/pre-commit/repottZtHZ/.ci/scripts
[2021-02-08T17:01:13.915Z] + [ -e /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/.cache/pre-commit/repottZtHZ/.ci/scripts ]
[2021-02-08T17:01:13.915Z] + export PATH=/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/.gvm/versions/go1.14.12.linux.amd64/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/src/github.com/elastic/e2e-testing/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/bin:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/src/github.com/elastic/e2e-testing/.ci/scripts:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712/.cache/pre-commit/repottZtHZ/.ci/scripts
[2021-02-08T17:01:13.915Z] + set -e
[2021-02-08T17:01:13.915Z] + git diff-tree --no-commit-id --name-only -r 156d301cfdec6dc702e2c5a8d7c901d5e5b089f2
[2021-02-08T17:01:13.915Z] + xargs pre-commit run --files
[2021-02-08T17:01:13.915Z] + tee pre-commit.out
[2021-02-08T17:01:14.174Z] Check for case conflicts.................................................Passed
[2021-02-08T17:01:14.174Z] Check that executables have shebangs.................(no files to check)Skipped
[2021-02-08T17:01:14.174Z] Check JSON...........................................(no files to check)Skipped
[2021-02-08T17:01:14.174Z] Check for merge conflicts................................................Passed
[2021-02-08T17:01:14.174Z] Check Yaml...........................................(no files to check)Skipped
[2021-02-08T17:01:14.174Z] Check Xml............................................(no files to check)Skipped
[2021-02-08T17:01:14.174Z] Fix End of Files.........................................................Passed
[2021-02-08T17:01:14.433Z] go fmt...................................................................Passed
[2021-02-08T17:01:14.433Z] go lint..................................................................Passed
[2021-02-08T17:01:14.433Z] Check files aren't using go's testing package........(no files to check)Skipped
[2021-02-08T17:01:14.535Z] Post stage
[2021-02-08T17:01:14.558Z] Recording test results
[2021-02-08T17:01:14.930Z] [Checks API] No suitable checks publisher found.
[2021-02-08T17:01:14.955Z] Archiving artifacts
[2021-02-08T17:01:15.146Z] Failed in branch Unit Tests
[2021-02-08T17:01:16.338Z] golangci-lint............................................................Passed
[2021-02-08T17:01:16.338Z] check-bash-syntax....................................(no files to check)Skipped
[2021-02-08T17:01:16.338Z] check-jenkins-pipelines..............................(no files to check)Skipped
[2021-02-08T17:01:16.338Z] - hook id: check-jenkins-pipelines
[2021-02-08T17:01:16.338Z] check-unicode-non-breaking-spaces........................................Passed
[2021-02-08T17:01:16.338Z] remove-unicode-non-breaking-spaces.......................................Passed
[2021-02-08T17:01:16.338Z] check-en-dashes..........................................................Passed
[2021-02-08T17:01:16.597Z] remove-en-dashes.........................................................Passed
[2021-02-08T17:01:16.597Z] check-jjbb...........................................(no files to check)Skipped
[2021-02-08T17:01:16.597Z] - hook id: check-jjbb
[2021-02-08T17:01:16.597Z] check-gherkin-lint...................................(no files to check)Skipped
[2021-02-08T17:01:16.603Z] $ ssh-agent -k
[2021-02-08T17:01:16.606Z] unset SSH_AUTH_SOCK;
[2021-02-08T17:01:16.607Z] unset SSH_AGENT_PID;
[2021-02-08T17:01:16.607Z] echo Agent pid 4439 killed;
[2021-02-08T17:01:16.672Z] [ssh-agent] Stopped.
[2021-02-08T17:01:16.843Z] Archiving artifacts
[2021-02-08T17:01:17.037Z] Recording test results
[2021-02-08T17:01:17.343Z] [Checks API] No suitable checks publisher found.
[2021-02-08T17:01:18.646Z] Stage "Build Docs" skipped due to earlier failure(s)
[2021-02-08T17:01:18.741Z] Stage "End-To-End Tests" skipped due to earlier failure(s)
[2021-02-08T17:01:18.843Z] Stage "Release" skipped due to earlier failure(s)
[2021-02-08T17:01:20.160Z] Running on worker-395930 in /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-712
[2021-02-08T17:01:20.235Z] [INFO] getVaultSecret: Getting secrets
[2021-02-08T17:01:20.333Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-08T17:01:22.666Z] + chmod 755 generate-build-data.sh
[2021-02-08T17:01:22.666Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-712/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-712/runs/1 FAILURE 204614
[2021-02-08T17:01:22.666Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-712/runs/1/steps/?limit=10000 -o steps-info.json
[2021-02-08T17:01:24.932Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-712/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-02-08T17:01:24.932Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-712/runs/1/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 38
Skipped 9
Total 47

@mdelapenya
Copy link
Contributor Author

One backport is missing in this branch. Closing and investigating

@mdelapenya mdelapenya closed this Feb 8, 2021
@mdelapenya
Copy link
Contributor Author

Superseded by #713

@mdelapenya mdelapenya deleted the backport/6.8.x/pr-689 branch June 2, 2021 05:38
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants