chore: refactor Fleet upgrade tests #671

mdelapenya · 2021-01-27T19:06:48Z

What does this PR do?

We are refactoring the installer process to support setting the version of the installer package, which will allow us to consume different artifacts at different places of the test suite (i.e. the upgrade test).

we are adding a few helper Make goals representing different use cases when running the tests:
- Run the E2E tests for Fleet test suite
- Run the E2E tests for Fleet test suite using CI snapshots
- Run the E2E tests for Fleet test suite using CI snapshots for a pull request
- Run the E2E tests for Fleet test suite in nightly, testing the stale version
- Run the E2E tests for Fleet test suite in nightly, testing the stale version, using CI snapshots

We do not consider running the upgrade tests against a PR, that's why there is no use case for the nightly tests for PR and CI snapshots.

we are bumping the stale version from 7.10.1 to 7.10.2, because there is not CI snapshot for 7.10.1 under the GCP bucket.
we are refactoring the creation of the installer struct including version and stale state.
we detected that the SNAPSHOT suffix should be added in certain cases when using the CI snapshots.
we are making a stronger check when flattening the layout for the TAR installer, ensuring it's clean in the container: we remove the target dir.
updated the unit tests to generate the GCP paths in different use cases
we are adding the @nightly tag to the upgrade test, so that it's executed in the nightly execution. See feat: support executing nightly tests on CI #669

Why is it important?

In one sentence, the main goal is to cover all use cases.

The upgrade test created an installer in the stale state, but the current source code did not know about stale state and always used the agentVersion passed as env var, failing the test executions under the use cases described in this PR. Here we are honouring the version of the stale agent, keeping old behavior as it was.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

Verified multiple use cases, as described above.
Want to check that the developer experience is clear and concise.

How to test this PR locally

These are the tested use cases, as described above:

make -C e2e fleet-fleet
make -C e2e fleet-fleet-ci-snapshots
make -C e2e fleet-fleet-pr-ci-snapshots
make -C e2e fleet-nightly
make -C e2e fleet-nightly-ci-snapshots

Related issues

Do not merge until feat: support executing nightly tests on CI #669 is merged

Follow-ups

I'd like to check with the Agent team about the consequences of running the upgrade tests on PRs

- e2e-fleet-nightly: run the nightly tests not using CI snapshots. It should download the binaries from the official artifactory. - e2e-fleet-nightly-ci-snapshots: run the nightly tests using the CI snapshots for a specific SHA commit from Beats, downloading them from a GCP bucket

This will allow selecting the proper binary, depending if we are using a stale agent or a regular one.

mdelapenya · 2021-01-27T19:10:44Z

e2e/_suites/fleet/ingest-manager_test.go

@@ -45,7 +45,7 @@ var agentVersion = agentVersionBase

 // agentStaleVersion is the version of the agent to use as a base during upgrade
 // It can be overriden by ELASTIC_AGENT_STALE_VERSION env var. Using latest GA as a default.
-var agentStaleVersion = "7.10.0"
+var agentStaleVersion = "7.10.2"


Pairing version with Jenkinsfile

mdelapenya · 2021-01-27T19:13:15Z

e2e/_suites/fleet/installers.go

-
-		return err
+	cmds := [][]string{
+		[]string{"rm", "-fr", "/elastic-agent"},


When running all Fleet tests in my local machine, I saw multiple errors when moving the agent, because mv will refuse to rename a directory to another directory if the target directory contains files.

It does not happen on CI but in my local machine happens after multiple runs. I'm adding it defensively, as it's harmless.

mdelapenya · 2021-01-27T19:17:54Z

e2e/_suites/fleet/services.go

@@ -163,7 +163,7 @@ func runElasticAgentCommand(profile string, image string, service string, proces
 // be defined by that value
 // Else, if the environment variable BEATS_USE_CI_SNAPSHOTS is set, then the artifact
 // to be downloaded will be defined by the latest snapshot produced by the Beats CI.
-func downloadAgentBinary(artifact string, version string, OS string, arch string, extension string) (string, string, error) {
+func downloadAgentBinary(artifact string, version string, OS string, arch string, extension string, stale bool) (string, string, error) {


As an improvement in a follow-up PR, I'm interested in creating a BinaryRequest struct to hold the references needed to download the binary, which would simplify the signature of the method.

elasticmachine · 2021-01-27T19:29:06Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #671 updated
- Start Time: 2021-02-02T10:48:17.512+0000
Duration: 32 min 13 sec
Commit: 81953d0

Test stats 🧪

Test	Results
Failed	7
Passed	108
Skipped	19
Total	134