Our goal is for the Fleet team to execute this automated e2e test suite while developing the product. The tests in this folder assert that the use cases (or scenarios) defined in the features directory are behaving as expected.
At the topmost level, the test framework uses a BDD framework written in Go, where we set the expected behavior of use cases in a feature file using Gherkin, and implementing the steps in Go code. The provisining of services is accomplish using Docker Compose and the testcontainers-go library.
The tests will follow this general high-level approach:
- Install runtime dependencies as Docker containers via Docker Compose, happening at before the test suite runs. These runtime dependencies are defined in a specific
profilefor Fleet, in the form of adocker-compose.ymlfile. - Execute BDD steps representing each scenario. Each step will return an Error if the behavior is not satisfied, marking the step and the scenario as failed, or will return
nil.
Because this framework uses Docker as the provisioning tool, all the services are based on Linux containers. That's why we consider this tool very suitable while developing the product, but would not cover the entire support matrix for the product: Linux, Windows, Mac, ARM, etc.
For Windows or other platform support, we should build Windows images and containers or, given the cross-platform nature of Golang, should add the building blocks in the test framework to run the code in the ephemeral CI workers for the underlaying platform.
The first step in determining the exact failure is to try and reproduce the test run locally, ideally using the DEBUG log level to enhance the log output. Once you've done that, look at the output from the test run.
It's important to configure Docker for Mac to allow it accessing the /var/folders directory, as this framework uses Mac's default temporary directory for storing tempoorary files.
To change it, please use Docker UI, go to Preferences > Resources > File Sharing, and add there /var/folders to the list of paths that can be mounted into Docker containers. For more information, please read https://docs.docker.com/docker-for-mac/#file-sharing.
-
Clone this repository, say into a folder named
e2e-testing.git clone [email protected]:elastic/e2e-testing.git
-
Configure the version of the product you want to test (Optional).
This is an example of the optional configuration:
# There should be a Docker image for the runtime dependencies (elasticsearch, package registry)
export STACK_VERSION=8.0.0-SNAPSHOT
# There should be a Docker image for the runtime dependencies (kibana)
export KIBANA_VERSION=pr12345
# (Fleet mode) This environment variable will use a fixed version of the Elastic agent binary, obtained from
# https://artifacts-api.elastic.co/v1/search/8.0.0-SNAPSHOT/elastic-agent
export ELASTIC_AGENT_DOWNLOAD_URL="https://snapshots.elastic.co/8.0.0-59098054/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz"
# (Fleet mode) This environment variable will use the snapshots produced by Beats CI. If the above variable
# is set, this variable will take no effect
export BEATS_USE_CI_SNAPSHOTS="true"
# (Stand-Alone mode) This environment variable will use the its value as the Docker tag produced by Beats CI (Please look up Google Cloud Storage CI bucket).
export BEAT_VERSION="78a762c76080aafa34c52386341b590dac24e2df"-
Define the proper Docker images to be used in tests (Optional).
Update the Docker compose files with the local version of the images you want to use.
TBD: There is an initiative to automate this process to build the Docker image for a PR (or the local workspace) before running the tests, so the image is ready.
-
Install dependencies.
- Install Go:
https://golang.org/doc/install(The CI uses GVM) - Install integrations
make -C e2e sync-integrations - Install godog (from project's root directory):
make -C e2e install-godog
- Install Go:
-
Run the tests.
If you want to run the tests in Developer mode, which means reusing bakend services between test runs, please set this environment variable first:
# It won't tear down the backend services (ES, Kibana, Package Registry) or agent services after a test suite. export DEVELOPER_MODE=true
cd e2e/_suites/fleet OP_LOG_LEVEL=DEBUG godogThe tests will take a few minutes to run, spinning up a few Docker containers representing the various products in this framework and performing the test steps outlined earlier.
As the tests are running they will output the results in your terminal console. This will be quite verbose and you can ignore most of it until the tests finish. Then inspect at the output of the last play that ran and failed. On the contrary, you could use a different log level for the
OP_LOG_LEVELvariable, being it possible to useDEBUG,INFO (default),WARN,ERROR,FATALas log levels.
This type of failure usually indicates that code for these tests itself needs to be changed.
See the sections below on how to run the tests locally.
Check if the scenario has an annotation/tag supporting the test runner to filter the execution by that tag. Godog will run those scenarios. For more information about tags: https://github.com/cucumber/godog/#tags
OP_LOG_LEVEL=DEBUG godog -t '@annotation'Example:
OP_LOG_LEVEL=DEBUG godog -t '@stand_alone_mode'Sometimes the tests could fail to configure or start a product such as Metricbeat, Elasticsearch, etc. To determine why
this happened, look at your terminal log in DEBUG mode. If a docker-compose.yml file is not present please execute this command:
## Will remove tool's existing default files and will update them with the bundled ones.
make clean-workspaceIf you see the docker images are outdated, please execute this command:
## Will refresh stack images
make clean-dockerNote what you find and file a bug in the elastic/e2e-testing repository, requiring a fix to the Fleet suite to properly configure and start the product.
Please open an issue here: https://github.com/elastic/e2e-testing/issues/new