[POC][tests-only][full-ci] run e2e tests with runner

[saw-jan]

Description

To better organize the growing e2e tests, we have to move features to their favorable suites (folders). Also, we have to be careful about the execution time of each e2e pipeline. With the current approach and feature file location it will be difficult to organize and manage the pipeline execution time.

Therefore, here is my proposal:

1. Move feature files into different suites (folders) and maintain this location: features/<suite>/*.feature
   • move folders/suites out of smoke suite

   features/smoke/shares -> features/shares
   features/smoke/search -> features/search
   ...
   

2. Use runner script to run the e2e tests
   • doesn't affect the current way of running e2e tests
   • organizing suites in CI is simplified

   -"featurePaths": [
   -    "tests/e2e/cucumber/features/{smoke,journeys}/*.feature",
   -]
   +"suites": [
   +    "journeys",
   +    "smoke",
   +]

   • ability to split and run existing suites into groups (with this ocis ci doesn't have to change anything if some suites are added to the web repo)

Run all suites

./run-e2e.sh

Run test providing paths

./run-e2e.sh cucumber/features/journeys cucumber/features/search/search.feature

Run single scenario

./run-e2e.sh cucumber/features/search/search.feature:122

Run selective suites

./run-e2e.sh --suites shares,spaces

Exclude suites

./run-e2e.sh --xsuites app-provider,search

Divide and run suites into parts (suitable for splitting e2e in oCIS ci)

./run-e2e.sh --run-part 2 --total-parts 4

./run-e2e.sh --run-part 1 --total-parts 4 --exclude app-provider

Help

$ ./run-e2e.sh -h

COMMAND [options] [paths]

Available options:
    --suites        - suites to run. Comma separated values (folder names)
                      e.g.: --suites smoke,shares
    --xsuites       - exclude suites from running. Comma separated values
                      e.g.: --xsuites spaces,search
    --run-part      - part to run out of total parts (groups)
                      e.g.: --run-part 2 (runs part 2 out of 4)
    --total-parts   - total number of groups to divide into
                      e.g.: --total-parts 4 (suites will be divided into 4 groups)
    --help, -h      - show cli options

Motivation and Context

Make it easier to add/skip/organize e2e test suites in a pipeline

How Has This Been Tested?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests
• Documentation
• Maintenance (e.g. dependency updates or tooling)

[saw-jan]

Some reviews have been done in #10557

[ScharfViktor]

LGTM

[SagarGi]

Wow this seems really cool to me as well. LGTM 👍

[SagarGi]

@saw-jan one thing that can be added but not necessary. can we have some color coded text for "ERR:" may be red?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud