[R&D] Stateful/stateless test harness options for observability

[jasonrhodes]

Currently to run tests in both a stateful and serverless context, the current recommendation is to "copy the tests that should run in stateful and serverless", which requires teams to maintain those files separately, as well. We would like the ability to tell CI to run a selection of tests in both contexts, rather than needing to copy/paste the files.

This issue is a time-boxed R&D for someone to look into whether we can create some kind of simple "test harness" wrapper for running a selection of our functional tests in both contexts, that would work for our observability purposes.

Existing Tests

Here's the current test coverage for the SLO and o11y alerting features for both serverless and stateful environments:

API Integration tests

Test	Type	ESS	Serverless	Notes
Custom threshold rule type	Alerting	✅	✅	Duplicate tests Serverless: x-pack/test_serverless/api_integration/test_suites/observability/custom_threshold_rule ESS: x-pack/test/alerting_api_integration/observability/custom_threshold_rule
Metric threshold rule type	Alerting	✅	❌	ESS: x-pack/test/alerting_api_integration/observability/metric_threshold_rule.ts
ES query rule type	Alerting	❌	✅	Serverless: x-pack/test_serverless/api_integration/test_suites/observability/es_query_rule
Burn rate rule type	Alerting	❌	✅	Serverless: x-pack/test_serverless/api_integration/test_suites/observability/burn_rate_rule
SLO tests	SLO	✅	✅	Different implementation in both environments. Serverless: x-pack/test_serverless/api_integration/test_suites/observability/slos ESS: x-pack/test/api_integration/apis/slos

Functional tests

Test	Type	ESS	Serverless	Notes
Rules page	Alerting	✅	❌	ESS: x-pack/test/observability_functional/apps/observability/pages/rules_page.ts
Rule details page	Alerting	✅	❌	ESS: x-pack/test/observability_functional/apps/observability/pages/rule_details_page.ts
Alerts page	Alerting	✅	❌	ESS: x-pack/test/observability_functional/apps/observability/pages/alerts
Overview page alerts table	Alerting	✅	❌	ESS: x-pack/test/observability_functional/apps/observability/pages/overview/alert_table.ts
SLO embeddable tests	SLO	✅	❌	ESS: x-pack/test/functional/apps/slo/embeddables

Our initial approach using Mocha tagging

We are going to write our tests in a new location and we will use ess & serverless configurations to specify environment specific options like extra services and Mocha options. Test suites and cases should be prefixed with specific tags to determine their execution in particular environments (@ess @serverless) or to exclude them from specific environments (@skipInServerless @skipInEss).

Test case example

describe('@ess @serverless SLO burn rate rule, () => {
  describe('Create rule', () => { 
  });

 describe('@skipInServerless missing something', () => { 
 });
}

Configuration examples

Tests tagged with @serverless are included, whereas tests tagged with @skipInServerless are excluded

 mochaOpts: {
    grep: '/^(?!.*@skipInServerless).*@serverless.*/',
 },

Tests tagged with @ess are included, whereas tests tagged with @skipInEss are excluded

mochaOpts: {
    grep: '/^(?!.*@skipInEss).*@ess.*/',
}

Final implementation using suiteTags

As per Platform's recommendation, FTR tests already make use of the it.tags(['my-tag-1', 'my-tag-2']) pattern, so we would like to stay with that pattern rather than introducing a new type of mocha tag in the it block's "description" as it was introduced in this POC. The main trade-off for this is that this tagging is only available at the suite level ("describe") and not individual test level ("it"). However, we consider this not a blocker since tests within one suite are usually all good for the same environment and the few exceptions can easily be handled.

Adding following in the config files:

serverless config

suiteTags: {
        include: ['serverless'],
        exclude: ['skipInServerless'],
      },

ess config

suiteTags: {
        include: ['ess'],
        exclude: ['skipInEss'],
      },

and then adding this.tags(['serverless', 'ess']) in the test suite instructs the test runner to run the same test suite in both environments.

Todo

• Get feedback from AppEx-qa and other solution teams
  • AppEx-qa: @pheyos
  • Security team: @MadameSheema @banderror
• Make a list of current functional and api integration tests for SLO and o11y alerting features
• Make a POC of reusing API integration (FTR) tests from x-pack/test_serverless/api_integration/test_suites/observability/burn_rate_rule/burn_rate_rule.ts
  • [SLO] POC Platform agnostic tests #182556
  • [Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113
• Make a plan to migrate existing tests to the new location
  • Integration tests([Meta] Migrate SLO/Alerting/Synthetics api integration tests to deployment agnostic solution #183263)
  • Functional tests
• Make a POC of deployment agnostic functional tests
  • Reuse Security's solution using Cypress
  • Investigate how we can reuse FTR tests

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[simianhacker]

The approach we could take is to create an abstraction that basically injects dependencies for each environment (Stateless vs Stateful), then we would refactor our tests to use the abstractions instead of writing tests in both places.

[paulb-elastic]

Linking #162698

[jasonrhodes]

We now have a plan for deployment agnostic testing. Thanks for all of this work and collaboration, @mgiota and everyone else!

[jasonrhodes]

Re-opening this so it can be closed by the PR in review.

[mgiota]

Closing this, PR is already merged! Here's the ticket for the migration plan.