Migrate Custom threshold rule tests to deployment agnostic solution

[mgiota]

Part of #183263

🍒 Summary

As part of this #187924 we introduced a way to write deployment agnostic tests using the deployment agnostic framework from Appex QA team. We want to migrate the existing metric threshold rule tests to the new agnostic framework.

Existing tests

Test	Type	ESS	Serverless	Notes
Custom threshold rule type	Alerting	✅	✅	Duplicate tests Serverless: x-pack/test_serverless/api_integration/test_suites/observability/custom_threshold_rule ESS: x-pack/test/alerting_api_integration/observability/custom_threshold_rule

New folder structure

How to migrate

In this PR oblt.stateful.config.ts and oblt.serverless.config.ts were added, so no need to create any config file.

Here's what you have to do:

• Move Custom threshold rule tests from x-pack/test/alerting_api_integration/observability/custom_threshold_rule within x-pack/test/api_integration/deployment_agnostic/apis/observability/alerting folder
• Load the related test files in this file
  loadTestFile(require.resolve('./custom_threshold...');
• Delete the Custom threshold rule test from the old location (Both serverless and stateful)
• Use the DeploymentAgnosticFtrProviderContext
• use a combination of supertestWithoutAuth and samlAuth to generate an API key for user roles and make API calls

const samlAuth = getService('samlAuth');
const supertestWithoutAuth = getService('supertestWithoutAuth');

• Use the config service to decide on the expected consumer

const config = getService('config');
const isServerless = config.get('serverless');
const expectedConsumer = isServerless ? 'observability' : 'logs'; // Need to verify if the consumer on stateful is correct

How to run

To run serverless

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="Custom Threshold rule"

To run stateful

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="Custom Threshold rule"

[maryam-saeidi]

Moved to backlog until the related Meta ticket is prioritized.

[jasonrhodes]

I think we need to update these instructions so they are up to date with the latest on deployment agnostic testing before this issue is "Ready", but then we need to pull this in ASAP.

[maryam-saeidi]

Example PR: #187924