[Data Discovery] Add Serverless functional test coverage

[drewdaemon]

Describe the feature:
Identify key user flows and create a fundamental suite of functional tests following instructions in the Kibana Serverless e2e Test Guide (Google docs).

Plan

Progress

• Migrate functional tests [Data Discovery] Run Data Discovery functional tests in Serverless #164656
• Migrate API integration tests: [Data Discovery] Run API integration tests in Serverless #163550
• Migrate example plugin tests: [Data Discovery] Run example plugin functional tests in Serverless #163411

Followup work to deduplicate tests: #164911.

Functional tests

We have a couple of options depending on if we want to have broader test coverage in Serverless by duplicating existing tests or just stick to basic smoke test coverage of shared functionality in Serverless and focus more on testing differences between stateful and Serverless.

Note

We decided on the "Duplicating existing tests" approach below in order to provide more thorough coverage in Serverless initially, and since deployment agnostic tests will eventually be supported by ​​#161574, which will allow us to share these tests.

Duplicating existing tests

This approach would involve copying a number of existing functional tests that cover major functionality in our applications to test_serverless and modifying them in place to run in Serverless. This would likely give us the best test coverage overall, but comes with the drawback of having to maintain two copies of the duplicated test files independently until support for sharing tests between stateful and Serverless is supported (see ​​#161574).

Recommended tests (in rough order of importance – we may not get to all of them)

• Discover
  • test/functional/apps/discover/
    • group1/_discover.ts
    • group1/_discover_histogram.ts
    • embeddable/_saved_search_embeddable.ts
    • group2/_adhoc_data_views.ts
    • group2/_data_grid_doc_navigation.ts
    • group2/_data_grid_doc_table.ts
    • group3/_request_counts.ts
    • group3/_sidebar.ts
  • test/functional/apps/context/
    • _context_navigation.ts
    • _discover_navigation.ts
    • _filters.ts
    • _size.ts
  • test/functional/apps/management/data_views/
    • _data_view_create_delete.ts
    • _exclude_index_pattern.ts
    • _runtime_fields.ts
    • _runtime_fields_composite.ts
    • _edit_field.ts
  • x-pack/test/functional/apps/discover/
    • reporting.ts
    • Visualize_field.ts
  • x-pack/test/functional_with_es_ssl/apps/discover_ml_uptime/discover/search_source_alert.ts
  • CCS:
    • x-pack/test/stack_functional_integration/apps/ccs/ccs_discover.js
    • test/functional/apps/management/ccs_compatibility/_data_views_ccs.ts

Basic smoke testing

With this approach we would write a few high level smoke tests to cover the main functionality of our applications and focus additional efforts on testing differences between stateful and Serverless. The smoke tests we write should focus on functionality that interacts with ES (as opposed to just UI tests) since the differences between stateful and stateless ES are where we’re most likely to encounter issues (otherwise our applications should currently run identically between stateful and Serverless).

Smoke tests

• Discover:
  • Navigate to Discover.
  • Switch to a non-default data view.
  • Create and execute a search including query, filters, and time range.
  • Add some columns to the table.
  • Change the sort column and direction.
  • Set a histogram breakdown field and change the histogram time interval.
  • Save the current search as a new saved search
  • Click the “New” button to clear the current saved search.
  • Open the previously created saved search.
  • Verify the search results in grid, total hits, and histogram.
  • Open the field list popover for a field that supports field statistics and verify it works as expected.
  • Open the expanded document flyout and verify the table looks as expected.
  • Navigate to Single document and Surrounding documents views and verify they display as expected.
• Data View Management:
  • Navigate to Data View Management.
  • Create a new data view that includes and excludes indices in its index pattern (e.g. kib*,-kibana_sample_data_e*).
  • Add two runtime fields to the data view and verify in the preview panel that they output as expected:
    • One which outputs a calculated value and uses a field formatter.
    • Another which is a composite runtime field.
  • Edit an existing field with a change such as a custom label.
  • Verify that the runtime fields exist, and the edited field changes are reflected in the fields table.
  • Switch to the field filters tab and add a new field filter.
  • Verify that the added field filter matches the expected fields.
  • Navigate to the Data View Management listing page and verify the created data view appears in the list.
  • Delete the created data view and verify that it’s removed from the list.

API integration tests

Our API integration tests should run the same in stateful and Serverless, and for the most part we should be able to copy and run them with few modifications:

• test/api_integration/apis/data_views
• test/api_integration/apis/data_view_field_editor
• test/api_integration/apis/kql_telemetry
• test/api_integration/apis/scripts
• test/api_integration/apis/search
• x-pack/test/api_integration/apis/management/rollup/index_patterns_extensions.js
• x-pack/test/api_integration/apis/search

Example plugin tests

We have a number of example plugins that we use to run functional tests against various components and services, which we should copy and run in Serverless:

• test/examples/data_view_field_editor_example
• test/examples/discover_customization_examples
• test/examples/field_formats
• test/examples/partial_results
• test/examples/search
• test/examples/unified_field_list_examples
• x-pack/test/examples/search_examples

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[davismcphee]

Closing this since all planned initial serverless test migrations are now complete.