[RFC] Moving future tests out of Functional Test Repo

[manasvinibs]

I will be doing POC on moving future tests out of FT repo. I will be eager for feedback now and on the first PR for this.

Problem:

Today we have Functional test repo which hosts integration tests for OpenSearch Dashboards along with tests for plugin components.
The existing system we employ involves a separation between the tests and test infrastructure (located in a functional test repo) and the primary repository containing our application code (OpenSearch Dashboards). This separation was initially implemented to centralize the test architecture in a single location, thereby allowing Dashboards, other plugin repositories, and hosted services to leverage a complete test library for their testing needs. However, this setup introduces an unintended challenge known as the "chicken and egg" problem. This problem arises because tests and code changes cannot occur simultaneously. Instead, one must precede the other, which can disrupt the continuous integration (CI) testing process of the other component.

1. Shared Test Infrastructure: In our current setup, the test infrastructure is centralized in a separate repository (Functional Test Repository). This means that the global Cypress functions and configuration are defined and shared from a common location for all plugin repositories, including OpenSearch Dashboards.
2. Multiple Tests in Different Plugin Components: When we have multiple plugin components (along with OpenSearch Dashboards and other plugins) running tests simultaneously or in a mixed order, there's a potential for conflicts. Since the global Cypress functions and configuration are shared, changes made by one test can affect the behavior of other tests.
3. Overwriting Global Functions: If one test in a particular plugin component makes changes to the global Cypress functions, such as modifying or reconfiguring them to suit its specific needs, those changes can inadvertently affect the behavior of tests in other plugin components. This can lead to unexpected failures or side effects in those tests.
4. Synchronization Issues: Tests in one component might be modifying Cypress functions at a time when the global configuration is not as expected, causing synchronization issues and test failures.
5. Test Isolation: Proper test isolation is essential to ensure that changes in one test do not impact the behavior of other tests. When different plugin components share a common testing infrastructure with global Cypress functions, maintaining this isolation can be challenging.

GOAL:

• Each component should be able to write code and its associated tests simultaneously and have the ability to include it in continuous integration.
• To ensure that changes in one component's tests do not disrupt the testing of other components.

* Note: Infra/Build team will continue to rely on Functional Test repo as-is. They can opt into running it better if they want. But ideally it can be ran without any changes on their side, and solves the problems we see today.

[Proposal] Test orchestration and execution across multiple repositories

To address the "chicken and egg" problem and the associated issues of overwriting global Cypress functions, we can use test-library, available for component repos and functional test repo to write in-house tests. Here's a proposed design:

• Multi-Repository Testing Framework:
  • Create a dedicated testing framework repository that contains the shared test infrastructure and configurations. Looks like we already have something similar in opensearch-dashboards-test-library. But current functional test repository might contain some of the common utility functions which should ideally be moved to test library.
  • This library should serve as a centralized place for maintaining common testing tools, fixtures, and configurations.
• Plugin Repositories:
  • Each plugin repository should be responsible for its own specific tests and test setup.
• Test Framework as a Dependency:
  • Set up a mechanism to include the testing framework repository as a dependency in each plugin's test repository. We already have a test library which provides utility functions, page object models, test fixture handling for developers to write functional tests for OpenSearch Dashboards and plugins. It is licensed under the Apache v2.0 License and can be imported by test case repositories.
• Configuration Overrides:
  • Allow each plugin repository to override specific configurations or fixtures provided by the testing framework. This way, plugins can customize their test environment as needed.
• CI/CD Integration:
  • Each plugin component should be able to configure their CI/CD pipeline to build and test their repository independently. This ensures that each plugin's tests run in isolation from others.
• Documentation and Best Practices:
  • Provide clear documentation and best practices for plugin developers to follow when setting up and running tests.
• Collaboration and Communication:
  • Encourage collaboration and open communication between plugin developers and those maintaining the testing framework. Developers should be able to request enhancements or report issues related to the testing environment.

Pros:

• The testing framework serves as a foundation for shared testing tools and configurations, ensuring consistency and code reusability across plugin repositories.
• At the same time, plugin-specific test repositories maintain their independence, allowing customization while avoiding conflicts with other plugins.
• The tests for various components can be executed on dedicated runners in their respective repositories, using resources effectively. Resource management is decentralized, reducing the potential for resource contention.
• The separation of tests and centralized control allows for timely feedback and notifications. Any test failures in plugins or Dashboards components can trigger immediate notifications, enabling quick response and issue resolution.

Cons:

• Dependencies between components, plugins, and shared testing resources must be managed carefully to prevent conflicts and ensure compatibility.
• Collecting and reporting test results from multiple repositories and CI systems can be complex.

Changes to existing Functional Test Repo:

Reference doc: #3728

To ensure that the existing Functional Test Repo can run continuous integration (CI) checks as part of the release workflow on all the plugin components' tests in one place, we can establish a CI/CD workflow that coordinates the execution of tests across all repositories.

• Central CI/CD Workflow:
  • Create a central CI/CD workflow in the Functional Test Repository. This workflow will serve as the orchestrator for running tests across all plugin components.
• Maintain Repository list:
  • This is highly supported within current build system. When a new OSD plugin wants to exist within OpenSearch Project they must add themselves to the manifest
• Test Execution Steps:
  • Within the central CI/CD workflow, include steps to iterate through the list of plugin repositories.
  • For each plugin repository, trigger the respective test workflow in that repository. We can use mechanisms such as Github actions API, web hooks to trigger the workflow.
• After triggering the test workflows in each plugin repository, collect the test results and reports from all repositories.
• Based on the test results from all plugin repositories, we can decide whether to fail or succeed the central CI workflow in the Functional Test Repository.
• Implement notification and alerting mechanisms to notify stakeholders or teams when a test fails in any of the plugin repositories. This ensures timely response and resolution.
• By setting up this central CI/CD workflow in the Functional Test Repository, we can efficiently run CI checks for all plugin components in one place. It allows us to coordinate and manage the testing process across multiple repositories, making it easier to track and report on the status of tests for all plugins. This approach centralizes testing and reporting while providing visibility into the overall health of the entire ecosystem of plugin components.

https://github.com/opensearch-project/opensearch-dashboards-test-library

Library/Module	Repo	Notes
Cypress	Test-library	Today in FTR
Cypress-based objects shareable across tests	Test-library
Cypress-based objects useful in just one plugin	the relevant plugin
Orchestrator	FTR	requires list of test locations
Test logic that accompanies a commit in a plugin	the relevant plugin	requires plugin to include test-library
Pre-existing test logic	FTR
Indices fixture loading for individual plugins	Test-library	phase 2. This can help optimize

Please add your suggestion in the comments below.

[SuZhou-Joe]

For Pre-existing test logic, I guess after this issue gets done, the pre-existing test logic should goes to respective plugin repos right?

[SuZhou-Joe]

As we have security plugin, how will all the test cases be triggered under security mode and unsecurity mode?

[manasvinibs]

For Pre-existing test logic, I guess after this issue gets done, the pre-existing test logic should goes to respective plugin repos right?
Eventually yes but not necessarily part of this issue. This RFC mainly focuses on giving ability for the tests to reside where code exist and still be part of the Functional test repo's release workflow. Pre-existing logic can still remain in FT repo as orchestrator will be able to support both current and future test runs.

[ruanyl]

@manasvinibs How will this change impact the current Jenkins integration test workflows?

[manasvinibs]

@manasvinibs How will this change impact the current Jenkins integration test workflows?

This change is incorporated by integrating it within integtest.sh script into the Functional Test (FT) Repo. The integtest script is currently being invoked by the existing Jenkins integration test workflow during the execution of Cypress tests within the Functional test repo. Consequently, the orchestrator script is also triggered as part of the workflow, initiating remote workflow tests.
For the initial iteration, the objective is to seamlessly integrate the script into the process and conduct a dry run to validate its functionality. This dry run will serve as a preliminary test to ensure the smooth execution of the script within the Jenkins job.