Automated UI E2E Testing

[charmi-v]

Overview

• To implement Cypress as the automated E2E testing framework across modules, focusing on core functionalities to maintain workflow consistency, facilitate early bug detection, and improve release reliability.
• This proposal outlines adding Cypress as a project dependency, developing fundamental tests for critical modules, and integrating these tests into the CI/CD pipeline for automatic execution in the development environment and deployment (possibly in the future).

Explain the topic in 2 sentences

• Integrating Cypress for E2E testing will enable automated validation of essential workflows across the application's core modules.
• This enhances release reliability and accelerates the detection of issues during the development lifecycle.

What's the benefit?

This implementation improves quality assurance by providing efficient, reliable testing automation. The streamlined CI/CD integration enables faster releases with a reduced risk of errors reaching production.

What are the Risks/Dependencies ?

It requires collaboration and review of the CI/CD pipeline to ensure seamless integration.

Detailed explanation

• To establish a robust E2E testing framework, Cypress will be added as a project dependency. Initial test coverage will focus on primary modules such as apps, services, and user management (which needs to be defined).
• Cypress tests will then be integrated into the CI/CD pipeline during development, enabling automated execution upon the creation of new features and ensuring consistent verification of workflows.

Current implementation

• Manual testing is currently used to validate workflows, which is both time-consuming and susceptible to human error, impacting the overall release process.

Proposed improvements

• Introducing Cypress as an automated E2E solution will replace much of the manual testing, allowing for continuous validation, early detection of bugs, and improved reliability in deployment processes.
• Additionally, configuring the @cypress/code-coverage plugin will collect and save code coverage metrics, enabling deeper insights into test coverage.

Feature Team

Contributor

• @charmi-v
• @shahmargi12

Committer

• Committer 1
• Committer 2

User Stories

• Issue 1, linked to the project repository for the addition of Cypress as a dependency.
  • [Setup] Installation & Configuration modification based on project (typescript)
  • [Scripts] Define test runner scripts
  • [Plugin Usage] Configure & Integrate the code coverage plugin
• Issue 2, linked to the project repository for creating a standard documentation/reference for defining the modules to be tested & any standard developer guide for writing tests
• Issue 3 (to be created), linked to the project repository for CI/CD pipeline integration while pull request creation or deployment

Acceptance Criteria

• Add Cypress as a project dependency to enable E2E testing capabilities.
• Develop essential Cypress tests to cover modules such as apps, services, user management, etc.
• Integrate Cypress tests into our CI/CD pipeline for development, enabling seamless automated execution upon deployment (possibly in the future)

Test Cases

• Create module-based tests based on provided FAQ steps (defined in https://github.com/eclipse-tractusx/portal-assets repository) to ensure the testing of modules according to their functionality.

Test Case 1

Steps

1. Do something
2. Click something
3. Add something

Expected Result

1. Expectation
2. Expectation
3. Expectation

Architectural Relevance

The following items are ensured (answer: yes) after this issue is implemented:

• This feature aligns with our current architectural guidelines
  • Data Sovereignty: All data sharing activities across company boundaries follow the Catena-X Regulatory Framework, in particular the Data Exchange Governance, and the Dataspace Protocol via a compliant Connector (like the tractusx-edc or similar, see Connector KIT)
    • CX-0010 Business Partner Number
    • CX-0013 Identity of Member Companies
    • CX-0018 Data Space Connectivity (EDC)
    • CX-0049 DID Document Schema
    • CX-0050 Framework Agreement Credential
    • CX-0149 Verified Company Identity
  • Interoperability: Digital Twins are used (compliant to the Digital Twin KIT and the Industry Core KIT)
    • CX-0001 EDC Discovery API
    • CX-0002 Digital Twins in Catena-X
    • CX-0018 Data Space Connectivity (EDC)
  • Data Format:
    • The data model is based on a published Semantic Model
• The impact on the overall system architecture has been assessed. The Feature does not require changes to the architecture or any existing standard? Please have a look here on the overarching architecture
• Potential risks or conflicts with existing architecture has been assessed

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification)

Additional information

• I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)

[evegufy]

Relates to #419, which had a backend api testing focus up to now, might be a good idea to rename the issue to Automated UI E2E Testing to differentiate it from #419.

Also, relates to eclipse-tractusx/portal-backend#617: a similar issue might be needed for portal frontend, and maybe it's better to use LocalDev as that sandbox could be kept more easily up to date with portal-backend changes than the umbrella, at the same time, LocalDev doesn't provide any interfaces other than the portal-backend and the keycloak instances, in comparison the umbrella values.yaml for portal. That decision depends on the kind of test cases and if they are supposed to include also functionalities where interfaces (other than portal-backend, centralidp, sharedidp) are involved.

[charmi-v]

@evegufy I won’t be able to join the refinement session today and would like to discuss it in the Open Planning on Wednesday.

[evegufy]

Sana will connect to @charmi-v, as she's also planning to introduce Cypress.

[charmi-v]

I have listed some direct references for key subtasks to kickstart Cypress implementation in the repo:

• Setup: Install Cypress and adjust configuration as needed (Install Guide, Config Guide).

• Scripts: Define scripts for Cypress commands (Command-Line Guide).

• Testing Modules: Create module-specific tests (apps, services, user management, etc.) (taking reference from Repo Link).

• Standards: Use Cypress recipes for credential storage, reusable functions, and test structuring (Example Recipes).

• Code Coverage: Use @cypress/code-coverage plugin (Plugin Guide).

• CI/CD Integration (In-depth assessment needed) : Set up Cypress in CI/CD, including GitHub Actions (CI Guide, GitHub Actions).

Please share thoughts regarding the CI/CD Integrations & defining testing modules
@evegufy @ss-nikunj @manojava-gk @oyo

[evegufy]

Committers:
@oyo
@ntruchsess
@Phil91
@evegufy

[evegufy]

Hi @charmi-v, I was informed by @mgarciaLKS and he and @CDiezRodriguez and @gomezbc, would like to contribute.
I think maybe in the context of ci/cd integration would make sense.

[mgarciaLKS]

Hi @charmi-v, I was informed by @mgarciaLKS and he and @CDiezRodriguez and @gomezbc, would like to contribute. I think maybe in the context of ci/cd integration would make sense.

Hello all, @gomezbc is not available, but @CDiezRodriguez and I would love to help with this task if possible.

[evegufy]

@mgarciaLKS great! I guess you could start by creating an issue in the portal-frontend for creating a GH action workflow which spins up the Umbrella sandbox in a Kind cluster.
@charmi-v what do you think?

[charmi-v]

Agreed, @evegufy , that sounds perfect.
I’m setting up an initial testing setup with common functionalities that can be executed locally.
I’ll be sharing the PR for review soon so we can combine efforts, test, and progress the initial testing setup and CI/CD workflow

[charmi-v]

We have an internal sync scheduled for tomorrow (at 2:15 IST) to plan the test automation alignment.
@CDiezRodriguez @mgarciaLKS could you please share your email so we can include you in the meeting? This will help us all gain clarity on the feature integration.

[charmi-v]

Meeting Points Discussed

1. Create a GitHub Issue

   • Clearly outline the scope of automation testing.
   • Define objectives, deliverables, and approximate timelines in the issue description.
   • Action Item: as discussed @sana, please create a new issue in Eclipse Tractus-X Portal Frontend Repository to kickstart discussions and finalize the functionalities to be automated and tested.

2. Define Standards for Test Cases

   • Establish clear guidelines for writing test cases to ensure consistency and quality.
   • Begin with both positive tests (expected scenarios) and negative tests (edge cases) for comprehensive coverage.
   • Suggestion: We should maintain a centralized file to document test-writing standards for easy reference and consistency across teams.

3. Cross-Browser Compatibility

   • Discuss and decide if cross-browser testing should be prioritized at this stage or deferred for later phases.
   • Evaluate the resources and timelines required to implement this effectively.
   • Note: This decision remains pending, requiring team alignment.

4. Integration Automation

   • Review the feasibility and scope of integration automation tests and their alignment with our overall testing strategy.
   • Define a standard procedure for executing these tests across production, development, and QA environments.
   • Tests will be directly added to the frontend repository and executed in an umbrella environment.
   • Action Item: @mgarciaLKS , please create an issue outlining the workflow for integrating automation tests with the repository.

---

Roadmap

Step 1:

• Create the GitHub issue to track setup, testing scope, and automation integration.
• Define test types and begin developing module-wise test cases.

Step 2:

• Finalize the decision regarding cross-browser compatibility and allocate resources accordingly.

Step 3:

• Implement the initial workflow for Cypress tests, including CI integration.
• Gather feedback from the team and refine processes for better efficiency and collaboration.

---

By following this roadmap, we can establish a robust and scalable automation testing strategy that aligns with our project's goals and timelines.

Please share your feedback and suggestions. @evegufy

[evegufy]

@charmi-v sounds like a plan 👍 maybe you could share the progress in nexts week open portal meeting Nov 28? There we could talk about open decision items as well.
Just one note from side regarding the testing scope. For me it would be important to have a maintainable set of tests which test the most important processes within the application (e.g. app release, app subscription, connector registration, ...).

[SanaSalem01]

@charmi-v

Since multiple contributors will work on the Cypress Test Automation,

It would be beneficial to adhere to the following best practices to maintain consistency and uniformity in the tests:

Test Design Best Practices

Test Structure:

• Follow the "Arrange-Act-Assert" pattern to structure tests for clarity and maintainability.
• Group related tests into meaningful categories using describe and context blocks.

Data Management:

• Use fixtures for test data to ensure consistency and avoid hardcoding.
• Implement dynamic data generation for scenarios requiring unique data in each run.

Chai:

Test Runner and Assertions:

• Assertion Library: Chai (already integrated with Cypress) is used for writing assertions. It allows to verify that the app's behavior matches the expected results.

Error Handling:

• Use Cypress's built-in retry functionality for flaky tests.
• Configure robust error-handling strategies to identify and isolate failures.

Additional Configuration

• Custom Commands: Extend Cypress's capabilities by creating reusable custom commands in cypress/support/commands.js.
• Parallel Testing: Enable parallelization to reduce execution time, especially for larger test suites.

Browser Compatibility Testing:

• Test across multiple browsers (e.g., Chrome, Edge, Firefox) using Cypress’s cross-browser support.

Performance and Optimization

Test Run Optimization:

• Categorize tests into smoke, regression, and critical path suites to run targeted test sets based on the need.
• Use Cypress's "test retries" feature to rerun only failed tests.

Reporting and Debugging

• Integrate plugins like Mochawesome or Allure to generate detailed, shareable test reports.
• Include screenshots and videos of failed tests for better debugging. (Optional but good practice)

Logging and Insights:

• Use Cypress's cy.log() and cy.task() commands for debugging and capturing runtime information.
• Set up dashboards like Cypress Dashboard for real-time insights into test runs.

[charmi-v]

@SanaSalem01 , the Test Design Best Practices you shared are helpful. I plan to incorporate them into the frontend cypress README as a reference for developers.

For the initial phase, I suggest excluding Browser Compatibility Testing, Performance and Optimization, and Reporting and Debugging, focusing on core practices to start with.