Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Added testomatio reporter for ui tests #21166

Merged
merged 3 commits into from
Oct 12, 2023
Merged

[WIP] Added testomatio reporter for ui tests #21166

merged 3 commits into from
Oct 12, 2023

Conversation

DavertMik
Copy link
Contributor

@DavertMik DavertMik commented Aug 22, 2023 •
edited
Loading

This PR adds Testoamt.io reporter to UI tests

  • mocha updated to 8.x
  • installed testomatio reporter

Reporter requires API key to be passed either as TESTOMATIO environment variable, so GitHub action should be updated to include that. Or set in configuration as it is shown on this PR.

Tests should be executed like this:

TESTOMATIO=tstmt_{validAPiKey}  ./console tests:run-ui

@tsteur
Copy link
Member

tsteur commented Aug 28, 2023

There should now be an action secret TESTOMATIO configured

@github-actions
Copy link
Contributor

If you don't want this PR to be closed automatically in 28 days then you need to assign the label 'Do not close'.

@github-actions github-actions bot added the Stale The label used by the Close Stale Issues action label Sep 12, 2023
@achakko achakko marked this pull request as ready for review September 22, 2023 01:52
@achakko achakko added Needs Review PRs that need a code review and removed Stale The label used by the Close Stale Issues action labels Sep 22, 2023
@achakko achakko requested review from sgiehl and bx80 September 22, 2023 01:53
sgiehl
sgiehl previously requested changes Sep 22, 2023
View reviewed changes
Copy link
Member

@sgiehl sgiehl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't easily update mocha. Looking at the test output, this now lacks any details which tests are running or failing. If a mocha update is required we might need a separate task to investigate the requirements and update mocha and maybe also chai...

@achakko achakko added the Do not close PRs with this label won't be marked as stale by the Close Stale Issues action label Sep 25, 2023
@achakko
Copy link
Contributor

achakko commented Sep 25, 2023

We can't easily update mocha. Looking at the test output, this now lacks any details which tests are running or failing. If a mocha update is required we might need a separate task to investigate the requirements and update mocha and maybe also chai...

Hi @sgiehl I will add this to backlog to likely prioritise in upcoming sprints.

@achakko achakko self-assigned this Sep 25, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 3, 2023

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions github-actions bot added the Stale The label used by the Close Stale Issues action label Oct 3, 2023
@achakko achakko requested a review from a team October 3, 2023 01:45
@github-actions github-actions bot removed the Stale The label used by the Close Stale Issues action label Oct 3, 2023
@michalkleiner
Copy link
Contributor

@achakko @DavertMik unfortunately our UI tests screenshots preview service currently can't handle branches with slashes in them, so we don't see what UI tests are failing.

I think it'd be best to close this PR and open a new one from a branch that doesn't contain one in its name. Also remove the WIP from the PR title if it's no longer WIP or make a draft one instead.

@sgiehl
Copy link
Member

sgiehl commented Oct 4, 2023

@michalkleiner Actually github isn't able to upload any screenshots for PRs coming from forks. So another branch name wouldn't help there. But as this requires a mocha/chai update, we won't be able to merge that anyway.

@sgiehl sgiehl marked this pull request as draft October 4, 2023 07:22
@sgiehl sgiehl removed the Needs Review PRs that need a code review label Oct 4, 2023
@achakko achakko changed the base branch from 5.x-dev to dev-17222-testomat-internal October 12, 2023 19:17
achakko
achakko approved these changes Oct 12, 2023
View reviewed changes
Copy link
Contributor

@achakko achakko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving pull request to internal branch

@achakko achakko removed request for a team and bx80 October 12, 2023 19:19
achakko
achakko approved these changes Oct 12, 2023
View reviewed changes
Copy link
Contributor

@achakko achakko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving pull request to internal branch

@achakko achakko dismissed sgiehl’s stale review October 12, 2023 19:21

Moving pull request to internal branch

@achakko achakko marked this pull request as ready for review October 12, 2023 19:23
@achakko achakko merged commit 089e585 into matomo-org:dev-17222-testomat-internal Oct 12, 2023
10 of 20 checks passed
achakko added a commit that referenced this pull request Oct 15, 2023
@DavertMik @achakko
[WIP] Added testomatio reporter for ui tests (#21166)
09f2338 
* added testomatio reporter for ui tests

* removed api key from code

---------

Co-authored-by: achakko <[email protected]>
@sgiehl sgiehl added c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. and removed Do not close PRs with this label won't be marked as stale by the Close Stale Issues action labels Oct 30, 2023
sgiehl added a commit that referenced this pull request Nov 13, 2023
@achakko @DavertMik
@mneudert @sgiehl
Update mocha, chai for testomat reporting internal (#21400)
6d10dce 
* [WIP] Added testomatio reporter for ui tests (#21166)

* added testomatio reporter for ui tests

* removed api key from code

---------

Co-authored-by: achakko <[email protected]>

* Added testomatio environment variable to matomo-tests.yml

* Added testomatio environment variable to matomo-tests.yml

* Added testomatio environment variable to matomo-tests.yml

* Added testomatio environment variable to matomo-tests.yml

* Added mocha-multi-reporter to package.jso to enable mocha reporting for standard spec and testomat, also added to config.

* Added mocha-multi-reporter to package.jso to enable mocha reporting for standard spec and testomat, also added to config.

* added testomatio reporter in addition regular spec

* added testomatio reporter in addition regular spec

* Updated one test to ensure it is reported into testomat correctly.

* Updated one test to ensure it is reported into testomat correctly.

* Added comments for multiple mocha reporters

* Adding nyc mocha to report test coverage

* Updated multiple mocha reporter comment and added mocha-multi-reporter dev dependency to screenshot-testing/package.json

* Added mocha-multi-reporter dev dependency to screenshot-testing/package.json

* Added mocha-multi-reporter dev dependency to screenshot-testing/package.json

* Added mocha-multi-reporter dev dependency to screenshot-testing/package.json

* Added mocha-multi-reporter dev dependency to screenshot-testing/package.json

* deleting .npmrc file

* updated lockfile

* Configured mocha-multi-reporter in config and run-tests files

* typo in reporters

* Trying to force mocha colours

* Trying to force mocha colours

* removed deprecated useColors method and kept color method

* Updated dependencies in package.json and updated condition to run spec tests locally and testomat in CI

* Update screenshot-testing lockfile

* Always export reporterOptions from UI config

* revert changes to package-lock.json

---------

Co-authored-by: Michael Bodnarchuk <[email protected]>
Co-authored-by: Marc Neudert <[email protected]>
Co-authored-by: sgiehl <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@achakko achakko achakko approved these changes

@sgiehl sgiehl sgiehl left review comments

Assignees

@achakko achakko

Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@DavertMik @tsteur @achakko @michalkleiner @sgiehl