Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm

[github-actions]

🚨 Failure Summary 🚨:

• 📋 Job Name: e2ePerformanceTests / Run E2E tests in AWS device farm
• 🔧 Failure in Workflow: Process new code merged to main
• 🔗 Triggered by PR: PR Link
• 👤 PR Author: @perunt
• 🤝 Merged by: @roryabraham
• 🐛 Error Message:
  warning: Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: realm/aws-devicefarm@7b9a912. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
  failure: Process completed with exit code 1.
  warning: No matching artifact of type LOG found for debug.log, skipping
  failure: Test run failed after 1500 seconds with: undefined. Timeout is set to 5400

⚠️ Action Required ⚠️:

🛠️ A recent merge appears to have caused a failure in the job named e2ePerformanceTests / Run E2E tests in AWS device farm.
This issue has been automatically created and labeled with Workflow Failure for investigation.

👀 Please look into the following:

1. Why the PR caused the job to fail?
2. Address any underlying issues.

🐛 We appreciate your help in squashing this bug!

[roryabraham]

cc @perunt

[hannojg]

cc @kirillzyusko

[hannojg]

The e2e tests are failing because a new e2e test was added. When we run the tests on the latest release version it doesn't include those tests yet.
Right now that causes an error. We've made a PR to only throw a warning in this case (since this is expected)

• [NoQA] e2e: ignore "missing test" error #39009

[hannojg]

However, in addition to the error mentioned there might be another problem that was introduced on main since this error occurred. However, to investigate that other error, we should get the PR mentioned earlier first.

The other error is about a bug where openReportID isn't being called and thus opening a thread is broken and resulting in a "Hmm... its not here" error

[mountiny]

Merged the PR

@kirillzyusko @hannojg Can you define the other error in separate issue with clear steps/ explanation? This will help us keep the concerns separated. Thank you

[kirillzyusko]

Can you define the other error in separate issue with clear steps/ explanation? This will help us keep the concerns separated. Thank you

I've tried to reproduce it again on latest main and it's not reproducible, so i think it's kind of got fixed 😅 Anyway if I can reproduce it again - I'll create a new issue, but it's not reproducible on my end right now.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.57-5 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] e2e: ignore "missing test" error #39009

If no regressions arise, payment will be issued on 2024-04-05. 🎊

For reference, here are some details about the assignees on this issue:

• @hannojg does not require payment (Contractor)

[hannojg]

Hey, the fix has landed in production, however the e2e tests keep failing.
The issue is due to the new comment linking test being stuck.

We are working on a fix immediately!

[melvin-bot]

Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.60-13 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] fix: temporarily disable comment linking test (e2e) #39279

If no regressions arise, payment will be issued on 2024-04-15. 🎊

For reference, here are some details about the assignees on this issue:

• @hannojg does not require payment (Contractor)

[roryabraham]

I don't think any payments are due here

[roryabraham]

Also, I think it's just disabled, not fixed

[mountiny]

@hannojg @kirillzyusko have we fixed this test? Or are we tracking it elsewhere? could we close this issue?

[kirillzyusko]

Hey @mountiny

I believe there was a bunch of changes:

• we made the error not fatal: [NoQA] e2e: ignore "missing test" error #39009
• we disabled a test: [NoQA] fix: temporarily disable comment linking test (e2e) #39279
• we wrote a handler that accepts test suit inconsistency between branches: fix: gracefully handle missing test cases between two apps #40689

Theoretically we can come back a test with linking back and see if it passes or not. But I remeber on my local setup it showed an error page (instead of the actual page).

And from what I know @perunt is investigating it right now.

[roryabraham]

seems like these are still disabled - are we good to turn them back on or is more needed here?

[kirillzyusko]

@roryabraham I can create a PR with uncommenting them back, but would need to trigger e2e test suite for this branch/PR then (or you can create it on your own - I think it would be faster). If they pass -we can merge the PR, if not - then need to investigate why they would fail.

[roryabraham]

Seems someone beat me to it and re-enabled these and they're passing.