👷 fix tags to match the test optimisation configuration attributes

[thomas-lebeau]

Motivation

The main objective of this PR is to play nice with CI Optimization product. It uses a combination of configuration attribute and test name to identify unique test and detects flakynes by checking if a unique test run more than once with different status for a give commit (So if we only discriminate test by name, a test that fails on Firefox but passes on Chrome would be incorrectly detected as flaky).

With the change on this PR, for a given commit, we should see only one test run per fingerprint (unless a job was retried). Playwright has build-in duplicate test name detection, but not Karma/Jasmine, so I created a simple reporter plugin to log an error if a test is duplicated. Reporters can't fail a job, so it's only a log, but it can help detecting them faster.

Another issue, is that playwright does not report all test run when it retries a test, so CI Optimization has no visibility on flaky test (unless the job is retries). To mitigate the issue I'm adding a test.flaky tag at tearDown when we detect a job has be retried. It won't appear in CI Optimisation but we can at least make a dashboard for it

Changes

Testing

• Local
• Staging
• Unit
• End to end

---

I have gone over the contributing documentation.

[cit-pr-commenter]

Bundles Sizes Evolution

📦 Bundle Name	Base Size	Local Size	𝚫	𝚫%	Status
Rum	147.99 KiB	147.99 KiB	0 B	0.00%	✅
Logs	52.20 KiB	52.20 KiB	0 B	0.00%	✅
Rum Slim	106.66 KiB	106.66 KiB	0 B	0.00%	✅
Worker	24.50 KiB	24.50 KiB	0 B	0.00%	✅

🚀 CPU Performance

Action Name	Base Average Cpu Time (ms)	Local Average Cpu Time (ms)	𝚫
addglobalcontext	0.005	0.002	-0.004
addaction	0.108	0.039	-0.069
addtiming	0.002	0.001	-0.001
adderror	0.207	0.062	-0.145
startstopsessionreplayrecording	0.018	0.010	-0.007
startview	0.791	0.379	-0.412
logmessage	0.037	0.022	-0.015

🧠 Memory Performance

Action Name	Base Consumption Memory (bytes)	Local Consumption Memory (bytes)	𝚫 (bytes)
addglobalcontext	31.68 KiB	30.77 KiB	-937 B
addaction	61.29 KiB	58.46 KiB	-2894 B
addtiming	30.61 KiB	28.69 KiB	-1965 B
adderror	65.39 KiB	63.00 KiB	-2448 B
startstopsessionreplayrecording	28.76 KiB	27.31 KiB	-1482 B
startview	430.54 KiB	428.15 KiB	-2440 B
logmessage	64.17 KiB	61.06 KiB	-3178 B

🔗 RealWorld

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.75%. Comparing base (aec6c32) to head (5329513).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3337   +/-   ##
=======================================
  Coverage   92.75%   92.75%           
=======================================
  Files         299      299           
  Lines        7852     7852           
  Branches     1792     1792           
=======================================
  Hits         7283     7283           
  Misses        569      569           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomas-lebeau]

skipping in beforeEach prevent createTest() to run, which is nice but then the tags used for CI optimisation fingerprinting are not set.

[thomas-lebeau]

Duplicate test, see line 255

[thomas-lebeau]

Duplicate test, see line 214