Many PRs blocked due to failing Taskcluster stability checks

[foolip]

With #14165 resolved, Taskcluster was made a required check again on Dec 12. Since then, many PRs have become blocked on a Taskcluster failure:

wpt-chrome-dev-stability failures:

• Made the trusted-types tests run correctly in a vanilla wptrunner. #14400
• Reland "Add quota usage details tests for all other storage backends." #14531

wpt-firefox-nightly-stability failures:

• [WebLocks]: Modifying weblocks algos to be O(1) #14420
• Snap the snapped_dest_rect for backgrounds #14469
• lazy-load: Increase timeout value to reduce flakes #14536

wpt-chrome-dev-stability + wpt-firefox-nightly-stability failures:

• Perform safe addIceCandidate together with signaling handshake #13906
• Update RTCPeerConnection-helper.js #14417
• Remove some useless test-level properties #14519
• [css-text] Correct test metadata #14551

Some issues filed:

• Many Resource Timing tests are flaky #14433
• Some User Timing tests are flaky #14434

Authors of those PRs, please comment here to say whether the failure was related to your changes. If we have too many PRs blocked for unrelated reasons, we'll have to reconsider.

[foolip]

@jgraham

[jgraham]

All but one seems to be an actual stability check failure, so if there's a problem here it isn't with Taskcluster as such, but with the stability checking.

[foolip]

The trouble would be if we have so much pre-existing flakiness that an unreasonable number of PRs would be blocked on pre-existing flakiness. I don't know where to draw the line, filed this issue to start collecting cases of it happening.

If it's too bad, I think we'd have to get our hands on data about which tests are most frequently affected and which of those are most flaky, and fix those.

[jgraham]

Fixing flaky tests seems like a good idea for many reasons.

[foolip]

@lukebjerring @mdittmer for any given test that shows up as flaky in these PRs, is there a straightforward way to get a good guess answer to "was it already flaky?" from wpt.fyi? data? The historical red/green boxes hidden behind a button on wpt.fyi still interleave different versions so it's hard to tell why a test is alternating between passing and failing.

[frivoal]

For #14551, I've restarted it a couple of times by pushing a rebased commit, and I believe that it was once reported as firefox flakiness, and twice as Chrome's. But I've never been able to reproduce locally, and while instability in a basic non animated layout reftest isn't totally impossible if the browser engine happens to have some kind of non deterministic bug, it is pretty unlikely. So I don't know what's causing this, but I'm pretty sure it isn't the test itself.

[foolip]

Self-assigning so that at least I'll check in later to see how many PRs are blocked on stability.

[lukebjerring]

cc @Hexcles

[lukebjerring]

Yes, we can now somewhat-reliably see existing flakiness in the last ten runs in staging:
https://staging.wpt.fyi/insights

[foolip]

This keeps being the main time sink when monitoring https://ecosystem-infra-rotation.appspot.com/, showing up as "blocked export PRs."

@lukebjerring @KyleJu do you think anything beyond the currently planned work on flakiness metadata will be needed to resolve this? More concretely, will the flakiness still show up as blocked exports and require us to add the metadata, or do we need to block Chromium CLs from landing? There's probably a tricky balance here between the amount of human involvement on our side vs. how much overhead there is for Chromium devs trying to get stuff landed and just using WPT because it's the default thing to do. Disincentives for using WPT would be bad :)

[foolip]

Closing this, work is underway to not block on known flakiness.