Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage/estimate-usage-details-application-cache.https.tentative.html is "flaky" (fails ./wpt run --verify) #15036

Open
foolip opened this issue Jan 24, 2019 · 1 comment

Comments

@foolip
Copy link
Member

foolip commented Jan 24, 2019

#14531 adds the test /storage/estimate-usage-details-application-cache.https.tentative.html which was failing the Taskcluster stability check using ./wpt run --verify):

Subtest Results Messages
OK
estimate() shows usage increase after app is cached FAIL: 5/10, PASS: 5/10 assert_greater_than: expected a number greater than 3364 but got 3364

I confirmed this locally in #14531 (comment).

The test isn't flaky in the sense of randomly failing, but it involves browser-global state, and ./wpt run --verify appears to run the test in such a way that the global state is shared across test invocations.

This could possibly be fixed in a manner similar to #15018.

@foolip
Copy link
Member Author

foolip commented Jan 24, 2019

@jgraham have you seen cases like this with Firefox as well? Will tests that have this problem ever run into trouble in Mozilla's CI setup? What I'm wondering is whether it's too high a bar to require tests to involve no global state. I don't know the details of the ./wpt run --verify scheduling, but I guess the tests are run in sequence at least, never in parallel? (Parallel would make it very hard to write fullscreen tests.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant